Fix/issue 509

[iparask]

This PR fixes the connection closed exception we were getting for long-running tasks. When the connection or the channel is closed, we receive an exception from pika. We then handle that exception by creating a new connection and channel and send the message again.

Unfortunately, we cannot know the actual RMQ server state for a connection or channel through their is_open or is_closed methods.

Although there is a replication in the messaging code, we avoid the overhead of creating a new connection and channel for every message.

[codecov]

Codecov Report

Merging #539 (1baf30f) into devel (4bb8e6c) will increase coverage by 0.48%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##            devel     #539      +/-   ##
==========================================
+ Coverage   61.80%   62.29%   +0.48%     
==========================================
  Files          21       21              
  Lines        2312     2339      +27     
==========================================
+ Hits         1429     1457      +28     
+ Misses        883      882       -1     

Impacted Files	Coverage Δ
src/radical/entk/execman/rp/task_manager.py	20.24% <0.00%> (-1.91%)	⬇️
src/radical/entk/execman/base/task_manager.py	78.83% <70.00%> (+10.08%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4bb8e6c...1baf30f. Read the comment docs.

[andre-merzky]

LGTM.

[mturilli]

Looks good to me. I would not have renamed channel but if you prefer that I am fine with it.

[iparask]

I would not have renamed channel but if you prefer that I am fine with it.

I did it because I was getting some error somewhere, but I do not remember right now what it was.

Thank you both for reviewing!