cli_common: reconnect on pulse errors

[rail]

closes #827

[garbas]

😄 i like the draft feature in github

[rail]

@La0 @garbas what do you think about this approach?

It doesn't reconnect, but at least it detects the issue and should exit, letting either heroku or kubernetes to restart the container.

We can probably wrap the function somehow so we reconnect, but I failed to do that properly for now. I may try and pursue this at some point.

[rail]

In eeac03f I added some reconnect logic. it should work (TM), but I'm not sure what would be the best way to test it...

[La0]

You could test it with mitmproxy between you and the pulse service, kill the proxy for a while and restart it: the pulse listener should try to reconnect...

Thanks for this patch, it will be used on pulselistener a lot 👍

[rail]

I actually have a real local rabbitmq server that I can use, was wondering which service I should try to emulate :)

[rail]

Let's try something that I know, shipit worker

[rail]

It worked fine locally.

• I tried the shipit worker without rabbitmq running. It keeps retrying until the server comes up and starts consuming messages afterwards.
• Killing the server keeps the worker running and retrying until the server comes up. The worker reconnects and starts consuming messages as expected.

I think this should just work. We should probably watch the consumers after this merged to production.