rabbitmqctl: "wait" could wait forever if rabbitmq-server fails to create a pidfile

[dumbbell]

The current implementation of rabbitmqctl wait can't distinguish between a node that did not start and a node that failed to create its pidfile. In both cases, rabbitmqctl wait will wait indefinitely. When the node fails to create a pidfile, one would expect the command to return with a non-zero exit status.

The current implementation of rabbitmqctl wait can't distinguish a node who did not start yet from a node who failed to create its pidfile. In both cases, rabbitmqctl wait will spin forever, but in the latter (eg. when used in an init script), one would expect it to return and the init script to notify the startup failure.

[cesarmun]

Is there an approximate timeline to fix this?

[dumbbell]

Not yet. We have more critical fixes to do first and we are busy with the 3.6.0 release at the moment.

[jmoney]

Could this make it into the 3.6.0 release? I've hit this quite a bit and its rather annoying to have to ctrl-c and kill the process.

[dumbbell]

Hi @jmoney8080!

No, unfortunately, it won't make it to 3.6.0. It requires non-trivial changes and we are too far in the release cycle.

[michaelklishin]

It possibly can make it into 3.6.x, we will see how intrusive this ends up being.

[jmoney]

Thanks! If you need an external source to test an RC with it I can help. This issue causes a lot of annoyances when our automation runs(not all the time but when it does happen to causes issues it's annoying to fix).

[michaelklishin]

@jmoney8080 we'll keep you posted ;)

[Dzol]

To distinguish a node which hasn't completed start-up yet, from a node which failed to create its PID file, we could consider the node to have failed after retrying or timing out.

Specifying a timeout on the command-line would be a friendly way to give the user control.

But the danger is that the broker may complete start-up after we consider it to have failed.

[Dzol]

@dumbbell & @michaelklishin: To clarify what I meant above: if the script doesn't get a response from the server (in a given time) saying that it has started successfully, then the script could exit with status/code 1, otherwise 0.

[dumbbell]

Just a random thought while working on something unrelated: we could start an ephemere Erlang node before starting RabbitMQ itself and use net_kernel:monitor_nodes/{1,2} to detect if the RabbitMQ node started but failed to boot.

[michaelklishin]

We will investigate if there are reasonably safe ways to fix this in 3.6.x.

[lourot]

IMHO this is fixed by 69454027 and can be closed, thanks!

EDIT: actually it seems like rabbitmqctl wait --timeout ... has been there for even longer