Daemonize after engines start

[QuLogic]

Currently, ipcluster can be run in the foreground or in the background using the --daemonize option. However, this daemonizing appears to happen immediately at startup. So if I run something like:

#!/bin/bash

ipcluster-3 start -n 4 --daemonize
python3 <<EOF
from ipyparallel import Client
client = Client()
print(client[:])
EOF

then the script fails because the engines are not available (ipyparallel.error.NoEnginesRegistered). Inserting a sleep 5 will fix it because the engines will have started, but this is basically a guess-and-hope-it-works approach.

It would be really nice if ipcluster would wait to background itself until after the engines have started instead.

[jakirkham]

To do that I have a function like this that I run, which returns the client with all engines up and registered (blocking until that happens). I'd be willing to PR it to ipyparallel if this is of general use and is likely to be accepted.

[QuLogic]

I think that is effectively still pushing the problem downstream when it could be directly solved (I say without looking into the work involved) in ipcluster.

[jakirkham]

I don't know that I agree.

The thing is ipyparallel is doing something pretty smart by letting you decide when you are ready to use the cluster even if you don't have all engines. In some cases a user might have to wait a while for all engines to be ready if they make a large ask of the cluster in terms of workers. Being able to do some work with the engines already available is pretty nice instead of sitting on your hands waiting for the whole request to be filled.

It could be a little nicer if it told you which engines are ready, but I'm not even sure that I would want it to do that for me automatically. A utility function one could use for that would be nice though.

[GF-Huang]

I can't found any docs about daemonize.

[jakirkham]

It's not called daemonize. Please read the linked PR. The function is wait_for_engines

[GF-Huang]

Any way to run ipcluster.exe start in background rather than hangs the command line?

[GF-Huang]

After run ipcluster.exe start --daemonize, why I still get this error?

[minrk]

The --daemonize argument only works on platforms with fork support. #494 actually changes this, so it should work on Windows as well.

Until then, ipcluster start must be launched in its own terminal or via the clusters tab nbextension, or via Popen calls.