make it possible to start engine in 'disabled' mode and 'enable' later

[kaazoo]

The problem I have is, that my tasks fail because of UnmetDependency. The dependency can only be met after I ran some code on the engines, which obviously needs to run before a task is given an engine. If a task is given before my code was run, the task will always fail.
A workaround could be to have a 'disabled' mode (for parallel tasks) for an engine at startup. Then it would be possible to run special code on an engine via direct view, 'enable' the engine and finally receive parallel tasks.

[minrk]

If you know what code you need to run when you are bringing up an engine, you can add it as startup script or command, either as:

$> ipengine -s myscript.py
$> ipengine -c "do_stuff()"

Which correspond to the IPEngineApp.startup_script and IPEngineApp.startup_command configurables, respectively, if you want to set them more permanently.

This code is run before the engine is made available to the Scheduler. Can you describe your case in more detail if this doesn't suffice?

[kaazoo]

I would like to get some information about each engine on startup and store it in MongoDB for later use (engine_id, hostname, arch, os, proctype, nbits, procspeed, ncpus, ncpucores, memory, load, address, pools). Depending on these information, I can give tasks to specific engines.
When I try to get these information via dview while long-running tasks are already running, my client is blocked / gets a timeout. That's why I would like to get that information first, before allowing the controller to give tasks to the engine.

I will try out your hint. But I guess that the engine isn't registered at the time excuting startup_script / startup_command, which would mean that the engine_id is unknown.

[kaazoo]

Is it possible to get the engine_id inside of the called startup_script?

[minrk]

Hm,

It's technically possible, though I'm not certain these are fully public APIs that I would trust to remain stable:

from IPython.config.application import Application

# grab the global application instance
app = Application.instance()
# and its integer engine ID
engine_id = app.engine.id

[kaazoo]

Thanks a lot, this works!

Now I can get all needed information about a connected engine at startup and store that info in MongoDB. Later, I can lookup that information with the engine_id of each engine stored in client.ids, without having to wait until the current task is finished. Sadly IPython doesn't seem to support a out-of-band DirectView on engines while LoadBalanced tasks are running.

[minrk]

Sadly IPython doesn't seem to support a out-of-band DirectView on engines while LoadBalanced tasks are running.

Yes and no. Obviously, only one task can run at a time (it is Python, after all), but task and mux queues are fair-queued, so if you submit a direct task, it will run before any queued LoadBalanced tasks, as soon as the current task finishes.

[minrk]

Also, I realize that you can possibly get your own 'disabled' mode in the startup script by running a command that blocks until a certain condition is met. I'm not sure if this would actually work in practice, but it's an idea.