Execute DirectView commands while running LoadBalancedView tasks #623

kaazoo opened this Issue Jul 24, 2011 · 2 comments

2 participants


While LoadBalancedView tasks are running on my engines, I would like to have some status from them (current system load for example). While a task is running, my DirectView commands are blocked. They seem to get executed when the current task is finished, which might be too late because status information could be not accurate enough.
Is this just normal and intended?

IPython member

Yes, this is as intended. On the Engine, there is no difference between a load-balanced task and a MUX task. The only difference between the two is the routing mechanism for determining the engine destination, but the distinction vanishes on arrival. There is a separate Control queue that jumps to the head of the line (abort, shutdown, etc. are implemented via this queue, but more queries could be added), but control messages still have to wait for the current task to complete, because the Engine is a single-threaded application (except for heartbeat, which is a pure-C thread).

If we do make the Engine a multithreaded application, then control messages might respond while tasks are running, but that still won't be true if tasks are doing GIL-holding operations, which is highly likely.

The general model is that engine state is to be gathered from the Hub, which tracks jobs run and running via requests like Client.queue_status(), but this obviously doesn't get things like actual CPU load of the remote machine, only which tasks have run / are running on each engine.


OK, thank you for the explanation. Then I will see if I can run my status task from time to time and cache the information between. This may not be very accurate but at least my application is not blocked while waiting for engine status.

@kaazoo kaazoo closed this Jul 27, 2011
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment