rewrite IPython.parallel scheduler #2866

minrk opened this Issue Jan 29, 2013 · 4 comments


None yet

3 participants

minrk commented Jan 29, 2013

This is mostly a note to myself.

The parallel scheduler should be mostly rewritten. It's a classic case of adding features as I go, and ending up with a weird complicated mess.

Notable changes that should happen:

  1. there should be an associated Application (required for Scheduler sharding / cloning)
  2. the Queue should be its own structure, rather than a mishmash of dicts, lists, and sets in the Scheduler
  3. Add some kind of support for priority
@minrk minrk was assigned Jan 29, 2013
@minrk minrk removed the type-enhancement label Jan 14, 2015

@minrk, we're interested in improving the parallel scheduler. Is this issue already resolved in the current version?

minrk commented Feb 21, 2015

It is not.


If this is a kind of "feature wishlist" :-) : add support for "initial tasks per engine", e.g. for running the equivalent of c[:].execute("%colors nocolor") even if the engine is added after the job is started.

minrk commented Feb 22, 2015

@JanSchulz it isn't, really, it's specific problems that have been caused by growing the Scheduler slowly over time to the point that some things need to be started from the beginning. But you can accomplish the same thing by using startup files or exec_lines, which initialize IPython before the engine is registered with the controller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment