Refactor the task dependencies system #105

ipython opened this Issue May 10, 2010 · 2 comments

2 participants


Original Launchpad bug 361419:
Reported by: ellisonbg (Brian Granger).

Currently the task dependency system does lots of work on the controller. This makes the system difficult to use and causes potential performance problems for the controller. It also leads to user code being run on the controller, which we want to avoid. To fix these issue, we will look into refactoring the dependency system in the following manner:

  1. We will create an exception TaskRejectError or something that users should raise in the task code if an engine does have the required deps.

  2. The user will specify the retries argument to get the task to be rerun.

  3. We will also add a new keyword argument to our task objects that tells which engines the task can be run on. We will also modify the scheduler to use this information.

The only issue that needs to be worked out is whether or not we want the schedular to distinguish between a task that fails with TaskRejectError and one that fails for another reason. If we don't distinguish tasks that truly do fail could be retried a large number of times.


[ LP comment 1 by: Brian Granger, on 2010-01-30 22:00:21.813487+00:00 ]

We are going to actually implement taskid based task dependencies to allow full DAG based scheduling.

IPython member

We are not doing any further work on the old Twisted based parallel code.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment