newly created task can't be deleted if no engines are running

[kaazoo]

This might have something to do with the changes in #1391

When I create a new task with IPython-0.13dev I can't delete it if no engines are running. The task status is pending.

{u'result_header': None, u'stdout': u'', u'started': None, u'completed': None, u'msg_id': u'1181f9eb-ed82-4977-80b2-c3776cfe951d', u'pyerr': None, u'header': {u'username': u'kaazoo', u'retries': 1, u'msg_id': u'1181f9eb-ed82-4977-80b2-c3776cfe951d', u'msg_type': u'apply_request', u'after': [], u'session': u'4f381d3f53c27bf3bc000000', u'timeout': 0.0, u'date': datetime.datetime(2012, 2, 12, 21, 12, 47, 977000), u'follow': [], u'targets': []}, u'engine_uuid': None, u'pyin': None, u'content': {}, u'resubmitted': None, u'client_uuid': u'5d27a1ca-5bf3-4fa7-9d19-8402b8cddea5', u'submitted': datetime.datetime(2012, 2, 12, 21, 12, 47, 977000), u'queue': u'task', u'stderr': u'', u'result_content': None, u'pyout': None}

When I try to delete it, I get the following exception:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/control_job.py", line 5, in <module>
    pkg_resources.run_script('DrQueueIPython==0.0.1', 'control_job.py')
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script
    execfile(script_filename, namespace, namespace)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/DrQueueIPython-0.0.1-py2.7.egg/EGG-INFO/scripts/control_job.py", line 89, in <module>
    main()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/DrQueueIPython-0.0.1-py2.7.egg/EGG-INFO/scripts/control_job.py", line 72, in main
    client.job_delete(job_id)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/DrQueueIPython-0.0.1-py2.7.egg/DrQueue/client.py", line 477, in job_delete
    self.ip_client.purge_results(task['msg_id'])
  File "<string>", line 2, in purge_results
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 66, in spin_first
    return f(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 1378, in purge_results
    raise self._unwrap_exception(content)
IPython.parallel.error.RemoteError: IndexError(msg pending: u'1181f9eb-ed82-4977-80b2-c3776cfe951d')
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/parallel/controller/hub.py", line 1083, in purge_results
    raise IndexError("msg pending: %r" % pending[0])
IndexError: msg pending: u'1181f9eb-ed82-4977-80b2-c3776cfe951d'

When I have a running engine while creating the task, then it's possible to delete it.
When I don't have a running engine and restart ipcontroller after task creation, then it's also possible the delete the task.

[minrk]

This is correct - you cannot delete tasks that are in-flight from the database, because that would leave the database in an inconsistent state. Perhaps you misunderstand what the database is - it is strictly a record of what has happened, and has exactly no influence on execution. The best way to think of the database (and the Hub in general) is a kind of überlog, which stores enough information that it can reconstruct tasks from the logged information. Interacting directly with the database should be considered read-only for the most part, and purge_results only exists for conserving memory by allowing you to delete old requests that have completed.

If it did let you delete tasks that are pending, it would go as follows:

1. submit task (no engines, so it sits)
2. request arrives at db
3. delete request from db
4. connect engine
5. request runs on engine, returning result
6. orphaned result enters db, not associated with any request

[kaazoo]

OK. Thank's for the explanation. So there is really no way of getting rid of a task, once it was given to any engine? Maybe without talking to the database, but communicating with ZMQ for example?

[minrk]

The way to abort tasks is to use 'client.abort', but I am sure this is not well behaved when there are no engines registered. I (or someone else!) can look into improving that behavior by handling aborts in the Scheduler as well as on the Engines (it is only handled directly on the engines at this point).