[Bug] Unpicklable submit argument kills the scheduler thread and hangs all futures forever

[pmrv]

[Bug] Unpicklable submit argument kills the scheduler thread and hangs all futures forever

Summary

If an argument passed to Executor.submit() cannot be serialized by cloudpickle, the error is raised inside the background scheduler thread, which dies. The submitted future is never resolved, so future.result() blocks forever, every other task on the same executor is abandoned, and shutdown() / the with block exit hang as well (future_queue.join() waits on task_done() calls that never come).

Reproduced on current main (4d1cf57) for all backends:

• FileTaskScheduler with the subprocess spawner (the SlurmClusterExecutor / FluxClusterExecutor code path): future stays pending, scheduler thread dead, cache directory never created, no job ever submitted
• SingleNodeExecutor, both one-to-one and block_allocation=True: future stays pending

Since SingleNodeExecutor is affected too, this is not cluster-specific — it can be hit while prototyping on a laptop.

Reproduction

from executorlib import SingleNodeExecutor


class Unpicklable:
    def __reduce__(self):
        raise TypeError("nope")


with SingleNodeExecutor(max_workers=1) as exe:
    f = exe.submit(len, [Unpicklable()])
    print(f.result(timeout=15))  # expected: TypeError; observed: TimeoutError (would block forever without timeout)

The same with SlurmClusterExecutor hangs identically. Real-world trigger that cost an overnight campaign: an ase.Atoms object still carrying a pyace.PyACECalculator (holds an unpicklable pyace.evaluator.ACEBEvaluator); cloudpickle.dumps on it raises TypeError instantly when called standalone, so the fail-fast error becomes an unbounded hang.

Observed with python 3.12, cloudpickle 3.1.1, executorlib main @ 4d1cf57 (also on 1.10.1 with pysqa 0.4.4 on a real Slurm cluster).

Why the failure is effectively silent

The only trace is the default threading.excepthook traceback on stderr when the scheduler thread dies. In the common "drive a campaign from a notebook inside a batch job" setup even that is lost entirely: under jupyter nbconvert --to notebook --execute --inplace, the kernel's stderr goes over ZMQ into the notebook's cell outputs, and with --inplace the notebook is only written when execution completes — which, on a hang, is never. Verified on a real hung run: a 4.5 h driver left a six-line job log and a notebook with zero captured outputs. The same applies to any driver that redirects or buffers stderr.

For the cluster backends the failure also happens before anything reaches the queuing system, so an external squeue/sacct watchdog sees nothing either. (Distinct from #1037, which has the same observable but concerns jobs dying after submission.)

Where the exception escapes

TaskSchedulerBase.submit() only queues the task dict (task_scheduler/base.py:167-178); the cloudpickle dump happens later in a background thread, and the two backends fail in slightly different ways:

File backend — execute_tasks_h5 calls serialize_funct → cloudpickle.dumps at task_scheduler/file/shared.py:132; the TypeError propagates out of the thread target before future_queue.task_done() (line 189) and before the cache directory is created. The future stays PENDING forever.

Interactive backend — execute_task_dict → SocketInterface.send_dict → cloudpickle.dumps at standalone/interactive/communication.py:65, uncaught in both blockallocation.py:320 and onetoone.py:265. This case is arguably worse: the future has already passed set_running_or_notify_cancel() (interactive/shared.py:41), so future.cancel() is refused and there is no clean way out for the caller at all.

Expected behaviour

A serialization failure should propagate — raised out of submit() or set on the future via set_exception() — rather than ending the scheduler thread and orphaning every pending future. Two complementary fixes suggest themselves:

1. Robustness: wrap the per-task handling in execute_tasks_h5 and the interactive execute loops in try/except Exception, route the exception to task_dict["future"].set_exception(e), call task_done(), and continue serving the remaining tasks.
2. Fail-fast (nice to have): perform the cloudpickle dump in submit() so the caller gets the TypeError at the call site; for the file backend the dump is needed there anyway for the task-key hash.

Workaround

Strip unpicklable members before submitting and verify in-process, e.g. for the ASE case: atoms.copy() drops the calculator, and cloudpickle.dumps(obj) as a pre-submit guard fails loudly instead of hanging in the queue.

---

Filed by Claude (an AI assistant), working on behalf of Marvin Poul — who sends their regards.

[pmrv]

A sketch of fix (1), verified locally against main (4d1cf57) — I can turn it into a PR with tests if the approach looks right to you.

Interactive backend (task_scheduler/interactive/shared.py, execute_task_dict): wrap the dispatch in try/except and fail the future instead of the worker thread.

     if not future_obj.done() and future_obj.set_running_or_notify_cancel():
         if error_log_file is not None:
             task_dict["error_log_file"] = error_log_file
-        if cache_directory is None:
-            return _execute_task_without_cache(
-                interface=interface, task_dict=task_dict, future_obj=future_obj
-            )
-        else:
-            return _execute_task_with_cache(
-                interface=interface,
-                task_dict=task_dict,
-                cache_directory=cache_directory,
-                cache_key=cache_key,
-                future_obj=future_obj,
-            )
+        try:
+            if cache_directory is None:
+                return _execute_task_without_cache(
+                    interface=interface, task_dict=task_dict, future_obj=future_obj
+                )
+            else:
+                return _execute_task_with_cache(
+                    interface=interface,
+                    task_dict=task_dict,
+                    cache_directory=cache_directory,
+                    cache_key=cache_key,
+                    future_obj=future_obj,
+                )
+        except Exception as exc:
+            # e.g. an argument cloudpickle cannot serialize - fail the future
+            # instead of killing the worker thread
+            future_obj.set_exception(exc)
+            return True
     else:
         return True

Two properties that make the broad except Exception safe here: ExecutorlibSocketError is only ever returned inside the output dict (communication.py:82), never raised, so the return False resubmission path is untouched; and cloudpickle.dumps in send_dict fails before any bytes hit the socket, so the worker connection stays usable for subsequent tasks.

File backend (task_scheduler/file/shared.py, execute_tasks_h5): wrap the body of the elif task_dict is not None: branch (everything from _convert_args_and_kwargs through the memory_dict bookkeeping, currently lines 121–188) the same way, keeping future_queue.task_done() outside so the queue accounting stays correct:

        elif task_dict is not None:
            try:
                task_args, task_kwargs, future_wait_key_lst = _convert_args_and_kwargs(
                    ...
                )
                ...  # existing branch body unchanged
            except Exception as exc:
                # e.g. an argument cloudpickle cannot serialize - fail the
                # future instead of killing the scheduler thread
                task_dict["future"].set_exception(exc)
            future_queue.task_done()

Verification (python 3.12.4, cloudpickle 3.1.1), each scenario in an isolated subprocess:

• The Unpicklable repro from the issue now resolves with TypeError for the file backend (subprocess spawner), SingleNodeExecutor one-to-one, and SingleNodeExecutor with block_allocation=True — no hang.
• After a poisoned submit, a subsequent healthy submit on the same executor still completes (bad.exception() → TypeError, good.result() → correct value) on both backends, and shutdown(wait=True) returns cleanly.

Fix (2) — doing the dump in submit() for fail-fast behaviour — is left out of the sketch since it changes when the caller sees the error and interacts with the cache_key override; it seems more of a design decision for you.