[Bug] job timeout hangs executor

[pmrv]

I ran into an issue where futures will hang indefinitely when the underlying job timeouts or otherwise dies.

Analysis and reproducer by Claude below.

executorlib SlurmClusterExecutor future hangs when its job dies (TIMEOUT/OOM/cancel)

Env: executorlib 1.10.1, pysqa 0.4.3 (file/HDF5 cluster backend), SLURM.

Symptom

When a SlurmClusterExecutor job is killed by the scheduler — walltime TIMEOUT (also OOM,
NODE_FAIL, scancel) — the submitting driver's future.result() blocks forever.

Root cause

The file backend resolves a future only when the task's output file <task_key>_o.h5 appears; the
result-polling path never checks whether the SLURM job is still alive.

executorlib/task_scheduler/file/shared.py:

def _check_task_output(task_key, future_obj, cache_directory, ...):
    file_name = os.path.join(cache_directory, task_key + "_o.h5")
    if not os.path.exists(file_name):
        return future_obj                    # <-- output not present -> return, future left UNRESOLVED
    exec_flag, no_error_flag, result = get_output(file_name=file_name)
    _update_future(future_obj, exec_flag, no_error_flag, result)
    ...

def _update_future(future_obj, exec_flag, no_error_flag, result):
    if exec_flag and no_error_flag: future_obj.set_result(result)
    elif exec_flag:                 future_obj.set_exception(result)
    # exec_flag False -> nothing; future never resolves

A TIMEOUT-killed job is terminated mid-run and never writes _o.h5, so _check_task_output returns
early on every poll and the future is never completed.

The information needed to detect death is already in scope: the poll loop
(execute_tasks_h5 -> _refresh_memory_dict) holds process_dict[task_key] = queue_id,
pysqa_config_directory, and backend, and executorlib/standalone/scheduler.py::terminate_with_pysqa
already does QueueAdapter(...).get_status_of_job(process_id=queue_id). The loop simply never calls it
to notice that a job has vanished.

Reproducer

executorlib_timeout_hang_repro.py — submit sleep(600) with run_time_max=60 (1-min walltime) on the
debug partition. Observed timeline:

[  2s] submitted slow(600) with 1-min walltime (debug partition)
[ 12s..92s] slurm_state=RUNNING   future.done()=False
[102s..152s] slurm_state=TIMEOUT  future.done()=False   <-- job dead, future still pending forever
HANG CONFIRMED

Proposed fix (~5 lines, reuses existing imports)

In the poll path, when <task_key>_o.h5 is absent, also query the job status; if the scheduler no
longer knows the job (status None) and there is still no output, fail the future instead of looping:

# in _check_task_output / _refresh_memory_dict, when the output file is missing:
from pysqa import QueueAdapter
qa = QueueAdapter(directory=pysqa_config_directory, queue_type=backend, execute_command=pysqa_execute_command)
if qa.get_status_of_job(process_id=queue_id) is None and not os.path.exists(file_name):
    future_obj.set_exception(RuntimeError(
        f"executorlib: queue job {queue_id} terminated without output "
        f"(timeout / failed / cancelled)."))

(Guard against the benign race where the job just finished and _o.h5 is about to appear by
re-checking os.path.exists(file_name) after the status query, as above.)

Impact / workarounds until patched

• Any job death (walltime, OOM, node failure, scancel) hangs the driver — not just timeouts.
• Workarounds: (a) set generous walltimes so jobs don't die; (b) monitor squeue/sacct externally
  and treat "job gone + no output" as failure (what this epoch's campaign drivers do via external
  Monitors + resubmit); (c) apply the patch above and upstream it.

executorlib_timeout_hang_repro.py