Recover jobs stuck in Cancelling when no worker acknowledges the cancel

[mattdawkins]

Problem

Users report that cancelling a job in the web interface sometimes leaves it at Cancelling for days.

Cancelling a job makes girder_plugin_worker set status CANCELING (824) and broadcast a Celery revoke(). The job only leaves that state when a live worker acknowledges — either the running task polls and notices the revoke, or Celery discards the revoked message at delivery. Nothing on the server ever times out, so there are permanent dead ends:

• No worker connected to the job's queue (private standalone queue whose worker is offline, or a worker that crashed/redeployed): the revoke broadcast reaches nobody and the job stays at Cancelling forever.
• Worker restarted after the revoke: Celery's revoked-task list is in-memory, so the revoke is lost; the queued task can later run to completion while the job still shows Cancelling (girder_worker swallows the invalid CANCELING → RUNNING transition in its prerun handler).

Fix

• New dive_server/stale_cancellation.py: a sweep that finds jobs sitting in CANCELING with no update for 60 seconds, moves them to CANCELED (a transition the worker plugin's table allows), appends an explanatory line to the job log, and re-broadcasts the revoke so a worker that reconnects later discards the still-queued message instead of running it.
• dive_server/__init__.py: runs the sweep every 30 seconds via a CherryPy Monitor in the Girder server process. The sweep never raises, since BackgroundTask permanently stops on the first uncaught exception.
• The status change reaches open clients through the existing job_status push notifications.

Jobs a worker is actively acknowledging never hit the timeout: any job update (including log writes) refreshes updated, and the subprocess monitor resolves a live cancel within ~30 seconds. Force-cancelling is safe even if the task message is still queued or a worker is in a long non-polling phase — a revoked delivery or late acknowledgment becomes a no-op status write, and a resurrected task fails its first status update instead of silently running.

Tests

tests/test_stale_cancellation.py covers the query cutoff, the force-cancel path, the revoke re-broadcast, and that one failing job doesn't halt the sweep.

🤖 Generated with Claude Code

[BryonLewis]

I think that #1769 is probably a better PR to resolve this issue. It resolves more the cause of the problem then addressing a symptom (the cancellation status) I think I may close this one in favor of that one.