Make it visible when ML results stop being processed instead of failing the jobs

[mihow]

Summary

When the workers that process ML results stop consuming, nothing says so. The platform keeps accepting results, queues them, and then quietly throws them away, while the jobs that produced them are marked failed for "no progress". From the outside it looks like the ML pipeline is broken; in fact inference succeeded and the results were discarded on arrival.

That is what happened in a recent production incident. Every result-processing worker had lost its connection to the message broker and never reconnected. The container runtime still reported them as running with a restart count of zero. Results were POSTed successfully — the ML workers logged result posts: 62 (success: 62, failed: 0) — accepted by the API, queued, and never processed. Two large jobs (roughly 37,000 and 11,000 images) were destroyed, and capacity had been silently halving for two and a half days before the last consumer died.

The infrastructure-side alert for this is being handled separately. This issue is about the part the application owns: making the condition observable and not silently destructive.

List of Changes

Change (user effect)	How (implementation)
Operators can see that results are arriving but not being processed	Expose queue depth and consumer count for the result queue as a health signal (management command, health endpoint, or existing monitoring hook — design open)
A job is not marked failed when its results are merely queued rather than lost	Have the stalled-job check distinguish "no results arriving" from "results arriving but not consumed" before it reaps
A late result cannot be silently discarded without trace	Count and surface discards, rather than only logging them at INFO

Detailed Description

Three separate things combined to make this invisible.

Container liveness means nothing here. All eight result-processing workers reported running with RestartCount=0 throughout. The only external symptom was that their logs stopped. Any check based on process or container state passed.

Results are accepted unconditionally. The API takes the POST and queues the work (Queued pipeline result: task_id=...). Nothing verifies that anything is on the other end of the queue. A worker that has lost its broker connection is indistinguishable, from the producer's side, from a healthy one.

The stalled-job check then blames the job. With results queued but unconsumed, process stays at 0% and jobs_health_check reaps the job for no progress. The resulting message points at the job, not at the missing consumers, which sends investigation in the wrong direction. In this incident it also destroyed work that had genuinely completed.

There is a fourth, quieter cost. Once a job is terminal, its late-arriving results hit the discard branch and are logged at INFO as result arrived after the job already finished ...; ignoring. During recovery, roughly 9,200 such messages were consumed and thrown away. That is correct behaviour in isolation, but nothing aggregates it, so a large volume of discarded inference produces no signal at all.

Suggested direction, not a settled design

• A queue-health signal is the strongest single indicator, because "messages ready > 0 and consumers == 0" states the problem directly and cannot be faked by a worker that is alive but not working.
• The reaper change is the higher-value correctness fix but needs care: it must not simply refuse to reap, or genuinely stuck jobs will accumulate. Distinguishing the two cases probably requires knowing whether results for that job are sitting unconsumed.
• Discard counting is cheap and worth doing regardless.

What we still need to verify

• Whether the application can read broker consumer counts cleanly in this deployment, or whether that inspection belongs entirely to infrastructure monitoring. celery inspect proved unreliable in this incident: it returned empty output when workers were wedged, which is precisely when it was needed.
• What the reaper should do when it detects queued-but-unconsumed results — hold the job, extend the deadline, or fail it with an accurate reason. Each has different consequences for a genuinely stuck job.
• Whether surfacing this belongs in an existing health surface rather than a new one, to avoid adding another thing that itself needs monitoring.

Related

• The database keepalive gap that set up this incident: Stop database connections hanging for two hours when the network drops them #1382
• Infrastructure alerting for the same condition is tracked separately in the deployment repo.

[mihow]

Claude says: Two clarifications worth adding so this issue is not mistaken for work already done, plus a correction to one of its stated fixes.

How this differs from #1168. That issue asked for detection of exhausted NATS consumers and was closed by the mark_lost_images_failed and jobs_health_check machinery. That handles the case where NATS has given up on the messages. The case here is different: the messages were never lost, they were sitting in the Celery broker queue in good order with nobody consuming them. Existing machinery has no signal for that, because from its point of view nothing failed — which is precisely why it ran for about seven hours unnoticed.

Correction to the "make time_limit fire" item. That is not independently fixable and should not be pursued as stated. Reading the pinned celery version's source: in the prefork pool the time limits are armed as callbacks on the parent process's event loop, not by a separate thread, and the hard limit is chained off the soft one — if the soft callback never runs, the hard timer is never even created. The parent's event loop was not running, which is why 35 tasks passed a 360-second hard limit without a single limit event. So this is a symptom of the wedge rather than a defect that can be fixed on its own, and lowering the limits would change nothing.

The related infrastructure cause, which now looks causal rather than merely a precondition, is in #1382.

Also relevant: #1276 and #1241 have both been reopened. #1276 fixes the reaper behaviour that destroyed the jobs in this incident, and #1241 makes the resulting failure explain itself instead of leaving progress["errors"] empty. Between them they cover a good part of what this issue asks for, so it may be worth scoping this one down to the queue-observability piece once those land.