When a job fails there is no way to run it again except through the object it belongs to: a document can be reprocessed, a source can be re-extracted. For anything else — bootstrap_ontology, adjudicate_entities, sync_source, materialize_inferences — a failed row in jobs is the end of it until whatever enqueues it comes round again.
This became worth writing down with #201. An OutOfCredit failure now goes to failed on the first attempt instead of spending three. That is right for the operator, who sees the real state seven minutes sooner, but it changes what happens to a batch:
500 documents queued, credit runs out halfway
before each job retried over ~7 min; a top-up inside that window rescued the rest by itself
after every remaining job is failed; after the top-up they have to be triggered again
Per document that is a click; across a batch it is the whole batch. Re-extracting the source works but re-runs the documents that already succeeded, which costs the model calls again.
What would cover it: a way to requeue failed jobs — by kind, by knowledge base, or from the alert that reported the failure. The queue already has what it needs (status, kind, payload, last_error); the handlers are idempotent, which is why orphan recovery can requeue running rows at startup.
Worth deciding whether the alert should carry the action. llm.out_of_credit is the one failure where the operator does something specific (tops up) and then wants the work to resume; an alert that can say "run those again" closes that loop where a queue page would be a new surface to build.
When a job fails there is no way to run it again except through the object it belongs to: a document can be reprocessed, a source can be re-extracted. For anything else —
bootstrap_ontology,adjudicate_entities,sync_source,materialize_inferences— a failed row injobsis the end of it until whatever enqueues it comes round again.This became worth writing down with #201. An
OutOfCreditfailure now goes tofailedon the first attempt instead of spending three. That is right for the operator, who sees the real state seven minutes sooner, but it changes what happens to a batch:Per document that is a click; across a batch it is the whole batch. Re-extracting the source works but re-runs the documents that already succeeded, which costs the model calls again.
What would cover it: a way to requeue failed jobs — by kind, by knowledge base, or from the alert that reported the failure. The queue already has what it needs (
status,kind,payload,last_error); the handlers are idempotent, which is why orphan recovery can requeuerunningrows at startup.Worth deciding whether the alert should carry the action.
llm.out_of_creditis the one failure where the operator does something specific (tops up) and then wants the work to resume; an alert that can say "run those again" closes that loop where a queue page would be a new surface to build.