Skip to content

fix: hand work back properly, and let go of what is gone - #138

Merged
thedancingdeveloper merged 1 commit into
mainfrom
fix/queue-lifecycle
Aug 3, 2026
Merged

fix: hand work back properly, and let go of what is gone#138
thedancingdeveloper merged 1 commit into
mainfrom
fix/queue-lifecycle

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Contributor

Closes #126. Closes #104. Closes #107.

#126 — retry was a no-op on the state most likely to prompt it

POST /api/work/{id}/retry called release(item_id, PENDING, error=None).
Two consequences neither intended nor visible:

  • attempts untouched, so an item at max_attempts was retired again by the
    next claim scan before any worker saw it. Observed live: exhausted → pending → exhausted in seconds, with nothing having run.
  • error=None wiped last_error — the only record of why it failed, and what
    the retirement message appends to itself. So the retry downgraded a
    diagnosable failure to gave up after N attempts with no cause.

WorkQueue.requeue resets the counter and keeps the error.

#104 — a claim held by a process that no longer exists

The lease is slow on purpose: a slow worker must not be evicted. But a worker
killed with its lease running leaves an item claimed by a pid that is gone,
unavailable to healthy workers, while new items dispatch around it.

reclaim_dead_workers runs at pool start and releases those claims. Bounded
carefully:

  • only claims owned by this host — a pid elsewhere says nothing;
  • os.kill(pid, 0), where PermissionError counts as alive;
  • any unexpected OSError is treated as alive, because releasing a live
    worker's item is far worse than waiting out a lease.

Tests cover the dead pid, a live pid, and another host.

#107 — a corrected graph did not stop work already in flight

claim checks dependencies once. Correcting a plan while an item is running
is a normal operator action, and the item went on through review, commit and
push on the strength of a check made minutes earlier.

unmet_dependencies is now re-checked at the last cheap point before the
reviewer is paid, and an invalidated item is emitted as
dependency_invalidated and returned to pending with its branch intact. No
agent is killed mid-item, per the standing rule.

Suite, ruff and mypy . green.

Three ways the queue mishandled a claim.

**Retry actually retries (#126).** The route called `release(..., PENDING,
error=None)`, which left `attempts` at the ceiling and wiped `last_error`. So
retrying an exhausted item put it back to pending, the next claim scan retired
it again before any worker saw it, and the reason it failed was destroyed on
the way -- while the call reported `{"ok": true}`. `requeue` resets the
counter and keeps the error.

**A dead worker's claim is reclaimed (#104).** The lease is deliberately slow
so a merely slow worker is not evicted, but after a pool restart the old
worker is provably gone: its pid is not running. Waiting out its lease left an
item stuck alongside newly dispatched work, with no session and nothing saying
why. `reclaim_dead_workers` runs before a pool starts, and only for claims
owned by this host -- a pid on another machine says nothing about whether it
is alive.

**A corrected graph stops work in flight (#107).** `claim` checked
dependencies once; an operator correcting a plan minutes later did not stop an
item that was no longer eligible from passing review, commit and push. The
check is repeated at the last cheap point before review spends money, and the
item goes back to pending with its branch intact. Nothing is killed mid-item.
@thedancingdeveloper
thedancingdeveloper merged commit 4fba501 into main Aug 3, 2026
2 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the fix/queue-lifecycle branch August 3, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant