Skip to content

feat(sweeper): Phase 2D — Orphan revival and delayed-retry promotion - #4

Merged
smallchungus merged 4 commits into
mainfrom
phase-2d-sweeper
Apr 18, 2026
Merged

feat(sweeper): Phase 2D — Orphan revival and delayed-retry promotion#4
smallchungus merged 4 commits into
mainfrom
phase-2d-sweeper

Conversation

@smallchungus

Copy link
Copy Markdown
Owner

Summary

Phase 2D: the durability loop.

  • internal/sweeper.SweepOnce does two passes:
    1. Orphan revival: finds status=running jobs whose worker_id has no live heartbeat (EXISTS heartbeat:<id> returns 0), calls MarkFailed with "worker died". Status goes back to queued (or dead if max attempts hit).
    2. Delayed-retry promotion: finds status=queued AND last_error IS NOT NULL AND next_run_at <= now() jobs and pushes their IDs onto the Redis stage queue. The last_error IS NOT NULL filter avoids re-pushing newly-enqueued jobs whose producer hasn't yet pushed them.
  • New queue.IsWorkerAlive(workerID) reader (paired with the Heartbeat writer from 2C).
  • New store.ListRunningJobs and store.ListReadyRetryJobs plus shared scanJobs helper.
  • cmd/sweeper binary: 5 s tick, signal-handled graceful shutdown, slog JSON.

6 new tests (2 queue + 2 store + 4 sweeper integration). Existing 22 integration tests still green.

Closes the durability loop. Combined with Phase 2C, a worker that crashes mid-job is recovered automatically; failed jobs back off correctly.

Test plan

  • make lint (0 issues)
  • make test-unit
  • make test-integration (sweeper, worker, queue, store, testutil, api — all green)
  • make k8s-validate (8/8 valid)
  • cmd/sweeper smoke run with docker compose stack — clean start/stop on SIGTERM
  • CI green

@smallchungus
smallchungus merged commit bb494f5 into main Apr 18, 2026
3 checks passed
@smallchungus
smallchungus deleted the phase-2d-sweeper branch April 18, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant