Suprnova v0.9.1
Three defects, all found by running the dogfood app under a containerised
harness rather than by reading the code. Every one of them is invisible to
a test suite that never stops a process the way production stops it.
They compound in a specific order: a rolling deploy SIGKILLs a worker
mid-job (the first), and that job then takes a reclaim path that never
counted the attempt (the second).
Fixed
-
schedule:work,queue:workandworkflow:workignored SIGTERM.
Each selected ontokio::signal::ctrl_c()alone, which installs a
SIGINT handler — so SIGTERM had no handler anywhere in the process, and
SIGTERM is whatdocker stop, Coolify, systemd and Kubernetes send. All
three already had a careful bounded drain behind thatselect!; none of
it had ever executed under a supervisor. Measured before the fix: a
docker stopon aqueue:workcontainer burned its whole 40s grace
window and exited 137 with the in-flight job destroyed. As PID 1 — which
is what a container runs — the kernel discards an unhandled SIGTERM
outright, so the process did not die badly; it did not die at all until
SIGKILL.Server::runalready handled both signals correctly and its
listener is now shared, which also closes a missed-signal window in the
scheduler's loop. -
A job that killed its worker could never be dead-lettered. A job
whose handler fails is nacked and its attempt counted, so it
dead-letters aftermax_tries. A job that kills its worker — OOM,
abort, segfault, or the SIGKILL above — settles nothing; its reservation
merely lapses, and every driver used to redeliver it byte-identical.
Such a job is immortal: it kills each worker that claims it, comes back
unchanged, and kills the next one, for as long as anything restarts
workers. All three drivers now charge the attempt where they learn a
worker died, because swappingQUEUE_DRIVERmust not change whether a
poison job can be stopped.attemptsnow means "deliveries to a worker"
rather than "handler failures" — documented inmanual/queues.md,
because a worker lost for unrelated reasons burns an attempt too. -
…and the exhausted job is now dead-lettered before it is dispatched.
Counting the attempt was necessary and not sufficient. Every
dead-letter decision lived in the worker's settlement path, which
assumes the handler returns — so it never ran for exactly the jobs that
could not return. With the driver fix alone the counter climbed
(measured: 0 → 1 → 2 across three killed workers) and nothing acted on
it. The budget is now spent before the handler runs. Caught only by
re-running the container experiment after the first fix looked correct. -
The daemons had no tracing subscriber.
servegets one from
init_telemetry;queue:work,schedule:work,schedule:runand
workflow:workcome through a different boot path and got nothing, so
everytracing::line they emit went nowhere andLOG_LEVELwas inert
for them. That is most of what they have to say — a worker
dead-lettering a job, a scheduler skipping a tick it lost, a lock it
could not release. In a container the only visible output was the
startup banner, and the process looked idle while doing all of it. Two
of the defects in this release were invisible until this was fixed. -
A dead-letter with no failed-jobs store bound was a silent deletion.
The persist step sat insideif let Some(store) = .., so with no store
the arm did not match and execution fell through to the ack — quieter
than the failure path directly above it, which at least leaves the
reservation intact. An absent store was treated as more successful than
a broken one. It now logs the full envelope at ERROR, because that is
whatqueue:retryre-pushes: the difference between work recoverable by
hand and work that ceased to exist. -
QUEUE_DRIVER=databasenow binds a failed-jobs store.failed_jobs
is part of that driver's contract —queue:retryreads it and
Queue::retry_failedcannot work without it — butbootstrap_from_env
wired the driver and left the store unset, so a database-backed queue
dead-lettered into nothing unless the app bound one by hand. Configurable
viaQUEUE_FAILED_DB_TABLE. Only for this driver:memoryis ephemeral
by construction andredishas no table to write to. -
Redis reclaim latency now follows
--visibility-timeout. The flag
sets XAUTOCLAIM's idle threshold, but a separate clock governs how often
a consumer looks, and the driver left it at sea-streamer's 30s default —
so--visibility-timeout 5really meant "up to 35 seconds". The
interval now tracks the configured timeout, clamped to 1s..=30s so a
short timeout cannot become an XAUTOCLAIM storm and a long one can only
make reclaim faster than before.
Added
-
TaskBuilder::on_one_server()/on_one_server_for(ttl)— run a
scheduled task exactly once per due tick across replicas. Without it
nothing elects a leader for a tick: eachschedule:workprocess
evaluates the schedule independently, and three replicas were measured
running every due task three times, every minute, with no variance. A
nightly billing job on three replicas billed every customer three times.without_overlapping()does not cover this and cannot: its lock is
keyed on the task and released when the handler returns, so a fast task
frees it before a second replica looks.on_one_serverkeys on the task
and the tick and holds the lock past the handler, letting it expire on
TTL. The two compose.Opt-in, matching Laravel. Diverges from Laravel in failing closed: the
election is only as shared as the cache behind it, so a production boot
withCACHE_DRIVER=memoryand a single-server task is refused, naming
the offending tasks, withSCHEDULE_ALLOW_MEMORY_LOCK_IN_PRODUCTION=true
for deployments that genuinely run one scheduler.
Changed
manual/deployment.mdno longer says "run exactly oneschedule:work
process" as the only option, and gains a Stopping cleanly section
covering the drain windows per subsystem, how to size a platform's
termination grace above them, and why PID 1 makes a missing signal
handler worse than it sounds.