Skip to content

Suprnova v0.9.1

Choose a tag to compare

@eas4ai eas4ai released this 01 Aug 11:57
· 71 commits to main since this release

Three defects, all found by running the dogfood app under a containerised
harness rather than by reading the code. Every one of them is invisible to
a test suite that never stops a process the way production stops it.

They compound in a specific order: a rolling deploy SIGKILLs a worker
mid-job (the first), and that job then takes a reclaim path that never
counted the attempt (the second).

Fixed

  • schedule:work, queue:work and workflow:work ignored SIGTERM.
    Each selected on tokio::signal::ctrl_c() alone, which installs a
    SIGINT handler — so SIGTERM had no handler anywhere in the process, and
    SIGTERM is what docker stop, Coolify, systemd and Kubernetes send. All
    three already had a careful bounded drain behind that select!; none of
    it had ever executed under a supervisor. Measured before the fix: a
    docker stop on a queue:work container burned its whole 40s grace
    window and exited 137 with the in-flight job destroyed. As PID 1 — which
    is what a container runs — the kernel discards an unhandled SIGTERM
    outright, so the process did not die badly; it did not die at all until
    SIGKILL. Server::run already handled both signals correctly and its
    listener is now shared, which also closes a missed-signal window in the
    scheduler's loop.

  • A job that killed its worker could never be dead-lettered. A job
    whose handler fails is nacked and its attempt counted, so it
    dead-letters after max_tries. A job that kills its worker — OOM,
    abort, segfault, or the SIGKILL above — settles nothing; its reservation
    merely lapses, and every driver used to redeliver it byte-identical.
    Such a job is immortal: it kills each worker that claims it, comes back
    unchanged, and kills the next one, for as long as anything restarts
    workers. All three drivers now charge the attempt where they learn a
    worker died, because swapping QUEUE_DRIVER must not change whether a
    poison job can be stopped. attempts now means "deliveries to a worker"
    rather than "handler failures" — documented in manual/queues.md,
    because a worker lost for unrelated reasons burns an attempt too.

  • …and the exhausted job is now dead-lettered before it is dispatched.
    Counting the attempt was necessary and not sufficient. Every
    dead-letter decision lived in the worker's settlement path, which
    assumes the handler returns — so it never ran for exactly the jobs that
    could not return. With the driver fix alone the counter climbed
    (measured: 0 → 1 → 2 across three killed workers) and nothing acted on
    it. The budget is now spent before the handler runs. Caught only by
    re-running the container experiment after the first fix looked correct.

  • The daemons had no tracing subscriber. serve gets one from
    init_telemetry; queue:work, schedule:work, schedule:run and
    workflow:work come through a different boot path and got nothing, so
    every tracing:: line they emit went nowhere and LOG_LEVEL was inert
    for them. That is most of what they have to say — a worker
    dead-lettering a job, a scheduler skipping a tick it lost, a lock it
    could not release. In a container the only visible output was the
    startup banner, and the process looked idle while doing all of it. Two
    of the defects in this release were invisible until this was fixed.

  • A dead-letter with no failed-jobs store bound was a silent deletion.
    The persist step sat inside if let Some(store) = .., so with no store
    the arm did not match and execution fell through to the ack — quieter
    than the failure path directly above it, which at least leaves the
    reservation intact. An absent store was treated as more successful than
    a broken one. It now logs the full envelope at ERROR, because that is
    what queue:retry re-pushes: the difference between work recoverable by
    hand and work that ceased to exist.

  • QUEUE_DRIVER=database now binds a failed-jobs store. failed_jobs
    is part of that driver's contract — queue:retry reads it and
    Queue::retry_failed cannot work without it — but bootstrap_from_env
    wired the driver and left the store unset, so a database-backed queue
    dead-lettered into nothing unless the app bound one by hand. Configurable
    via QUEUE_FAILED_DB_TABLE. Only for this driver: memory is ephemeral
    by construction and redis has no table to write to.

  • Redis reclaim latency now follows --visibility-timeout. The flag
    sets XAUTOCLAIM's idle threshold, but a separate clock governs how often
    a consumer looks, and the driver left it at sea-streamer's 30s default —
    so --visibility-timeout 5 really meant "up to 35 seconds". The
    interval now tracks the configured timeout, clamped to 1s..=30s so a
    short timeout cannot become an XAUTOCLAIM storm and a long one can only
    make reclaim faster than before.

Added

  • TaskBuilder::on_one_server() / on_one_server_for(ttl) — run a
    scheduled task exactly once per due tick across replicas. Without it
    nothing elects a leader for a tick: each schedule:work process
    evaluates the schedule independently, and three replicas were measured
    running every due task three times, every minute, with no variance. A
    nightly billing job on three replicas billed every customer three times.

    without_overlapping() does not cover this and cannot: its lock is
    keyed on the task and released when the handler returns, so a fast task
    frees it before a second replica looks. on_one_server keys on the task
    and the tick and holds the lock past the handler, letting it expire on
    TTL. The two compose.

    Opt-in, matching Laravel. Diverges from Laravel in failing closed: the
    election is only as shared as the cache behind it, so a production boot
    with CACHE_DRIVER=memory and a single-server task is refused, naming
    the offending tasks, with SCHEDULE_ALLOW_MEMORY_LOCK_IN_PRODUCTION=true
    for deployments that genuinely run one scheduler.

Changed

  • manual/deployment.md no longer says "run exactly one schedule:work
    process" as the only option, and gains a Stopping cleanly section
    covering the drain windows per subsystem, how to size a platform's
    termination grace above them, and why PID 1 makes a missing signal
    handler worse than it sounds.