Skip to content

wurk v1.10.0

Choose a tag to compare

@github-actions github-actions released this 03 Nov 00:57
b68f355

Upgrade if you run on Dragonfly. 1.9.0 and earlier could lose jobs on a Dragonfly at maxmemory, on Redis-compatible paths that are safe on Redis and Valkey. This release also closes several other ways to lose or double-run jobs, wraps every job in the Rails reloader, and makes more of the Sidekiq gem ecosystem work. Read Changed before upgrading: jobs now run with the query cache on, a connection pool smaller than concurrency can raise under contention, Sentry gets store errors it used to filter, and wurkswarm exits after sustained crash-looping. No Redis key, job JSON field or score format changed.

Added

  • Queues nothing listens to are now visible. The leader logs one WARN an hour per non-empty queue with no live subscriber, after 5 minutes in that state, so a rolling restart stays quiet. New metric wurk_queue_consumers{queue=}, a "No consumer" badge on the dashboard Queues page, and Wurk::Testing.unconsumed_job_queues(listened:|config_file:) for a guard test. GET /api/queues gains a consumers field. No new Redis keys.
  • Wurk::Sentry.install!(config, report_after_attempts: N) also reports attempt N, on top of the terminal failure.
  • Dragonfly >= 1.34 moves to "works in production; tested in CI" in docs/deployment.md, with configuration requirements and durability limits (RPO is the snapshot cadence). New non-required compat CI leg on v1.34.0 with stock Lua settings.
  • New examples/kubernetes/: rolling and Recreate workers, Service + ServiceMonitor + NetworkPolicy, PrometheusRule with an absent() alert. A unit test keeps every metric in them in step with the exporter.
  • docs/deployment.md: WURK_HEALTH_PORT, Recreate variant, scraping and alerting, fiber servers, host-shutdown grace, corrected reclaim timing after SIGKILL.
  • The demo turns on the health listener by default.
  • mgr.load_hash, mgr.load_file and mgr.authoritative! on config.periodic. The loaders take sidekiq-cron's shape (cron, class, queue, args, retry, tz, status). authoritative! is opt-in and deletes every loop the block did not declare.
  • wurk health: an exec-style liveness probe. It boots no app and exits 0 when a process on this host has a heartbeat within --window seconds (default 60; --min COUNT). It reads REDIS_URL or --redis-url. A Redis it cannot reach passes unless --strict. wurk --help now lists it.
  • WURK_HEALTH_PORT turns on the health listener from the environment, so a deployment manifest can add a probe with no app change. Off by default. config.health_check wins when both are set. A non-numeric value raises at boot.
  • Metrics wurk_leader, wurk_fetch_errors_total, wurk_reaper_reclaimed_total. The two counters are per process, so under a swarm they sample the child that owns the port.
  • Boot log names the backend and version (Wurk 1.9.0 on dragonfly 1.34.0), read from INFO, with a WARN below the oldest version tested. Nothing branches on the backend.
  • Boot WARNs under wurk / wurkswarm: live processes entries that are not Wurk, and loaded job classes that set lock: while sidekiq-unique-jobs is not loaded (the option does nothing, so duplicates run).

Changed

  • Wurk::Sentry now reports what the server answered with. The default filter narrowed from RedisClient::Error to RedisClient::ConnectionError + ConnectionPool::TimeoutError. MISCONF, OOM, READONLY, MASTERDOWN, NOPERM, WRONGPASS and other CommandErrors are reported, rate-limited per code per process (first, then one a minute, with extra[:suppressed]). Upgrade note: expect new Sentry events for these. Passing filtered_error_classes: still replaces the list.
  • Wurk::Sentry reports deaths it could not predict. install! adds a deduped death handler for poison-pill kills and sidekiq_retry_in answering :kill / :discard. Those were silent before. It also adds Wurk::JobRetry::Handled and Wurk::Shutdown to Sentry's excluded_exceptions, including a later Sentry.init.
  • wurkswarm exits 1 when every slot has crash-looped for 10 minutes (config.swarm_crash_exit_after, default 600; nil disables). A slot's streak is unbroken crashes within 60s of boot. It reports Wurk::Swarm::CrashLoop through error_handlers first. Upgrade note: a release that cannot boot now ends the container instead of leaving a Running pod that processes nothing. A crash from an unreachable store or one that cannot serve yet (RedisClient::ConnectionError, pool timeout, LOADING, BUSY, CLUSTERDOWN, MASTERDOWN, READONLY, OOM / ERR Out of memory, MISCONF) does not count, so an outage alone never ends the swarm. A bad WURK_HEALTH_PORT now fails the parent's boot. A swarm co-hosted in a web process only reports, once per window, and keeps running.
  • Fetch-loop errors reach error_handlers at most once a minute per exception class per process, with context[:suppressed]. Any handler benefits. The default handler's log volume for this path drops too. wurk_fetch_errors_total still counts every error.
  • The swarm child no longer calls ActiveRecord::Base.establish_connection. Rails' fork hook already rebuilds the pools from the parent's config. The bare call re-derived config from DATABASE_URL, silently switching a host's custom database or pool. Upgrade note: a host relying on the child reading DATABASE_URL afresh, which differs from what the parent established, now gets the parent's config.
  • An ActiveRecord pool smaller than concurrency logs at ERROR, was WARN.
  • wurkswarm loads a Rails target's config/application.rb before the WURK_PRELOAD group Bundler.require, so gems that expect Rails at require time load. This applies to -r <app dir> and -r config/environment.rb. The latter now also defines the ActiveJob adapter first.
  • A web process that forks a swarm in production logs a WARN unless WURK_EMBED=1. The default is unchanged.
  • Upgrading: reclaim and dead-set behaviour. A job that previously looped now lands in the dead set. Watch dead-set growth after upgrading if you had such payloads: they were pegging a thread before. Fetcher::PrivateListKey.parse_owner is gone (internal, nothing else used it). Fetcher::OrphanGrace has a new constructor (internal). Reclaim after a host loss or pod restart is slower (see the figures), on purpose. super_fetch:reaper:full is no longer written and expires on its own. Reaper#reclaim_full!, FULL_INTERVAL, FULL_LOCK_KEY and full_interval:/full_lock_key: are gone (internal).
  • Known issue, Dragonfly at maxmemory: on 1.34 and 2.0, between shards, LMOVE/BLMOVE on the fetch path can pop a job and drop the push, so a job runs with no private copy, and is gone if the process then dies or its retry is refused (2 of 16 in a test on 2.0.2). Do not let a Dragonfly job store reach maxmemory: alert well below it. See docs/reliability.md.
  • Under Rails, every job now runs inside Rails.application.reloader, in every boot mode (railtie swarm, wurk, wurkswarm, embedded). Before this, nothing wired it, despite the docs saying otherwise: ActiveSupport::CurrentAttributes set by one job was visible to the next on the same thread, a raw ActiveRecord::Base.connection stayed leased after the job, the query cache was off and development never reloaded code. Check these when upgrading: the query cache is now on inside a job, so a polling loop can read stale rows; a pool smaller than concurrency now waits and then raises ActiveRecord::ConnectionTimeoutError under contention rather than having threads hold on to connections; Rails.error subscribers now see job failures tagged job.sidekiq, including Wurk::JobRetry::Skip and Handled (spec §14 places the retrier inside the reloader), so filter those in any error tracker subscribed to Rails.error; in development, reload waits for running jobs. Set config[:reloader] = nil to opt out.
  • Under the railtie, workers start after all host after_initialize hooks. Boot moved to a last Finisher initializer, so a Redis URL set in an after_initialize hook now applies, and a forked child has the routes and reload hooks Rails installs after those hooks fire. The relative boot order is unchanged.
  • Sidekiq::Worker and Sidekiq::Job are the same module, as they are upstream, so is_a? and prepend work on either name.
  • The release lane separates what runs third-party code from what can publish. build runs the bundle, bun install and the suite with no credential; publish alone holds id-token: write and runs only pinned actions; release alone holds contents: write. The unpinned gem exec rubygems-await is gone.
  • A release that fails after gem push can be finished by a re-run. Preflight reads the gem, the tag and the GitHub Release, and a live gem with the rest missing skips the push and finishes, instead of reporting green having done nothing. A tag on another commit is now refused before the push. RELEASE.md has the full table.
  • deploy-demo proves the built commit is serving. The image carries its commit, the demo reports it at /demo/build, and a frozen rollout fails the run. It used to compare gem versions, which matched on the first attempt on any push without a version bump.
  • The demo's dependencies are patched and watched. demo/Gemfile.lock moves off advisories in rack, crass, sqlite3, loofah, nokogiri and rails-html-sanitizer; Dependabot covers /demo, and a weekly non-required audit workflow runs bundle-audit and bun audit.

Fixed

  • A limiter built at class load no longer exits the process when Redis is down, LOADING or OOM. Registration is deferred to its next operation, with a WARN.
  • A failed requeue can no longer lose a job on Dragonfly. Dragonfly at maxmemory does not abort a MULTI or a script on a refused write, so UnitOfWork#requeue, bulk_requeue and the self-reconcile applied the LREM, were refused the RPUSH, and the job was in no list. The moves now push first. A failing job on a full Dragonfly stays in the private list. Affects 1.9.0 and earlier on Dragonfly; Redis and Valkey were safe.
  • The reaper no longer loses jobs on a full Dragonfly. Its LMOVE between shards could pop the job and drop the push: 5-6 of 24 orphans gone in one sweep on 1.34.0 and 2.0.2. It moves through a push-first script, with the same effect as LMOVE RIGHT RIGHT.
  • The scheduler's burial of an unpromotable job, and a debounce replacement, no longer drop the job on a refused write. Add first, then remove.
  • A dashboard retry refused at maxmemory puts the job back. ZREM is not denyoom; the push and the restoring ZADD were, so the entry was lost, on Redis too. Past that, on Dragonfly, the payload is written to the error log.
  • The reaper keeps working at maxmemory on Redis. Its sweep lock's SET NX was refused, so no periodic sweep ran, and a refused poison counter ended each drain after one job. A job moved while Redis is full is not poison-counted.
  • A live worker's jobs are no longer run twice after a Redis restart from an older snapshot, or when CPU-bound jobs hold back its heartbeat. Every heartbeat-judged owner now waits out a 30s grace (it was 15s, and only some owners). Reclaim after losing a host goes up to 90-141s; a swarm child stays at 1.2s.
  • A dead owner's jobs come back within a sweep whichever queue they belong to. A sweep was scoped to what the lock's winner served, so a queue served by 1 process of 10 waited about 13 intervals. Every sweep now covers every list, and the hourly full sweep is gone.
  • A sole survivor sweeps every interval, not every second one: the lock outlived the tick (180s became 120s, before the grace).
  • A host bug in a failure hook no longer kills a healthy job. A raising backtrace_cleaner, a sidekiq_retry_in result that cannot convert, a death handler raising past StandardError and a raising error reporter all raised out of retry booking, and the Processor sent the job to the dead set on its first failure.
  • A job whose claim reply was lost no longer waits for the process to restart. A Redis restart, a stall past the 2.5s read timeout or a reset socket left up to one job per processor thread in a live worker's private list, silently, until that process exited. A job that failed while Redis was down, with both its retry booking and its requeue refused, ended up in the same place. Every worker now checks its own lists on each reaper tick and puts back anything no thread and no pending ACK accounts for, once it has been seen stranded twice at least 30s apart. Such a job runs up to two reaper intervals late. A running job is never touched. A failed requeue now also logs at ERROR.
  • A full Redis no longer stops workers draining it. LMOVE is refused at maxmemory, so under noeviction a backlog that filled Redis stopped every worker for good. Workers now claim through an allow-oom script, still through the private list, and retry and dead-set booking from a failing job work the same way. Redis 7.0+ and Valkey. Dragonfly has no such header and keeps its own behaviour. Queues under global_concurrency are not claimed on this path. An error report fires once a minute per fetcher while it lasts.
  • A payload that fails the same way every time no longer hot-loops. Valid JSON that is not an object ([1,2], 123, true, null) now goes to the dead set. A job that cannot be booked for a reason other than Redis (a retry_count that is not a number, args that cannot be re-serialized) runs once and goes to the dead set with death handlers fired, instead of being requeued at up to ~560 re-runs of perform per second. A Redis or pool error while booking still requeues the job, now after a 1s pause.
  • Orphan reclaim no longer depends on OBJECT IDLETIME. Reading a private list (a monitor's LLEN, our own runbook) reset that clock, so a polled dead owner's list was never reclaimed. The fallback on servers without OBJECT only advanced in a process that won the sweep lock twice, so reclaim got slower as a fleet grew. Reclaim now takes the same time on every server and fleet size.
  • The reaper's sweep scans the keyspace once, not once per served queue: 199,300 SCAN calls and 43s per sweep at 100 queues and 200k keys became about 200.
  • An idle capsule serving many queues stops paying per thread for the same empty answer. One empty fetch_first pass now answers for the whole fetcher for 250ms. At 1000 queues and concurrency 500, the idle cost was 70% of a worker core and 26% of a Redis core.
  • Thread.exit in a job no longer loses a processor thread for the life of the process. The Manager is told and replaces it; the job is recovered by the reconcile above, with poison-pill counting.
  • UnitOfWork#requeue no longer double-pushes on a lost reply. On that lost reply it now raises rather than guessing; the job is either moved or still in the private list, and both are recovered.
  • The default scheduler no longer loses jobs. Scheduled::Enq claims due members (rescoring them to a 60s lease) instead of popping them, pushes through the client chain as before, then removes them. A store out of memory, a lost reply on a stalled socket or a kill -9 between the steps now costs a re-promotion when the lease lapses. Measured: a store at maxmemory lost 50 of 50 due jobs before and 0 now; 40,000 jobs through four 4s stalls lost 4 silently before and 0 now, on real Redis. Client middleware still runs on promotion, and promotion got 25-60% faster depending on load (reviewer about 25% under load; mine 55-60% solo and about 55-75% with four redis-benchmark loaders), because the removals share a round trip per batch of 10.
    • Upgrade notes: failures that used to lose a job can now run it twice, so a job should tolerate a duplicate (already the documented contract). A claimed member's score briefly reads about 60s in the future: visible on the dashboard for milliseconds normally, and up to the lease after a crash. A starved local connection pool now leaves the member in its set; it used to send it to the dead set.
  • reliable_push no longer blocks callers during an outage. The first failed push opens a circuit for that Redis: later pushes buffer without touching the socket (under 1ms against a stopped server, where each previously blocked 6s), and a background drainer probes and replays in FIFO order.
    • Upgrade notes: a process using reliable_push! now starts the drainer thread on its first failed push, even without reliable_push_drainer. While a circuit is open the drainer probes about every 0.25s, and buffered jobs replay then, not on the very next push. A clean exit flushes what is left, bounded at 5s, and logs an ERROR for what it could not deliver; a SIGKILL in that window loses those pushes. A batched push still tries Redis and raises.
    • Upgrade note: with no caller blocked by the outage, producers run at full speed, so the 1000-entry buffer fills far faster and starts dropping the oldest jobs sooner (46,000 pushes in 5s from 20 threads, measured). Raise reliable_push_buffer, or set reliable_push_overflow = :raise, if an outage must not drop work.
  • With reliable_push!, the default scheduler never counts a buffered push as delivered: promotions reach Redis or stay in their set. The scheduler bypasses the buffer and circuit.
  • Scheduler first sweep is jittered: 10-15s instead of exactly 10s, so a fleet booted together does not burst. An explicit config[:scheduler_initial_wait] is still taken exactly.
  • RELIABLE_SCHEDULE_PROMOTE carries a --!df flags=allow-undeclared-keys header, so reliable_scheduler! works on stock Dragonfly. The script's SHA changes; Redis and Valkey read the line as a comment.
  • The heartbeat keeps working at maxmemory. On a refused beat it falls back to one allow-oom script that writes beat, the :work mirror and the expiry, and refreshes global-concurrency holds the same way. Before, a worker with every thread in a long job stood still and was TERMed after 60s of OOM, and holds could lapse so a cap over-admitted. On a server that refuses the script (Dragonfly) the failure is as before. On Redis and Valkey the :work mirror of running jobs now stays in place while a beat is refused, because UNLINK and its rewrite are one transaction (2 more commands in the beat pipeline, only when jobs are running); a full Dragonfly still erases it, as before.
  • GET /live stays 200 on a quiet (TSTP) process, and answers 503 only once it is shutting down. With /live as the livenessProbe, a dashboard quiet got the pod restarted, and the replacement fetched again. Upgrade note: anything relying on /live to detect quiet should read wurk_process_quiet.
  • A crash in a swarm child outside a lifecycle event reaches error_handlers (context: 'swarm-child-boot'). It only reached the log. Crash streaks of 3+ log at ERROR.
  • A respawn no longer raises ConnectionPool::PoolShuttingDownError in host threads. The fork path of a co-hosted swarm terminally shut pools the web process was using. It now closes idle sockets and leaves the pools usable. A socket checked out at that instant is inherited as before, unused by the child.
  • Launcher#stop is bounded when Redis hangs. Redis-touching teardown shares one budget ending 4s past timeout (minimum 1s). Past it the process moves on; the heartbeat expires and the leader lock times out as after a SIGKILL. No job is at stake: managers have already drained. exe/wurk could overrun timeout by a minute.
  • A late config.redis= logs a WARN when a pool is already built, since it silently does not apply to live connections.
  • An unweighted queue in a weighted list is consumed. [["a", 2], "b"] silently never fetched b: its jobs sat in the queue with no error. An all-zero weight list now means unweighted instead of raising.
  • sidekiq-cron fires in a real worker. Wurk's native periodic poller overwrote the gem's @cron_poller, so the gem's suite stayed green while no entry ever ticked. Under wurkswarm the gem also crash-looped each child with FrozenError, and under the railtie its load-time configure_server block was dropped. bin/test-ecosystem now boots a real worker for this gem, under both sidekiq and wurkswarm, and asserts ticks.
  • perform_in, perform_at and set(wait:) accept ActiveSupport::Duration, and honour frozen time. perform_in(5.minutes) raised ArgumentError. Intervals now read Time.now, so freeze_time and travel_to apply.
  • sidekiq-unique-jobs boots and enqueues. Config#redis_info and #to_json were missing, which stopped the server at startup and raised on each locked enqueue.
  • retry, kill, add_to_queue and retry_all no longer destroy jobs when a gem prepends delete_by_value. JobSet#remove_job went through that public method and read the gem's return value as "not removed", so the job was gone with nothing pushed. It also yields the raw JSON string now, as upstream does.
  • sidekiq-failures no longer breaks set iteration. SortedEntry parses on construction, so a gem reading @item in initialize works. This costs 2.5–10µs per entry only on walks that read no field. A set member that is valid JSON but not an object, or holds invalid bytes, reads as an empty job, so retry_all and kill_all no longer raise on it.
  • Sidekiq::Work#raw, #[] and #dig exist, for code written against Sidekiq 7.
  • bundle exec sidekiq works: the shim gem ships a sidekiq executable.
  • MyJob.jobs.clear clears, and Sidekiq::Rails::Reloader resolves.
  • perform_inline no longer reports a spurious error. The top-level middleware chains had no config, so Metrics::History raised and error handlers saw undefined method 'redis_pool' for nil.
  • sidekiq_options stringifies two key levels, and sidekiq_options_hash= exists, as upstream.
  • CurrentAttributes.persist("Name") works. The String form raised on every perform_async. Load now sits outermost on the server chain, and a payload with nothing to persist carries no cattr key.
  • ProcessSet reads a stock Sidekiq process correctly. A nil HMGET value overrode the info JSON and gave concurrency=0.
  • The reaper's sweep loop can no longer be lost, and no longer dies with its own error report. Two holes in that loop's lifetime let a sweep thread end for good, silently — the thread sets report_on_exception = false, so an exception escaping it leaves no trace, and orphans are then never reclaimed again for the life of the process. start handed back the reference to a previous loop rather than the loop itself: a stop whose bounded join timed out leaves its straggler referenced, and once that straggler had wound down, a restart returned the dead thread and spawned nothing, so the reaper was dead for good while start looked like it had worked. start now reuses a live loop and puts a fresh one where a dead reference sits, so "no second sweep loop alongside a straggler" still holds while the straggler is alive. And the sweep's error report ran inside the tick's rescue, so a reporter that raised (a logger that cannot write, an error handler that raises) escaped the raw loop thread and killed it; the report is guarded now, so the loop survives its own reporter the way it already survives a Redis blip (K1). The CI red this answers is run 41454136586, job "compat (ruby 4.0, rails 8.1, valkey)" (job 125306502572): the failing step is "rake test" — bundle exec rake test, the same full suite (unit + integration + engine) the required test job runs — which ran 13:35:44Z to 13:41:22Z and failed, and the check's only diagnostic is Process completed with exit code 1. The actions/cache and bundler-cache restore lines in that job are setup warnings, not the failure: every setup step of it (checkout, "resolve store port", setup-ruby with bundler-cache: true, setup-bun, setup-node, "build dashboard SPA", "dummy app setup") concluded success, and "rake test" is the only failing step. The job's log body is not readable from a checkout without an authenticated GitHub token (the job-logs endpoint answers HTTP 403), so the failing test and assertion are not quoted from it; what the public API does show is that the leg's store did not float — valkey/valkey:8.1 has resolved to 8.1.11 since 2026-10-27, the image this red run and that leg's last green one (41303800609) both used — and that the reaper's own unit suite is green against that image (13 consecutive runs, no failures). Commit e01987ad485b39b4bb5bec6a2fa301a8fe9ed1d8 did not introduce the failure: its diff touches one file, .dz/briefs/improve.md (+14/-0), a docs brief with no Ruby, CI or test in it; the only commit between that leg's last green run and this red one that changed runtime code is 4684178 (#869), which changed the idle-connection reaper's wait in lib/wurk/redis_pool.rb and the wait idiom in two test files. The two regression cases live in test/unit/fetcher_reaper_test.rb and fail on the unpatched tree ("a dead straggler is not a sweep loop", and the raising reporter escaping tick_once through run_loop). The divergence ledger's two reaper anchors moved with the code and were re-pinned with it — docs/idea/parity-divergences.md now names lib/wurk/fetcher/reaper.rb:262-291,300-310, and the pin records in test/conformance/ledger_anchors/client-fetch.yml with it — because the conformance gate rejects a pin that no longer names the code it describes. The reaper's unit suite and that ledger-anchor gate are green against a local Valkey 8.1.11 and a local Redis 7.4.11. Nothing on the wire moves: no Redis key, job-JSON field or score format changed.
  • The idle-connection reaper's timed wait is the repo's Mutex+ConditionVariable again, and the compat cell (ruby 3.2.11 / rails 8.1 / redis) is green. The reaper's sweep waited on Thread::Queue#pop(timeout:) — the one timed wait in lib/ that did not use the Mutex+ConditionVariable idiom the tree's periodic, stop-signalled loops use (lib/wurk/client/buffered.rb, lib/wurk/timer_loop.rb) — and it ran that wait inside a raw Thread.new, so a wait that raised there took the sweep thread down with it (the thread exceptions in CI run 41303800609, job "compat (ruby 3.2.11, rails 8.1, redis)"). The failing assertion on that leg was one of the two that wait the same way: "the sweep after the failing one still ran" (test/unit/fetcher_reaper_test.rb) and "the job ran within one fetch cycle" (test/integration/fetch_many_queues_test.rb). The reaper now waits on a Mutex+ConditionVariable pair: it still sweeps every min(timeout / 2, 10) seconds, and stop still wakes the sweep at once, so the reaper's own cases — an idle connection reaped on the interval, a stop that takes effect at once — hold unchanged. Both cases take their value with the same idiom, so their assertions run on the support floor. Nothing on the wire moves: no Redis key, job-JSON field or score format changed.
  • A failed USR2 log reopen is reported instead of swallowed. Wurk::CLI, the swarm parent and each forked child reopened their logger on USR2 (logrotate) and discarded any exception, so a reopen that failed (permissions, full disk, a closed stream) left the process writing to the rotated, possibly deleted, file with no sign of it. The failure now goes to stderr — not to the logger, which is what just failed — naming the exception; signal handling carries on as before.
  • A malformed dashboard-extension locale file is logged, not silently dropped. An extension's locales/en.yml that failed to parse was replaced by an empty Hash, so its strings fell back to their keys with no clue why. The file is still skipped (the dashboard keeps serving), and Wurk.logger now warns once with the file path and the parse error.