Repository navigation
wurk v1.10.0
Upgrade if you run on Dragonfly. 1.9.0 and earlier could lose jobs on a Dragonfly at maxmemory, on Redis-compatible paths that are safe on Redis and Valkey. This release also closes several other ways to lose or double-run jobs, wraps every job in the Rails reloader, and makes more of the Sidekiq gem ecosystem work. Read Changed before upgrading: jobs now run with the query cache on, a connection pool smaller than concurrency can raise under contention, Sentry gets store errors it used to filter, and wurkswarm exits after sustained crash-looping. No Redis key, job JSON field or score format changed.
Added
- Queues nothing listens to are now visible. The leader logs one WARN an hour per non-empty queue with no live subscriber, after 5 minutes in that state, so a rolling restart stays quiet. New metric
wurk_queue_consumers{queue=}, a "No consumer" badge on the dashboard Queues page, andWurk::Testing.unconsumed_job_queues(listened:|config_file:)for a guard test.GET /api/queuesgains aconsumersfield. No new Redis keys. Wurk::Sentry.install!(config, report_after_attempts: N)also reports attempt N, on top of the terminal failure.- Dragonfly >= 1.34 moves to "works in production; tested in CI" in
docs/deployment.md, with configuration requirements and durability limits (RPO is the snapshot cadence). New non-requiredcompatCI leg on v1.34.0 with stock Lua settings. - New
examples/kubernetes/: rolling and Recreate workers, Service + ServiceMonitor + NetworkPolicy, PrometheusRule with anabsent()alert. A unit test keeps every metric in them in step with the exporter. docs/deployment.md:WURK_HEALTH_PORT, Recreate variant, scraping and alerting, fiber servers, host-shutdown grace, corrected reclaim timing afterSIGKILL.- The demo turns on the health listener by default.
mgr.load_hash,mgr.load_fileandmgr.authoritative!onconfig.periodic. The loaders take sidekiq-cron's shape (cron,class,queue,args,retry,tz,status).authoritative!is opt-in and deletes every loop the block did not declare.wurk health: an exec-style liveness probe. It boots no app and exits 0 when a process on this host has a heartbeat within--windowseconds (default 60;--min COUNT). It readsREDIS_URLor--redis-url. A Redis it cannot reach passes unless--strict.wurk --helpnow lists it.WURK_HEALTH_PORTturns on the health listener from the environment, so a deployment manifest can add a probe with no app change. Off by default.config.health_checkwins when both are set. A non-numeric value raises at boot.- Metrics
wurk_leader,wurk_fetch_errors_total,wurk_reaper_reclaimed_total. The two counters are per process, so under a swarm they sample the child that owns the port. - Boot log names the backend and version (
Wurk 1.9.0 on dragonfly 1.34.0), read from INFO, with a WARN below the oldest version tested. Nothing branches on the backend. - Boot WARNs under
wurk/wurkswarm: liveprocessesentries that are not Wurk, and loaded job classes that setlock:while sidekiq-unique-jobs is not loaded (the option does nothing, so duplicates run).
Changed
Wurk::Sentrynow reports what the server answered with. The default filter narrowed fromRedisClient::ErrortoRedisClient::ConnectionError+ConnectionPool::TimeoutError.MISCONF,OOM,READONLY,MASTERDOWN,NOPERM,WRONGPASSand otherCommandErrors are reported, rate-limited per code per process (first, then one a minute, withextra[:suppressed]). Upgrade note: expect new Sentry events for these. Passingfiltered_error_classes:still replaces the list.Wurk::Sentryreports deaths it could not predict.install!adds a deduped death handler for poison-pill kills andsidekiq_retry_inanswering:kill/:discard. Those were silent before. It also addsWurk::JobRetry::HandledandWurk::Shutdownto Sentry'sexcluded_exceptions, including a laterSentry.init.wurkswarmexits 1 when every slot has crash-looped for 10 minutes (config.swarm_crash_exit_after, default 600;nildisables). A slot's streak is unbroken crashes within 60s of boot. It reportsWurk::Swarm::CrashLoopthrougherror_handlersfirst. Upgrade note: a release that cannot boot now ends the container instead of leaving a Running pod that processes nothing. A crash from an unreachable store or one that cannot serve yet (RedisClient::ConnectionError, pool timeout,LOADING,BUSY,CLUSTERDOWN,MASTERDOWN,READONLY,OOM/ERR Out of memory,MISCONF) does not count, so an outage alone never ends the swarm. A badWURK_HEALTH_PORTnow fails the parent's boot. A swarm co-hosted in a web process only reports, once per window, and keeps running.- Fetch-loop errors reach
error_handlersat most once a minute per exception class per process, withcontext[:suppressed]. Any handler benefits. The default handler's log volume for this path drops too.wurk_fetch_errors_totalstill counts every error. - The swarm child no longer calls
ActiveRecord::Base.establish_connection. Rails' fork hook already rebuilds the pools from the parent's config. The bare call re-derived config fromDATABASE_URL, silently switching a host's custom database or pool. Upgrade note: a host relying on the child readingDATABASE_URLafresh, which differs from what the parent established, now gets the parent's config. - An ActiveRecord pool smaller than concurrency logs at ERROR, was WARN.
wurkswarmloads a Rails target'sconfig/application.rbbefore theWURK_PRELOADgroupBundler.require, so gems that expect Rails at require time load. This applies to-r <app dir>and-r config/environment.rb. The latter now also defines the ActiveJob adapter first.- A web process that forks a swarm in production logs a WARN unless
WURK_EMBED=1. The default is unchanged. - Upgrading: reclaim and dead-set behaviour. A job that previously looped now lands in the dead set. Watch dead-set growth after upgrading if you had such payloads: they were pegging a thread before.
Fetcher::PrivateListKey.parse_owneris gone (internal, nothing else used it).Fetcher::OrphanGracehas a new constructor (internal). Reclaim after a host loss or pod restart is slower (see the figures), on purpose.super_fetch:reaper:fullis no longer written and expires on its own.Reaper#reclaim_full!,FULL_INTERVAL,FULL_LOCK_KEYandfull_interval:/full_lock_key:are gone (internal). - Known issue, Dragonfly at
maxmemory: on 1.34 and 2.0, between shards,LMOVE/BLMOVEon the fetch path can pop a job and drop the push, so a job runs with no private copy, and is gone if the process then dies or its retry is refused (2 of 16 in a test on 2.0.2). Do not let a Dragonfly job store reachmaxmemory: alert well below it. Seedocs/reliability.md. - Under Rails, every job now runs inside
Rails.application.reloader, in every boot mode (railtie swarm,wurk,wurkswarm, embedded). Before this, nothing wired it, despite the docs saying otherwise:ActiveSupport::CurrentAttributesset by one job was visible to the next on the same thread, a rawActiveRecord::Base.connectionstayed leased after the job, the query cache was off and development never reloaded code. Check these when upgrading: the query cache is now on inside a job, so a polling loop can read stale rows; a pool smaller than concurrency now waits and then raisesActiveRecord::ConnectionTimeoutErrorunder contention rather than having threads hold on to connections;Rails.errorsubscribers now see job failures taggedjob.sidekiq, includingWurk::JobRetry::SkipandHandled(spec §14 places the retrier inside the reloader), so filter those in any error tracker subscribed toRails.error; in development, reload waits for running jobs. Setconfig[:reloader] = nilto opt out. - Under the railtie, workers start after all host
after_initializehooks. Boot moved to a last Finisher initializer, so a Redis URL set in anafter_initializehook now applies, and a forked child has the routes and reload hooks Rails installs after those hooks fire. The relative boot order is unchanged. Sidekiq::WorkerandSidekiq::Jobare the same module, as they are upstream, sois_a?andprependwork on either name.- The release lane separates what runs third-party code from what can publish.
buildruns the bundle,bun installand the suite with no credential;publishalone holdsid-token: writeand runs only pinned actions;releasealone holdscontents: write. The unpinnedgem exec rubygems-awaitis gone. - A release that fails after
gem pushcan be finished by a re-run. Preflight reads the gem, the tag and the GitHub Release, and a live gem with the rest missing skips the push and finishes, instead of reporting green having done nothing. A tag on another commit is now refused before the push.RELEASE.mdhas the full table. deploy-demoproves the built commit is serving. The image carries its commit, the demo reports it at/demo/build, and a frozen rollout fails the run. It used to compare gem versions, which matched on the first attempt on any push without a version bump.- The demo's dependencies are patched and watched.
demo/Gemfile.lockmoves off advisories in rack, crass, sqlite3, loofah, nokogiri and rails-html-sanitizer; Dependabot covers/demo, and a weekly non-requiredauditworkflow runs bundle-audit andbun audit.
Fixed
- A limiter built at class load no longer exits the process when Redis is down, LOADING or OOM. Registration is deferred to its next operation, with a WARN.
- A failed requeue can no longer lose a job on Dragonfly. Dragonfly at
maxmemorydoes not abort aMULTIor a script on a refused write, soUnitOfWork#requeue,bulk_requeueand the self-reconcile applied theLREM, were refused theRPUSH, and the job was in no list. The moves now push first. A failing job on a full Dragonfly stays in the private list. Affects 1.9.0 and earlier on Dragonfly; Redis and Valkey were safe. - The reaper no longer loses jobs on a full Dragonfly. Its
LMOVEbetween shards could pop the job and drop the push: 5-6 of 24 orphans gone in one sweep on 1.34.0 and 2.0.2. It moves through a push-first script, with the same effect asLMOVE RIGHT RIGHT. - The scheduler's burial of an unpromotable job, and a debounce replacement, no longer drop the job on a refused write. Add first, then remove.
- A dashboard retry refused at
maxmemoryputs the job back.ZREMis notdenyoom; the push and the restoringZADDwere, so the entry was lost, on Redis too. Past that, on Dragonfly, the payload is written to the error log. - The reaper keeps working at
maxmemoryon Redis. Its sweep lock'sSET NXwas refused, so no periodic sweep ran, and a refused poison counter ended each drain after one job. A job moved while Redis is full is not poison-counted. - A live worker's jobs are no longer run twice after a Redis restart from an older snapshot, or when CPU-bound jobs hold back its heartbeat. Every heartbeat-judged owner now waits out a 30s grace (it was 15s, and only some owners). Reclaim after losing a host goes up to 90-141s; a swarm child stays at 1.2s.
- A dead owner's jobs come back within a sweep whichever queue they belong to. A sweep was scoped to what the lock's winner served, so a queue served by 1 process of 10 waited about 13 intervals. Every sweep now covers every list, and the hourly full sweep is gone.
- A sole survivor sweeps every interval, not every second one: the lock outlived the tick (180s became 120s, before the grace).
- A host bug in a failure hook no longer kills a healthy job. A raising
backtrace_cleaner, asidekiq_retry_inresult that cannot convert, a death handler raising pastStandardErrorand a raising error reporter all raised out of retry booking, and the Processor sent the job to the dead set on its first failure. - A job whose claim reply was lost no longer waits for the process to restart. A Redis restart, a stall past the 2.5s read timeout or a reset socket left up to one job per processor thread in a live worker's private list, silently, until that process exited. A job that failed while Redis was down, with both its retry booking and its requeue refused, ended up in the same place. Every worker now checks its own lists on each reaper tick and puts back anything no thread and no pending ACK accounts for, once it has been seen stranded twice at least 30s apart. Such a job runs up to two reaper intervals late. A running job is never touched. A failed requeue now also logs at ERROR.
- A full Redis no longer stops workers draining it.
LMOVEis refused atmaxmemory, so undernoevictiona backlog that filled Redis stopped every worker for good. Workers now claim through anallow-oomscript, still through the private list, and retry and dead-set booking from a failing job work the same way. Redis 7.0+ and Valkey. Dragonfly has no such header and keeps its own behaviour. Queues underglobal_concurrencyare not claimed on this path. An error report fires once a minute per fetcher while it lasts. - A payload that fails the same way every time no longer hot-loops. Valid JSON that is not an object (
[1,2],123,true,null) now goes to the dead set. A job that cannot be booked for a reason other than Redis (aretry_countthat is not a number, args that cannot be re-serialized) runs once and goes to the dead set with death handlers fired, instead of being requeued at up to ~560 re-runs ofperformper second. A Redis or pool error while booking still requeues the job, now after a 1s pause. - Orphan reclaim no longer depends on
OBJECT IDLETIME. Reading a private list (a monitor'sLLEN, our own runbook) reset that clock, so a polled dead owner's list was never reclaimed. The fallback on servers withoutOBJECTonly advanced in a process that won the sweep lock twice, so reclaim got slower as a fleet grew. Reclaim now takes the same time on every server and fleet size. - The reaper's sweep scans the keyspace once, not once per served queue: 199,300 SCAN calls and 43s per sweep at 100 queues and 200k keys became about 200.
- An idle capsule serving many queues stops paying per thread for the same empty answer. One empty
fetch_firstpass now answers for the whole fetcher for 250ms. At 1000 queues and concurrency 500, the idle cost was 70% of a worker core and 26% of a Redis core. Thread.exitin a job no longer loses a processor thread for the life of the process. The Manager is told and replaces it; the job is recovered by the reconcile above, with poison-pill counting.UnitOfWork#requeueno longer double-pushes on a lost reply. On that lost reply it now raises rather than guessing; the job is either moved or still in the private list, and both are recovered.- The default scheduler no longer loses jobs.
Scheduled::Enqclaims due members (rescoring them to a 60s lease) instead of popping them, pushes through the client chain as before, then removes them. A store out of memory, a lost reply on a stalled socket or akill -9between the steps now costs a re-promotion when the lease lapses. Measured: a store atmaxmemorylost 50 of 50 due jobs before and 0 now; 40,000 jobs through four 4s stalls lost 4 silently before and 0 now, on real Redis. Client middleware still runs on promotion, and promotion got 25-60% faster depending on load (reviewer about 25% under load; mine 55-60% solo and about 55-75% with fourredis-benchmarkloaders), because the removals share a round trip per batch of 10.- Upgrade notes: failures that used to lose a job can now run it twice, so a job should tolerate a duplicate (already the documented contract). A claimed member's score briefly reads about 60s in the future: visible on the dashboard for milliseconds normally, and up to the lease after a crash. A starved local connection pool now leaves the member in its set; it used to send it to the dead set.
reliable_pushno longer blocks callers during an outage. The first failed push opens a circuit for that Redis: later pushes buffer without touching the socket (under 1ms against a stopped server, where each previously blocked 6s), and a background drainer probes and replays in FIFO order.- Upgrade notes: a process using
reliable_push!now starts the drainer thread on its first failed push, even withoutreliable_push_drainer. While a circuit is open the drainer probes about every 0.25s, and buffered jobs replay then, not on the very next push. A clean exit flushes what is left, bounded at 5s, and logs an ERROR for what it could not deliver; a SIGKILL in that window loses those pushes. A batched push still tries Redis and raises. - Upgrade note: with no caller blocked by the outage, producers run at full speed, so the 1000-entry buffer fills far faster and starts dropping the oldest jobs sooner (46,000 pushes in 5s from 20 threads, measured). Raise
reliable_push_buffer, or setreliable_push_overflow = :raise, if an outage must not drop work.
- Upgrade notes: a process using
- With
reliable_push!, the default scheduler never counts a buffered push as delivered: promotions reach Redis or stay in their set. The scheduler bypasses the buffer and circuit. - Scheduler first sweep is jittered: 10-15s instead of exactly 10s, so a fleet booted together does not burst. An explicit
config[:scheduler_initial_wait]is still taken exactly. RELIABLE_SCHEDULE_PROMOTEcarries a--!df flags=allow-undeclared-keysheader, soreliable_scheduler!works on stock Dragonfly. The script's SHA changes; Redis and Valkey read the line as a comment.- The heartbeat keeps working at
maxmemory. On a refused beat it falls back to oneallow-oomscript that writesbeat, the:workmirror and the expiry, and refreshes global-concurrency holds the same way. Before, a worker with every thread in a long job stood still and was TERMed after 60s of OOM, and holds could lapse so a cap over-admitted. On a server that refuses the script (Dragonfly) the failure is as before. On Redis and Valkey the:workmirror of running jobs now stays in place while a beat is refused, because UNLINK and its rewrite are one transaction (2 more commands in the beat pipeline, only when jobs are running); a full Dragonfly still erases it, as before. GET /livestays 200 on a quiet (TSTP) process, and answers 503 only once it is shutting down. With/liveas the livenessProbe, a dashboard quiet got the pod restarted, and the replacement fetched again. Upgrade note: anything relying on/liveto detect quiet should readwurk_process_quiet.- A crash in a swarm child outside a lifecycle event reaches
error_handlers(context: 'swarm-child-boot'). It only reached the log. Crash streaks of 3+ log at ERROR. - A respawn no longer raises
ConnectionPool::PoolShuttingDownErrorin host threads. The fork path of a co-hosted swarm terminally shut pools the web process was using. It now closes idle sockets and leaves the pools usable. A socket checked out at that instant is inherited as before, unused by the child. Launcher#stopis bounded when Redis hangs. Redis-touching teardown shares one budget ending 4s pasttimeout(minimum 1s). Past it the process moves on; the heartbeat expires and the leader lock times out as after a SIGKILL. No job is at stake: managers have already drained.exe/wurkcould overruntimeoutby a minute.- A late
config.redis=logs a WARN when a pool is already built, since it silently does not apply to live connections. - An unweighted queue in a weighted list is consumed.
[["a", 2], "b"]silently never fetchedb: its jobs sat in the queue with no error. An all-zero weight list now means unweighted instead of raising. sidekiq-cronfires in a real worker. Wurk's native periodic poller overwrote the gem's@cron_poller, so the gem's suite stayed green while no entry ever ticked. Underwurkswarmthe gem also crash-looped each child withFrozenError, and under the railtie its load-timeconfigure_serverblock was dropped.bin/test-ecosystemnow boots a real worker for this gem, under bothsidekiqandwurkswarm, and asserts ticks.perform_in,perform_atandset(wait:)acceptActiveSupport::Duration, and honour frozen time.perform_in(5.minutes)raisedArgumentError. Intervals now readTime.now, sofreeze_timeandtravel_toapply.sidekiq-unique-jobsboots and enqueues.Config#redis_infoand#to_jsonwere missing, which stopped the server at startup and raised on each locked enqueue.retry,kill,add_to_queueandretry_allno longer destroy jobs when a gem prependsdelete_by_value.JobSet#remove_jobwent through that public method and read the gem's return value as "not removed", so the job was gone with nothing pushed. It also yields the raw JSON string now, as upstream does.sidekiq-failuresno longer breaks set iteration.SortedEntryparses on construction, so a gem reading@itemininitializeworks. This costs 2.5–10µs per entry only on walks that read no field. A set member that is valid JSON but not an object, or holds invalid bytes, reads as an empty job, soretry_allandkill_allno longer raise on it.Sidekiq::Work#raw,#[]and#digexist, for code written against Sidekiq 7.bundle exec sidekiqworks: the shim gem ships asidekiqexecutable.MyJob.jobs.clearclears, andSidekiq::Rails::Reloaderresolves.perform_inlineno longer reports a spurious error. The top-level middleware chains had no config, soMetrics::Historyraised and error handlers sawundefined method 'redis_pool' for nil.sidekiq_optionsstringifies two key levels, andsidekiq_options_hash=exists, as upstream.CurrentAttributes.persist("Name")works. The String form raised on everyperform_async. Load now sits outermost on the server chain, and a payload with nothing to persist carries nocattrkey.ProcessSetreads a stock Sidekiq process correctly. A nilHMGETvalue overrode the info JSON and gaveconcurrency=0.- The reaper's sweep loop can no longer be lost, and no longer dies with its own error report. Two holes in that loop's lifetime let a sweep thread end for good, silently — the thread sets
report_on_exception = false, so an exception escaping it leaves no trace, and orphans are then never reclaimed again for the life of the process.starthanded back the reference to a previous loop rather than the loop itself: astopwhose bounded join timed out leaves its straggler referenced, and once that straggler had wound down, a restart returned the dead thread and spawned nothing, so the reaper was dead for good whilestartlooked like it had worked.startnow reuses a live loop and puts a fresh one where a dead reference sits, so "no second sweep loop alongside a straggler" still holds while the straggler is alive. And the sweep's error report ran inside the tick's rescue, so a reporter that raised (a logger that cannot write, an error handler that raises) escaped the raw loop thread and killed it; the report is guarded now, so the loop survives its own reporter the way it already survives a Redis blip (K1). The CI red this answers is run 41454136586, job "compat (ruby 4.0, rails 8.1, valkey)" (job 125306502572): the failing step is "rake test" —bundle exec rake test, the same full suite (unit + integration + engine) the requiredtestjob runs — which ran 13:35:44Z to 13:41:22Z and failed, and the check's only diagnostic isProcess completed with exit code 1.The actions/cache and bundler-cache restore lines in that job are setup warnings, not the failure: every setup step of it (checkout, "resolve store port", setup-ruby withbundler-cache: true, setup-bun, setup-node, "build dashboard SPA", "dummy app setup") concludedsuccess, and "rake test" is the only failing step. The job's log body is not readable from a checkout without an authenticated GitHub token (the job-logs endpoint answers HTTP 403), so the failing test and assertion are not quoted from it; what the public API does show is that the leg's store did not float —valkey/valkey:8.1has resolved to 8.1.11 since 2026-10-27, the image this red run and that leg's last green one (41303800609) both used — and that the reaper's own unit suite is green against that image (13 consecutive runs, no failures). Commite01987ad485b39b4bb5bec6a2fa301a8fe9ed1d8did not introduce the failure: its diff touches one file,.dz/briefs/improve.md(+14/-0), a docs brief with no Ruby, CI or test in it; the only commit between that leg's last green run and this red one that changed runtime code is 4684178 (#869), which changed the idle-connection reaper's wait inlib/wurk/redis_pool.rband the wait idiom in two test files. The two regression cases live intest/unit/fetcher_reaper_test.rband fail on the unpatched tree ("a dead straggler is not a sweep loop", and the raising reporter escapingtick_oncethroughrun_loop). The divergence ledger's two reaper anchors moved with the code and were re-pinned with it —docs/idea/parity-divergences.mdnow nameslib/wurk/fetcher/reaper.rb:262-291,300-310, and the pin records intest/conformance/ledger_anchors/client-fetch.ymlwith it — because the conformance gate rejects a pin that no longer names the code it describes. The reaper's unit suite and that ledger-anchor gate are green against a local Valkey 8.1.11 and a local Redis 7.4.11. Nothing on the wire moves: no Redis key, job-JSON field or score format changed. - The idle-connection reaper's timed wait is the repo's Mutex+ConditionVariable again, and the compat cell (ruby 3.2.11 / rails 8.1 / redis) is green. The reaper's sweep waited on
Thread::Queue#pop(timeout:)— the one timed wait inlib/that did not use the Mutex+ConditionVariable idiom the tree's periodic, stop-signalled loops use (lib/wurk/client/buffered.rb,lib/wurk/timer_loop.rb) — and it ran that wait inside a rawThread.new, so a wait that raised there took the sweep thread down with it (the thread exceptions in CI run 41303800609, job "compat (ruby 3.2.11, rails 8.1, redis)"). The failing assertion on that leg was one of the two that wait the same way: "the sweep after the failing one still ran" (test/unit/fetcher_reaper_test.rb) and "the job ran within one fetch cycle" (test/integration/fetch_many_queues_test.rb). The reaper now waits on a Mutex+ConditionVariable pair: it still sweeps everymin(timeout / 2, 10)seconds, andstopstill wakes the sweep at once, so the reaper's own cases — an idle connection reaped on the interval, a stop that takes effect at once — hold unchanged. Both cases take their value with the same idiom, so their assertions run on the support floor. Nothing on the wire moves: no Redis key, job-JSON field or score format changed. - A failed USR2 log reopen is reported instead of swallowed.
Wurk::CLI, the swarm parent and each forked child reopened their logger on USR2 (logrotate) and discarded any exception, so a reopen that failed (permissions, full disk, a closed stream) left the process writing to the rotated, possibly deleted, file with no sign of it. The failure now goes to stderr — not to the logger, which is what just failed — naming the exception; signal handling carries on as before. - A malformed dashboard-extension locale file is logged, not silently dropped. An extension's
locales/en.ymlthat failed to parse was replaced by an empty Hash, so its strings fell back to their keys with no clue why. The file is still skipped (the dashboard keeps serving), andWurk.loggernow warns once with the file path and the parse error.