Skip to content

Releases

wurk v1.15.1

Choose a tag to compare

@github-actions github-actions released this 12 Nov 13:58
0fbd0c0

Changed

  • Dashboard: no decorative motion. Page headers, tables and the search landing no longer fade up on every page load (the title visibly jumped twice: once as the loading skeleton, again as the real header). Stat numbers render their value instead of counting up from zero, and loading placeholders are static instead of shimmering. The live-status dot and the spinner still move, since they show state.

Fixed

  • Dashboard Metrics page fetched the queue history twice. The depth and latency panels used different query keys for the same /api/queue-history URL; they now share one request.

Documentation

  • Rolling cutover for stock Sidekiq 8 apps. docs/migrate-from-sidekiq.md now offers a rolling cutover (both fleets live on one Redis for a while) to a stock Sidekiq 8 app, within what test/integration/mixed_fleet_test.rb covers: that test runs real Sidekiq 8.1 next to a wurkswarm in its own CI job. Two conditions: stop Wurk processes with TERM, and check that no Wurk private list is left before the last Wurk process goes, because stock Sidekiq never reclaims them. Sidekiq 7.x, Pro and Enterprise apps stay on the drain cutover.

Upgrade notes

  • No action needed. Dashboard-only change plus a docs addition; no Redis, job format or configuration change.

wurk v1.15.0

Choose a tag to compare

@github-actions github-actions released this 12 Nov 03:24
5ea710e

Added

  • More /metrics families:

    • child restarts by reason (wurk_swarm_child_restarts_total{reason}) and the crash streak (wurk_swarm_crash_streak);
    • Redis round-trip time;
    • pool checkout timeouts and free slots;
    • poison kills, reclaims by source and leader changes;
    • scheduler promotions and lag;
    • outage-buffer drops;
    • the oldest running job;
    • per-child PSS;
    • per-class duration histograms (the 20 busiest classes plus other).

    Every family is listed in docs/metrics.md. The restart counters live in the swarm parent's memory and add no Redis key.

  • wurk health --max-job-age SECONDS fails the probe when a job has run longer than that.

  • config.metrics_retention= sets how long the 10-minute metrics buckets are kept. It changes the TTL only, has a 3-day floor, and the default stays 30 days.

  • /v1 HTTP API:

    • docs/api/openapi.yaml, an OpenAPI 3.1 spec that a test keeps in step with the routes;
    • operator routes POST /retries/:jid/retry, POST /dead/:jid/retry and POST /queues/:name/clear (admin scope, accept Idempotency-Key), plus GET /batches;
    • an optional ?score= on DELETE /jobs/:jid and the retry routes, which finds the job with one ZRANGEBYSCORE instead of scanning the set.
  • Python client: clear_queue, retry_retrying, retry_dead, batches and cancel(score=), plus an end-to-end test against a real wurk api.

  • docs/job-format.md documents the job JSON contract for producers in other languages; a parity test checks it.

  • Dashboard:

    • health notices for queues nothing consumes, non-Wurk processes, mixed Wurk versions and embedded processes;
    • Retries, Scheduled and Dead filter on error class, message and args, and "Matching" actions act on every match (up to 1,000 per request, following a cursor);
    • Batches has a Dead tab (?dead=1) and a tag filter (?tag=);
    • queue pages list oldest first with ?direction=asc;
    • ?days=N (1–180) works on the history endpoints and opens the matching range in the dashboard.
  • docs/dashboard-guide.md.

  • Ops examples: examples/capistrano/wurk.rake, plus StartLimit* settings in examples/wurk.service. docs/deployment.md gains a sizing table, and RELEASE.md sets an "Upgrade notes" convention for release notes.

Changed

  • Profile upload to the Firefox profiler is a same-origin POST. GET /profiles/:key no longer uploads: it redirects to an existing upload or answers 405.
  • Dashboard bulk actions look up all their keys in one Redis round trip (100 keys: 100 → 1).
  • Batch checks read children in pipelines. A batch's child-death check and invalidate_all on a tree with 40 children and 40 grandchildren take 3 round trips instead of 162.
  • Finding a job by jid in a retry, scheduled or dead set scans 1,000 entries per ZSCAN instead of 100.
  • Fetcher allocations: weighted, random and paused-queue fetchers allocate less per fetch pass (strict mode with a paused queue: 4 → 0 objects; weighted and random: 4–5 → 2). Each job still takes 3 Redis commands.
  • reliable_push outage replay sends 100 jobs per push instead of one (250 buffered jobs: 250 pushes → 3).
  • The heartbeat info JSON is built once per process, byte-identical to before.
  • The statsd queue.size gauge is sent by the leader only.

Fixed

  • Dashboard compatibility: it works under a strict nonce-based CSP and when the host app is mounted under a path prefix.
  • Dashboard actions over plain HTTP (not localhost) are accepted when Origin (or Referer) matches the request's own scheme, host and port. They are refused when neither header is present. A refusal returns {"error":"cross_origin"} and shows its own toast.
  • Geist Sans now applies. The dashboard asked for a font family name the package does not register, so the UI fell back to the system font.

Upgrade notes

  • behaviour-change: the statsd queue.size gauge comes from the leader only; a dashboard that sums it across the process tag should take max (or last) instead.
  • behaviour-change: GET /profiles/:key no longer uploads to the Firefox profiler; the dashboard uploads with a same-origin POST, and a bookmarked GET redirects to an existing upload or answers 405.

wurk v1.14.0

Choose a tag to compare

@github-actions github-actions released this 11 Nov 20:10
9f70c45

Added

  • GET <retries|scheduled|dead>/:key in the dashboard API: one entry with its backtrace, which the job detail now fetches when opened.
  • docs/dashboard.md "What the dashboard costs Redis".
  • memory_limit_metric (WURK_MAXMEM_METRIC): recycle swarm children on rss (default), pss or uss. An unknown value raises at boot; a limit on a host without a readable /proc logs one WARN.
  • Wurk::Client.reliable_push_buffer_bytes (opt-in): caps the outage buffer by payload bytes as well as by count. Past it the overflow mode applies; a payload larger than the cap on its own is dropped (or raised under :raise).
  • docs/deployment.md "Memory": copy-on-write and the pre-fork warmup, allocator guidance (jemalloc, MALLOC_ARENA_MAX) and a table of Wurk-only keys with their TTLs.

Changed

  • The dashboard costs a fixed number of Redis round trips. The landing payload, each live-update tick and the queue list take 2 (were 9 and 6), and open streams in one process share one read per second. Queue detail and the retry, scheduled and dead pages take 1 at any depth; batches and flows take 2 at any depth (page 999 of batches was 25,276). The Limits tab checks limiter metadata at most once a minute per process.
  • Dashboard list payloads are smaller. Batch rows report failed_count and dead_count instead of every failed and dead jid (the batch page still lists them), and job rows no longer carry error_backtrace. The HTTP API (/v1) rows are unchanged.
  • "Retry All" removes and pushes a page of 50 jobs in one round trip each (0.06 round trips per job, was 2.02). When a gem hooks the per-entry methods (sidekiq-unique-jobs), it keeps the per-entry path so the gem still runs.
  • A concurrent-limited job costs 2 Redis round trips (was 6), 3 when the limiter is built inside perform (was 7): the counters are booked inside the acquire and release scripts.
  • The periodic (cron) tick reads every loop in 2 round trips instead of 1 + 2 per loop, and boot registration takes 2–3 instead of up to ~26,000 with many classes.
  • Fast deletes (Queue#delete_job, #delete_by_class) walk the queue 1,000 entries per Lua call and remove at most 100 matches per call. Each call holds Redis well under 2 ms when nothing matches, up to ~10 ms when it removes entries halfway into a 1M-entry queue; the single script this replaced held Redis for the whole queue (~300 ms at 100k jobs). An entry another client removes ahead of the walk can be missed by that call; a second call finds it.
  • Dashboard metrics: the 30-day top-jobs view allocates 60% less, the queue-history series reads only the requested queues (1.44M → 2,880 fields), queue gauges no longer store zero values, and a metrics flush sends one EXPIRE per shared key, as Sidekiq 8.1 does.
  • The job path allocates 26 fewer objects per job (105 → 79 in Processor#process_one). The default log formatter builds each line in one buffer and caches pid=; built-in server middleware (interrupt handler, expiry, timeout, status, batch, limiter, statsd, metrics history) reuse one instance per chain entry, while host middleware still gets a fresh instance per job; the fetcher sends prebuilt commands. Log bytes, Redis keys, job JSON and the 3 commands per job are unchanged.
  • The swarm warms the heap before its first fork on every path, the Rails auto-boot swarm and Ruby 3.2 included (full GC and compaction where Process.warmup is missing), and loads what children would otherwise load each (fiddle, cron time zones). Shared memory at boot measured 0.57 → 0.72 against no warmup.
  • require "wurk" loads 41% fewer files (163 → 95; 186 → 110 ms, 17.5 → 12.5 MB RSS): the server, dashboard, flow, iterable-job, deploy and encryption code load on first use. A server boot takes 233 → 176 ms and the swarm still loads everything before it forks (Wurk.eager_load!), so children share it. require "sidekiq/web" and the other sidekiq/* sub-paths load only what they name. Every Sidekiq::* alias resolves to the same constant as before.
  • The dashboard's first load is 777 KB → 352 KB raw, ~104 KB served brotli (before: no compression). Locales other than English load on demand; the 47 icons in use are inline SVGs instead of the Font Awesome fonts and stylesheet; the logo is resized and the unused Open Graph image and tags are gone. rake frontend:build writes .br/.gz next to each asset and the engine serves them by Accept-Encoding, with Vary: Accept-Encoding.
  • zlib loads on the first compressed backtrace or profile, not at boot. A new integration test pins an idle worker's threads and Redis connections.
  • The heartbeat drains signals with one counted RPOP instead of ten; a CLI boot reads Redis INFO once instead of twice.
  • Outside a server process (test runs, rails runner, web processes), a skipped configure_server block logs at DEBUG instead of WARN, as Sidekiq skips it silently. The block still runs if the process later enters server mode.

Fixed

  • A memory limit below a freshly booted child's size no longer fork-loops the swarm. A child already over the limit within its first 60s is reported once at ERROR and kept; a slot recycles at most once for that cause.
  • Per-process caches are bounded: the queue-without-consumers warning map, the error coalescer (1,024 keys) and the Sentry middleware's thread-locals.
  • A dead-set kill builds its synthetic backtrace only when death handlers will see it.

wurk v1.13.0

Choose a tag to compare

@github-actions github-actions released this 11 Nov 06:44
112e11b

Added

  • bundle exec sidekiq keeps working after the gem swap. Wurk ships a sidekiq executable (and sidekiqswarm); the sidekiq shim gem no longer ships one, so only one gem in a bundle provides it. The shim depends on exactly its own Wurk version, and the documented recipe pins tag: "vX.Y.Z" — add the tag to existing shim lines.
  • Sidekiq Web's probe and stats routes: HEAD / (Redis probe), GET /stats, GET /stats/queues and GET /dashboard/stats are served on every mount with upstream's payload shapes, behind the dashboard's authorization.
  • Process titles like Sidekiq's: workers title themselves sidekiq <Sidekiq::VERSION> <tag> [x of y busy] (plus stopping when quiet), so ps-based probes that grep ^sidekiq keep working; Sidekiq::Launcher::PROCTITLES is extensible. A swarm parent titles itself <binary> <tag> [x of y processes].
  • wurk boot-check --runner wurk checks the single-process boot path; the default stays wurkswarm.
  • More boot warnings: a topology set but ignored by single-process wurk, Rails framework queues (mail, Active Storage, Action Mailbox) nothing listens to, and the hiredis driver under a fiber scheduler.
  • More wurk preflight --app rows: millisecond time fields read as seconds, a connection_pool pin below 3, Rails.error subscribers (they see every attempt, as with Sidekiq), a Procfile web process that would also run workers, flushes in boot code, plugin versions off their tested pins, and defined?(Sidekiq::Pro…) guards.
  • Wurk::Sentry.excluded_exceptions: the retry signals (Handled, Skip, Shutdown) for Sentry.init's excluded_exceptions in a process that does not call install!.
  • docs/idempotency.md: when a job runs twice, and how to guard each case.
  • rails g wurk:install prints the pinned shim line when a bundled gem depends on sidekiq, plus Procfile notes; the gem has a short post-install message.

Changed

  • Job log elapsed is in seconds (elapsed=0.007), as Sidekiq logs it. It was milliseconds; log parsers that compensated must change.

  • A swarm parent returns a dead child's in-flight jobs the moment it reaps it, by exact key (measured 0.15–0.17s after a SIGKILL), and boot-time sweeps share the super_fetch:reaper lock, so a deploy or rolling restart of N processes is one keyspace pass.

  • A skipped configure_client block in a server logs at DEBUG, as Sidekiq skips it silently; a WARN fires only when no configure_server block ran at all.

  • The swarm size line reads Wurk swarm size: N children (<source>) instead of Wurk swarm: forking N children (<source>).

  • On a shutdown that passes its deadline, each interrupted job is logged at WARN with its class, jid and queue (at most 20 lines) instead of one DEBUG dump.

  • Measurement tooling for contributors: bin/profile vs_sidekiq (per-job CPU and per-stage frame attribution against stock Sidekiq), bin/rake bench:require_profile (require cost per entry point), bin/rake bench:rss_soak (per-child Rss/Pss over 100k jobs), bin/rake bench:soak (chaos soak with a jid ledger; nightly soak.yml), and bin/rake bench:algo / bench:algo_counts (dashboard and feature round-trip budgets, enforced in CI). Results from 2026-11-11 are in docs/benchmarks.md with raw data under docs/benchmarks/2026-11-11/.

Fixed

  • wurk boot-check can no longer write Redis. Without --redis, Wurk's pools open no connection; with it, they send reads only. Boot code that touches Redis (a flushdb in a dev-only configure_server block, an enqueue) fails the check, naming its file and line, instead of running against whatever Redis it reached. wurk preflight --app boots behind the same read-only guard. Found by an adopter.
  • A poison kill whose script reply was lost still fires death handlers, :death batch callbacks and on_poison: the reclaim script is sent once and its outcome read back (#908).
  • Jobs queued by Wurk < 1.12 with a string timeout ("5") are cut at their declared time instead of running unbounded.
  • Docs: docs/reliability.md lists a job that finished while Redis was down as a duplicate window (its ACK fails, so it runs again once requeued; never a loss), and docs/batches.md says a batch callback is enqueued at most once but its job runs at least once, so callbacks must be idempotent. Both were observed by the new chaos soak.

wurk v1.12.0

Choose a tag to compare

@github-actions github-actions released this 11 Nov 03:01
8a4eda4

Added

  • wurk boot-check boots the app exactly as wurkswarm does (eager load and routes included), then exits without forking or fetching; --redis adds a read-only server check. Run it in CI to catch the worker-only crash loops (an engine controller referenced from an initializer, a missing adapter, load order) that a web or test boot never hits. docs/migrate-from-sidekiq.md §9.1 has the CI step.
  • Boot warnings for silent misconfiguration: jobs whose unique_for:/unique_until: do nothing because Sidekiq::Enterprise.unique! was never called; the real sidekiq gem bundled next to Wurk (ERROR naming what pulled it in and the pinned shim line; WURK_STRICT=1 refuses to boot); an Active Record pool smaller than the job threads. Swarms and embedded workers started by the railtie now run these checks too. docs/running.md lists every boot message with its fix.
  • One role-and-size line per process (wurk, wurkswarm parent, swarm parent in <server>, embedded in <server>), e.g. wurkswarm parent; 2 processes × 3 threads (redis pool 10, fetch 3); AR pool 25.

Changed

  • The "pool too small" startup error names both sizes and the setting that fixes it.

Fixed

  • Unique jobs declared with an ActiveSupport::Duration release their lock. sidekiq_options unique_for: 5.minutes (ActiveJob included) is written into the job JSON as the string "300", as Sidekiq writes it; the server middleware and death handler ignored a string TTL, so the lock was held until it expired. Payloads already in Redis are covered. Found in production by an adopter.
  • retry_for given as a Duration keeps its retry budget. sidekiq_options retry_for: 48.hours was written to the job as null, so the job fell back to 25 counted retries; a per-call set(retry_for: 2.hours) raised "retry_for over 1000000000". Both now store Integer seconds, as Sidekiq does.
  • timeout: and deadline: given as Durations are written as plain numbers. A timeout: 5.seconds reached the job as the string "5", which the timeout middleware ignored, so the job ran unbounded. Jobs already queued with a string timeout are still unbounded until they finish.
  • Docs: the poison-pill docs describe the reclaim script (the kill comes on the 4th orphaning, goes straight from the private list to the dead set, and fires death handlers; PoisonPill.recovered! replaces track!); the wiki and migration guide no longer say clustered Puma forks a duplicate swarm, that sentry-sidekiq cannot be installed, or that reliable_scheduler! must be kept; the install template links to hosted docs the gem does not ship.
  • A poison-pill job is never handed to a live worker while it is being killed. The reaper pushed a reclaimed job back onto its public queue and only then counted the recovery, so a worker parked on an idle queue could take the job a 4th time while the reaper moved the same job to the dead set and fired its death handlers (batch :death, unique-lock release) mid-run. The count and the requeue-or-kill decision now happen in one script, before the job is visible. Orphans returned by a worker's own reconcile go through the same path. The recovery counter is written together with its 72h TTL, so a Redis blip can no longer leave super_fetch:recovered:<jid> with no expiry. Keys, job JSON and dead-set scores are unchanged.

wurk v1.11.1

Choose a tag to compare

@github-actions github-actions released this 06 Nov 19:48
8de7488

Periodic fixes from a production rollback: a loop whose class is gone no longer becomes a retry storm, and config.periodic registers when the server boots instead of while initializers load.

Changed

  • config.periodic registrations reach Redis when the server boots, not when the initializer runs. The launcher writes them, and prunes superseded loops, just before the cron poller starts, as Sidekiq Enterprise registers on startup. A periodic block no longer builds a Redis pool before a later config.redis = (which logged "config.redis= called after a Redis pool was already built" and, in a single process, kept the old settings), and a process that never boots a server (console, rake, a skipped embed) no longer rewrites or prunes the registry. mgr.authoritative! inside the block runs at that point too. A standalone Wurk::Cron::Manager.new.register still writes at once. Upgrade note: code that read Wurk::Cron::LoopSet right after config.periodic, without booting a server, now finds the loops missing; call config.apply_periodic!. (#898)

Fixed

  • A periodic loop whose class no longer exists no longer turns into a retry storm. The leader pushed a loop whose class it could not load as a bare class name, without the worker's sidekiq_options, so a job declaring retry: 0 failed with NameError under the default 25 retries on every tick (seen in production after a rollback left a newer deploy's */10 loop in Redis: 16 errors from 3 ticks). The occurrence is now claimed and skipped, nothing is pushed, a WARN names the class and lid, and the error handlers get one Wurk::Cron::UnknownWorkerClass per loop per process. The loop stays registered until Wurk::Cron.unregister(lid) or mgr.authoritative!. During a rolling deploy, a loop for a newly added class is skipped while the leader still runs the old code. The dashboard's Enqueue action is unchanged (#898).
  • The built-in extension partials (_paging, _job_info) ship inside the gem; an installed 1.11.0 raised extension <name> has no view _paging (#890).
  • A capped queue refused at capacity backs off from when the refusal arrived, not when the claim was sent, so a slow Redis no longer turns a full capped queue into a spin (#894).
  • The demo lock is off the Rails 8.1.4 advisories. demo/Gemfile.lock moves to Rails 8.1.4.1 (actionpack, actionview, activestorage, actionmailbox, activesupport): IP spoofing in ActionDispatch::RemoteIp, a Strong Parameters bypass, restoration of destroyed sessions in CacheStore, missing SameSite on session cookies, content-negotiation cache poisoning, a session leak and a DoS in Live streaming, XSS in the Action View text/tag/strip_tags/highlight helpers, a signature replay in the Action Mailbox Mailgun ingress, arbitrary file read, write and RCE plus size-limit circumvention in Active Storage, and a MessagePack XSS and a safe_constantize DoS in Active Support. The demo image installs the lock frozen, so this is what it ships; nothing outside the Rails family moves.

wurk v1.11.0

Choose a tag to compare

@github-actions github-actions released this 03 Nov 07:07
fb2f6f9

Fixes a 1.10.0 boot regression under wurkswarm: upgrade if your config/application.rb relies on the swarm's preload. Native periodics now keep firing through leader deaths, and push_bulk is cheaper.

Added

  • wurk preflight: what a live Redis holds and what carries over, before a cutover. Read-only: SCAN and bounded sampling through a client that refuses every other command, never LLEN. Reports Redis version, eviction, topology and Dragonfly; live processes by version; every queue:*|* key as owner-alive, reclaimable or unparseable; timestamp shapes and retry_for jobs; in-flight batches with no callbacks; encrypt payloads without an envelope; uniquejobs:* and unique:* counts including SHA1-length digests. --app PATH boots the app to list lock: classes with no gem and known-broken add-ons, and runs your initializers. --json, --limit (a truncated scan is a finding), --sample, --pause-ms. Exit 0 clean, 1 findings, 2 could not run. About 10s at 1M keys with Redis latency under 5ms in a local test.
  • wurk verify: the post-cutover check. Queues with jobs have a subscriber, no non-Wurk process is live, private-list count is sane, a live Wurk process holds the leader lock, every unpaused loop has a next fire. Read-only; --canary CLASS explicitly pushes one job per subscribed queue.
  • docs/migrate-from-sidekiq.md §10: a path for Sidekiq 7.x Pro/Enterprise apps that keep their add-on gems, with what was verified per gem, and what a 7.x rollback shows.
  • Sidekiq::WebHelpers (alias of Wurk::Web::Extension::Helpers) with job_params, parse_key, parse_params, qparams, to_query_string, display_args, filtering, redirect_with_query, text_direction, rtl?, locale, stats, workset, processes, sorted_processes, SAFE_QPARAMS, RETRY_JOB_KEYS. A subset of free §25.7, not all of it.
  • Sidekiq::Paginator (page, page_items), and page on the extension Action. Action also gains config.
  • erb in extensions honours locals:; built-in _paging and _job_info partials; upstream string keys read as English (DeleteAll → Delete All).
  • Sidekiq.gem_version, Sidekiq.freeze!.
  • sidekiq-unique-jobs, sidekiq-failures and sentry-sidekiq in test:ecosystem, and a real-worker boot check for all four harnesses, covering sidekiq and wurkswarm with preload on, off, and fake mode at load.
  • bin/repin-anchors: re-pins the ledger's anchor records, syncs both of the ledger's reference styles and test/conformance/cases.yml after lines move, with --check to report without writing.
  • mgr.grace = 300 in config.periodic lets a slot fire later than the default 90s, for a cut-over from sidekiq-cron. Process-wide, default unchanged.

Changed

  • mount Sidekiq::Web in a Rails app now shows the dashboard. Extension routes still serve at their own paths; every other path goes to the engine, through the engine's own Wurk::Web.use middleware, authorization hook and read-only gate. The machine API is not served under this mount. Upgrade note: a host that kept this mount as a 404 (so it only exposed extension pages) now exposes the whole dashboard. Gate it as you would mount Wurk::Engine. As with stock Sidekiq, a mount with no auth is open.
  • A worker started by wurk, sidekiq, a swarm child or Wurk::Embedded now refuses Sidekiq::Testing fake or inline mode: it logs an ERROR and disables the mode for that process. Previously, as on stock Sidekiq, every enqueue went to an in-memory array and was lost. Upgrade note: enqueues that were being dropped start reaching Redis. Move rspec-sidekiq to your Gemfile's test group. A Rails web process enqueuing in fake mode is not covered, and the guard stands down under RAILS_ENV=test or RACK_ENV=test.
  • A job with encrypt set and a last argument that is not a Wurk envelope no longer reaches perform. It goes to the dead set with encryption_error: Wurk::Encryption::NotSealedError, without retries, and death handlers fire. That kill, like every decrypt failure's, no longer trims the dead set, and logs an ERROR with class and jid, so a misconfigured producer cannot destroy a backlog through dead_max_jobs. Before, another implementation's ciphertext was passed to perform as if decrypted. Upgrade note: this also catches a plaintext job from a producer that has encrypt: true on the class but crypto not enabled, which used to run in the clear. Enable crypto on producers first. A dead-set retry seals the argument as it stands, which recovers plaintext jobs; do not retry foreign ciphertext. Only applies with crypto enabled on the worker.
  • Sidekiq::Web as a bare Rack app answers a non-extension path with a 404 whose body names the fix, and logs one WARN.
  • Extension truncate cuts like upstream: text[0..n] followed by ..., not ….
  • An extension's tab on a path the SPA also renders natively (sidekiq-cron's Cron) is no longer hidden. With sidekiq-cron, the nav shows the native Cron entry and the gem's Cron tab.
  • An extension view that names a view that doesn't exist now raises extension <name> has no view <view>, instead of has no root_dir.
  • push_bulk is cheaper: 15.0µs to 9.4µs per job (stock Sidekiq 6.8µs, 10 paired rounds). It normalizes once per slice instead of once per job. Payload key order is unchanged. Upgrade note: jobs in one bulk slice now share a created_at; they used to differ by the milliseconds the loop took. Stock Sidekiq shares one across the whole bulk. A per-job at keeps the old walk, since expiry and deadline are measured from it.
  • --help differs per binary and names environment variables and the docs.
  • bin/rails g wurk:install prints next steps, including that the dashboard has no login of its own.
  • Each release gets a CycloneDX SBOM asset and an SBOM attestation, from a separate attest.yml that can't reach publishing.

Fixed

  • docs/migrate-from-sidekiq.md no longer claims: :while_executing maps to unique_until: :start; "worst case one extra run" on dropping sidekiq-unique-jobs; leftover gem locks "do nothing" (they silently block enqueues after a rollback; UNLINK them); same encryption keys carry Enterprise jobs over; the default scheduler has a job-loss window; reliable_scheduler! must stay on; the Sidekiq web UI "keeps working" unqualified. docs/compatibility.md pointed the parity-divergence rule at docs/target; it is the ledger.
  • docs/sentry.md no longer says sentry-sidekiq reports the wrong errors on Wurk. Against a dummy transport it reported the right exception per attempt; it has not been tested with a real Sentry.
  • wurkswarm no longer crashes apps whose config/application.rb relies on the swarm's preload. Since 1.10.0, an app that skips Bundler.require because the swarm already ran it, then uses a preloaded gem, died at boot with wurkswarm: uninitialized constant Dotenv (or whichever gem) and exit 1. It now restarts itself once with the preload first, under bundle exec, a binstub or load Gem.bin_path, keeping its pid, and logs a warning naming the first error. Gems that need Rails at require time keep working. Such an app does start twice per boot; a Bundler.require in config/application.rb avoids that.
  • Boot's lock: check no longer dies on a class that redefines self.name. With a gem like Faker loaded (Faker::Travel::Airport), ArgumentError: missing keywords skipped the whole check, so jobs setting a lock: that did nothing got no warning. Sidekiq::Testing's queue check had the same flaw.
  • Native periodics survive a leader's death. A follower rechecked the lock on a flat 60s, so a killed leader left up to 90s with no lease; it now rechecks when the lease it read runs out, about a second after a lapse (31s at most after a SIGKILL). The lock and its TTL are unchanged. A new leader ticks the moment it wins rather than at its poller's next minute. Under repeated leader kills, periodics fired 8-9 of 10 slots, up from 0-1.
  • A leader's first tick is no longer spent on a stale slot. A slot over the grace is still skipped and logged, and the occurrence due now fires in its place, under the same claim, so nothing fires twice.
  • The leader lease survives maxmemory. Renewal and a new leader's token go through allow-oom scripts, as the heartbeat does. Dragonfly refuses those, and falls back to what it did: the lease lapses until memory returns.
  • A leader that fails one renewal retries in 15s, not 60s, while it may still hold its lease.
  • wurk and wurkswarm print one line for failures they fully understand: a bad flag, a bad config path, an unreachable Redis. Those traces now need -v. Any other error, including anything raised by the app, still prints its backtrace. Exit codes are unchanged.
  • A stray word on the command line (wurk foo) now warns. It still boots as before.
  • A missing -r path is logged at ERROR and named. It was a generic INFO message.
  • No more Signal INFO not supported warning at every Linux boot.
  • docs/migrate-from-sidekiq.md, the wiki and llms.txt no longer say the railtie starts a swarm in every Rails process or that each clustered Puma worker forks its own. It boots only inside a running web server or with WURK_EMBED=1, and a preforking server refuses with a warning.
  • The same docs no longer promise a threads-only fallback on JRuby, TruffleRuby and Windows. wurkswarm and the railtie's swarm call Process.fork with no fallback, and no CI leg runs those platforms.

Upgrade notes

  • A minutely loop that had been skipped on recovery now fires once on the first tick after an outage. Loops with no occurrence inside the grace behave as before.
  • A process that wins the lock ticks at once instead of after up to 60s, including just after boot.
  • Followers send about 2 commands per 15-30s (EVALSHA, PTTL) instead of 1 per 60s.
  • A Redis restored from a snapshot older than the las...
Read more

wurk v1.10.0

Choose a tag to compare

@github-actions github-actions released this 03 Nov 00:57
b68f355

Upgrade if you run on Dragonfly. 1.9.0 and earlier could lose jobs on a Dragonfly at maxmemory, on Redis-compatible paths that are safe on Redis and Valkey. This release also closes several other ways to lose or double-run jobs, wraps every job in the Rails reloader, and makes more of the Sidekiq gem ecosystem work. Read Changed before upgrading: jobs now run with the query cache on, a connection pool smaller than concurrency can raise under contention, Sentry gets store errors it used to filter, and wurkswarm exits after sustained crash-looping. No Redis key, job JSON field or score format changed.

Added

  • Queues nothing listens to are now visible. The leader logs one WARN an hour per non-empty queue with no live subscriber, after 5 minutes in that state, so a rolling restart stays quiet. New metric wurk_queue_consumers{queue=}, a "No consumer" badge on the dashboard Queues page, and Wurk::Testing.unconsumed_job_queues(listened:|config_file:) for a guard test. GET /api/queues gains a consumers field. No new Redis keys.
  • Wurk::Sentry.install!(config, report_after_attempts: N) also reports attempt N, on top of the terminal failure.
  • Dragonfly >= 1.34 moves to "works in production; tested in CI" in docs/deployment.md, with configuration requirements and durability limits (RPO is the snapshot cadence). New non-required compat CI leg on v1.34.0 with stock Lua settings.
  • New examples/kubernetes/: rolling and Recreate workers, Service + ServiceMonitor + NetworkPolicy, PrometheusRule with an absent() alert. A unit test keeps every metric in them in step with the exporter.
  • docs/deployment.md: WURK_HEALTH_PORT, Recreate variant, scraping and alerting, fiber servers, host-shutdown grace, corrected reclaim timing after SIGKILL.
  • The demo turns on the health listener by default.
  • mgr.load_hash, mgr.load_file and mgr.authoritative! on config.periodic. The loaders take sidekiq-cron's shape (cron, class, queue, args, retry, tz, status). authoritative! is opt-in and deletes every loop the block did not declare.
  • wurk health: an exec-style liveness probe. It boots no app and exits 0 when a process on this host has a heartbeat within --window seconds (default 60; --min COUNT). It reads REDIS_URL or --redis-url. A Redis it cannot reach passes unless --strict. wurk --help now lists it.
  • WURK_HEALTH_PORT turns on the health listener from the environment, so a deployment manifest can add a probe with no app change. Off by default. config.health_check wins when both are set. A non-numeric value raises at boot.
  • Metrics wurk_leader, wurk_fetch_errors_total, wurk_reaper_reclaimed_total. The two counters are per process, so under a swarm they sample the child that owns the port.
  • Boot log names the backend and version (Wurk 1.9.0 on dragonfly 1.34.0), read from INFO, with a WARN below the oldest version tested. Nothing branches on the backend.
  • Boot WARNs under wurk / wurkswarm: live processes entries that are not Wurk, and loaded job classes that set lock: while sidekiq-unique-jobs is not loaded (the option does nothing, so duplicates run).

Changed

  • Wurk::Sentry now reports what the server answered with. The default filter narrowed from RedisClient::Error to RedisClient::ConnectionError + ConnectionPool::TimeoutError. MISCONF, OOM, READONLY, MASTERDOWN, NOPERM, WRONGPASS and other CommandErrors are reported, rate-limited per code per process (first, then one a minute, with extra[:suppressed]). Upgrade note: expect new Sentry events for these. Passing filtered_error_classes: still replaces the list.
  • Wurk::Sentry reports deaths it could not predict. install! adds a deduped death handler for poison-pill kills and sidekiq_retry_in answering :kill / :discard. Those were silent before. It also adds Wurk::JobRetry::Handled and Wurk::Shutdown to Sentry's excluded_exceptions, including a later Sentry.init.
  • wurkswarm exits 1 when every slot has crash-looped for 10 minutes (config.swarm_crash_exit_after, default 600; nil disables). A slot's streak is unbroken crashes within 60s of boot. It reports Wurk::Swarm::CrashLoop through error_handlers first. Upgrade note: a release that cannot boot now ends the container instead of leaving a Running pod that processes nothing. A crash from an unreachable store or one that cannot serve yet (RedisClient::ConnectionError, pool timeout, LOADING, BUSY, CLUSTERDOWN, MASTERDOWN, READONLY, OOM / ERR Out of memory, MISCONF) does not count, so an outage alone never ends the swarm. A bad WURK_HEALTH_PORT now fails the parent's boot. A swarm co-hosted in a web process only reports, once per window, and keeps running.
  • Fetch-loop errors reach error_handlers at most once a minute per exception class per process, with context[:suppressed]. Any handler benefits. The default handler's log volume for this path drops too. wurk_fetch_errors_total still counts every error.
  • The swarm child no longer calls ActiveRecord::Base.establish_connection. Rails' fork hook already rebuilds the pools from the parent's config. The bare call re-derived config from DATABASE_URL, silently switching a host's custom database or pool. Upgrade note: a host relying on the child reading DATABASE_URL afresh, which differs from what the parent established, now gets the parent's config.
  • An ActiveRecord pool smaller than concurrency logs at ERROR, was WARN.
  • wurkswarm loads a Rails target's config/application.rb before the WURK_PRELOAD group Bundler.require, so gems that expect Rails at require time load. This applies to -r <app dir> and -r config/environment.rb. The latter now also defines the ActiveJob adapter first.
  • A web process that forks a swarm in production logs a WARN unless WURK_EMBED=1. The default is unchanged.
  • Upgrading: reclaim and dead-set behaviour. A job that previously looped now lands in the dead set. Watch dead-set growth after upgrading if you had such payloads: they were pegging a thread before. Fetcher::PrivateListKey.parse_owner is gone (internal, nothing else used it). Fetcher::OrphanGrace has a new constructor (internal). Reclaim after a host loss or pod restart is slower (see the figures), on purpose. super_fetch:reaper:full is no longer written and expires on its own. Reaper#reclaim_full!, FULL_INTERVAL, FULL_LOCK_KEY and full_interval:/full_lock_key: are gone (internal).
  • Known issue, Dragonfly at maxmemory: on 1.34 and 2.0, between shards, LMOVE/BLMOVE on the fetch path can pop a job and drop the push, so a job runs with no private copy, and is gone if the process then dies or its retry is refused (2 of 16 in a test on 2.0.2). Do not let a Dragonfly job store reach maxmemory: alert well below it. See docs/reliability.md.
  • Under Rails, every job now runs inside Rails.application.reloader, in every boot mode (railtie swarm, wurk, wurkswarm, embedded). Before this, nothing wired it, despite the docs saying otherwise: ActiveSupport::CurrentAttributes set by one job was visible to the next on the same thread, a raw ActiveRecord::Base.connection stayed leased after the job, the query cache was off and development never reloaded code. Check these when upgrading: the query cache is now on inside a job, so a polling loop can read stale rows; a pool smaller than concurrency now waits and then raises ActiveRecord::ConnectionTimeoutError under contention rather than having threads hold on to connections; Rails.error subscribers now see job failures tagged job.sidekiq, including Wurk::JobRetry::Skip and Handled (spec §14 places the retrier inside the reloader), so filter those in any error tracker subscribed to Rails.error; in development, reload waits for running jobs. Set config[:reloader] = nil to opt out.
  • Under the railtie, workers start after all host after_initialize hooks. Boot moved to a last Finisher initializer, so a Redis URL set in an after_initialize hook now applies, and a forked child has the routes and reload hooks Rails installs after those hooks fire. The relative boot order is unchanged.
  • Sidekiq::Worker and Sidekiq::Job are the same module, as they are upstream, so is_a? and prepend work on either name.
  • The release lane separates what runs third-party code from what can publish. build runs the bundle, bun install and the suite with no credential; publish alone holds id-token: write and runs only pinned actions; release alone holds contents: write. The unpinned gem exec rubygems-await is gone.
  • A release that fails after gem push can be finished by a re-run. Preflight reads the gem, the tag and the GitHub Release, and a live gem with the rest missing skips the push and finishes, instead of reporting green having done nothing. A tag on another commit is now refused before the push. RELEASE.md has the full table.
  • deploy-demo proves the built commit is serving. The image carries its commit, the demo reports it at /demo/build, and a frozen rollout fails the run. It used to compare gem versions, which matched on the first attempt on any push without a version bump.
  • The demo's dependencies are patched and watched. demo/Gemfile.lock moves off advisories in rack, crass, sqlite3, loofah, nokogiri and rails-html-sanitizer; Dependabot covers /demo, and a weekly non-required audit workflow runs bundle-audit and bun audit.

Fixed

  • A limiter built at class load no longer exits the process when Redis is down, LOADING or OOM. Registration is deferred to its next operation, with a WARN.
  • A failed requeue can no longer lose a job on Dragonfly. Dragonfly at maxmemory does not abort a MULTI or a script on a refused write, so UnitOfWork#requeue, bulk_requeue and the...
Read more

wurk v1.9.0

Choose a tag to compare

@github-actions github-actions released this 29 Oct 21:22
81bfcc1

Changed

  • concurrent-ruby is no longer a runtime dependency. wurk.gemspec dropped spec.add_dependency 'concurrent-ruby', '~> 1.2': Wurk never required it directly. An app that pulls it in through Rails or another gem keeps it; nothing else in the dependency set moved (#792).
  • Dashboard metrics history is 30 days, not 3. Wurk::Metrics::History::MID_TERM moved from 3 days to 30 days, so every 10-minute j|… bucket now lives ten times longer — up to 4320 keys per 30-day window instead of 432 per 3 days — and the dashboard's 7d/30d tabs are backed by data end-to-end. Sidekiq's ExecutionTracker keeps the 3-day TTL upstream; this is a deliberate divergence, and retention alone changes no key, field, or score, so a mixed Sidekiq/Wurk fleet still reads the same buckets. With it, the class-level Wurk::Metrics::Query.top_jobs/for_job cap moved from MAX_HOURS = 72 to MAX_HOURS = 720, and a minutes: window past 480 promotes onto the 10-minute buckets instead of raising — only a window past 43200 minutes raises WindowTooWide. The Sidekiq-compatible instance API (Sidekiq::Metrics::Query#top_jobs/#for_job) is unchanged: it still clamps silently at 480 minutes / 72 hours, exactly as upstream, and the parity oracle pins it. GET /api/metrics?minutes=… now serves the requested window out to 30 days and answers 400 past it (it used to clamp silently to 480 minutes), and the SPA requests the 7d/30d tabs as ?hours=168/?hours=720 with the response echoing both hours and minutes so the label matches the served period.
  • The demo's error reporting is opt-in and sanitized. Sentry turns on only when WURK_DEMO_REPORT_ERRORS=1 and SENTRY_DSN are both set; with either one missing the demo never loads the SDK, never initialises Sentry and never installs Wurk::Sentry, so Wurk's server middleware and error handlers stay exactly as they were. Events from the demo's two deliberately-failing jobs (BrokenJob, FlakyWebhookJob) and soak-tagged events are dropped, and request headers, cookies and job arguments are scrubbed of credential- and personal-data-shaped material before an event leaves the demo. The rules live in the demo app (demo/lib/demo_sentry_policy.rb), not in the gem: the gem ships the integration, the demo decides what it may send.

Fixed

  • The reaper's stop now bounds its sweep to the list it is already draining. Reaper#stop waits for a running sweep only up to its bounded join timeout, and a sweep that outlasts that bound is deliberately left alive to wind down — still tracked, so a restart cannot start a second sweep alongside it. That sweep now finishes the one private list it is already draining (those jobs are still put back on their queue for a live worker to pick up: at-least-once, so nothing is lost even though a job may run twice), starts no further list, and makes at most one more Redis round trip instead of walking the rest of the keyspace. A stopping worker therefore no longer keeps requeueing orphans or holding the reaper's sweep locks past its shutdown. The reaper's cadence and grace period are unchanged, and nothing on the wire moves: no Redis key, job-JSON field or score format changed.

  • Re-registering the auto-registered Wurk::Batch::ServerMiddleware no longer moves it inside Expiry and the Limiter. Adding the batch middleware to the server chain again — a configure_server block, a test helper, a later add — now keeps it outside Wurk::Middleware::Expiry, which skips an expired job, and outside Wurk::Limiter::ServerMiddleware, which reschedules an over-limit job. An expired job is therefore counted as its batch's success instead of stranding the batch: the batch's pending count drops, the batch can complete, and its :complete and :success callbacks fire rather than the batch staying open forever. An over-limit job is no longer booked as a batch failure for a job that never ran. A middleware class may now declare an outer_of anchor — the classes it must stay outside of — and Chain#add places it before the first of them that is registered, keeping the slot it already holds if it is already outside them; a class that declares no anchor keeps Sidekiq's documented append (spec §10.1), and prepend, insert_before, insert_after and remove still relocate middleware deliberately, as before. The default server chain order is unchanged, and nothing on the wire moves: no Redis key, job-JSON field or score format changed.

  • A promoted minutes: window no longer rounds up to whole hours. The class-level dashboard API (Wurk::Metrics::Query.top_jobs / for_job) promotes a minutes: window past 480 onto the 10-minute buckets, but computed the promoted window as ceil(minutes / 60) hours and read hours × 6 of them — so minutes: 490 served 9 hours (54 buckets) while /api/metrics echoed 490, and the top_jobs totals included up to 50 extra minutes of executions that fell outside the window the caller asked for. The promotion now counts 10-minute buckets directly (ceil(minutes / 10); 490 minutes is exactly 49), so the served data stays inside the echoed window up to the trailing partial bucket. Explicit hours: windows are unchanged.

  • The demo's error-reporting scrubber no longer passes a serialized cookie jar through untouched. scrub_request_field returned early for any header or cookie field that was not a Hash, so an event whose request carried its cookies as a String ("session=abc123" — the shape a raw request env or a reconstructed event arrives in) was sent with the session value intact. A populated field in any shape but a Hash now fails closed like the rest of the policy: a per-entry scrub has no entries to work on, so nothing inside such a field can be vouched for and the event is dropped rather than sent half-scrubbed. An empty one still passes — it holds nothing.

  • push_bulk no longer silently normalizes symbol-keyed option hashes. Sidekiq::Client#push has always raised ArgumentError on symbol keys (spec §31.1) but the bulk path stringified them and pushed, so {class: MyJob, args: [[1]]} quietly worked while the single-job push did not — two entry points, two answers on what a valid push is. push_bulk now rejects any Symbol key, whole or mixed with String keys, with ArgumentError before any Redis write, matching push exactly. String-keyed bulks are unchanged: perform_bulk, the ActiveJob adapter's bulk path and the API's bulk endpoint all hand push_bulk String keys.

  • The release lane now classifies a GitHub Release's state explicitly instead of guessing at it. The scheduled release-due checkup read a Release as gem-less when its author was developerz-ai[bot] OR it carried zero assets, so a sanctioned break-glass manual release — a developerz-ai[bot]-authored Release with the .gem attached — was reported as a defect and aborted the checkup before the diff window was computed. The trigger is now the asset count: zero assets is the gem-less marker (and the only state the release-deletion runbook applies to), while a Release that carries the .gem is a real release whoever authored it. The author is read for context only. When no Release can be read at all, the state is left unclassified rather than collapsed into one undifferentiated error — a release.yml preflight no-op (the version is already on RubyGems), a break-glass publication, and a failure before the Release was created all stay open until the RubyGems per-version probe and the last release.yml run corroborate which one it was. RubyGems, not the absence of a Release, is the authority on publication.

  • A zero-asset GitHub Release no longer counts as proof that no gem shipped. release.yml pushes the gem before it creates the Release and uploads the .gem asset, so a succeeded push whose asset upload failed leaves zero assets on a version RubyGems has already burned. The release-due brief's gem-less diagnostic and prose now require the RubyGems per-version probe to confirm non-publication before the release-deletion runbook is named — the precondition the "Release states" table already put on the zero-asset row.

  • The Wurk::Queue pause/unpause docs no longer promise an unconditional true. The "Pausing a queue" section of docs/api.md claimed that both calls "return true whether or not they changed anything", but Wurk::Queue#pause! and #unpause! return true only when the call changed the queue's paused state — pause! when it added the queue to the paused SET, unpause! when it removed it — and false when there was no transition. That is the Sidekiq Pro behaviour, and it is what the method table in the same file already described. Documentation-only: no runtime behaviour changed, and the return values are still the SADD/SREM state-change booleans the unit and parity suites assert.

  • The migration guide's b-<bid>* compatibility row no longer promises unqualified Pro batch-schema compatibility. The b-<bid>*, batches row in §4 of docs/migrate-from-sidekiq.md read "✅ Pro batch schema", contradicting the guide's own §9 cutover warning: Wurk's b-<bid>-success/b-<bid>-complete markers mean "already fired" (callbacks enqueued, not necessarily executed), while Sidekiq Pro §2.8 describes the same keys as "callbacks pending", so a Pro batch in flight at the swap may miss its callbacks. The row now names that limitation and points at the §9 cutover warning, and §4's prose says the same in its own words. Documentation-only: no runtime behaviour changed.

  • The Pro §3.2 spec no longer says the processes membership SET expires. The bullet read "Process death: heartbeat key processes expires after 60s", but processes is a TTL-less membership SET: the 60s TTL belongs to the per-identity heartbeat hash <identity>, and liveness is judged by existence — an identity is alive only while its heartbeat hash still exists, so a SET member whose hash ha...

Read more

wurk v1.8.1

Choose a tag to compare

@github-actions github-actions released this 16 Oct 18:40
c268490

Fixed

  • rails runner, rails console and other one-off commands no longer fork a swarm under puma >= 6. puma's lib/puma.rb registers autoload :Launcher, and defined?(::Puma::Launcher) reports an unloaded autoload as defined, so every process with puma in its Gemfile counted as a running server. The server check now requires the launcher to actually be loaded.