Possible thread/memory leak

[mperham]

When a Sidekiq job fails, Sidekiq creates a retry in Redis but it also throws away the entire Processor object and Thread to ensure all jobs execute with "known good state".

A Customer reports memory leakage after upgrading from Ruby 3.4 to 4.0, we suspect due to new Processor threads continuing to hold reference to old Processor threads. Their investigation led to the "fix" in commit 2ada6ab. After deploying the fix, they saw the attached memory graph showing much lower memory usage.

[mperham]

Fixed by 2ada6ab

[mperham]

To be clear, I still do not understand the root cause. The customer debugged their heap and suggested the fix. This may be a bug introduced in Ruby 4.0.

[navidemad]

Hi @mperham — thanks for the quick patch. Posting a verified end-to-end walkthrough of the call chain plus a heap-dump confirmation of the retention edge, in case it's useful for the 8.1.6 release notes.

TL;DR

Pre-fix, Manager#processor_result returns the value of @plock.synchronize { … }, which on the failure path is p.start — i.e. the freshly started replacement Thread. That value propagates back up through Processor#run → Component#safe_thread's Thread.new block and ends up as the dying thread's Thread#value. So after every job failure:

dying T_old #value = T_new (alive)

The + nil in 2ada6ab cuts that propagation. On shutdown paths the if !@done && @count > @workers.size branch in processor_result is skipped, so synchronize returns nil and the chain is never built — pre-fix and post-fix behave identically there. The leak fires only on uncaught job exceptions.

Heap-dump confirmation

Reproduced the chain pattern on Ruby 4.0.3, dumped the heap with ObjectSpace.dump_all, and categorised every reference emanating from each surviving Thread. The defining edge — Thread → Thread — appears exactly +1 time on chain-link Threads vs baseline:

Thread	Status	total refs	outgoing Thread → Thread refs
VM system thread	—	7	0
live worker (mid-replace_self)	live	8	1 (frame local → next Thread)
live Manager.@worker (fresh)	live	7	0
chain-link Threads	dead	8 each	1 each (th->value → next Thread)

The +1 is the leak. The nil patch removes that edge: each dead Thread's value becomes nil, breaking the dead-to-dead link, so each Processor is reachable only via the live Manager.@workers slot and becomes collectable as soon as it's replaced.

The chain structure is the same in both Ruby 3.4.9 and 4.0.3 dumps — the bug shape is Ruby-version-independent; only how aggressively the GC reclaims it changes between versions.

Mitigation while waiting for 8.1.6

For Ruby 4.0.x apps on Sidekiq 8.1.5 with non-trivial retry rates:

# config/initializers/sidekiq_thread_leak_patch.rb
# Backport of 2ada6ab5 (Sidekiq #7006) until 8.1.6 is released.
# Remove once sidekiq >= 8.1.6 is deployed.
# Must be loaded at boot — before any Sidekiq::Manager is constructed.
if Gem::Version.new(Sidekiq::VERSION) < Gem::Version.new("8.1.6")
  require "sidekiq/manager"
  Sidekiq::Manager.prepend(
    Module.new do
      def processor_result(processor, reason = nil)
        super
        nil
      end
    end,
  )
end

Ruby-side root cause

Not isolated by this writeup. The chain forms in both 3.4 and 4.0 (heap dumps confirm), but neither a synthetic Thread.new-chain repro nor a self-contained Sidekiq-with-failing-jobs repro reproduces the production magnitude of the leak across versions. Your read still applies: "I still do not understand the root cause". The patch is mechanism-agnostic regardless — it removes the only Sidekiq-controlled edge that lets the chain form, so whatever the underlying Ruby behaviour is, Sidekiq is no longer the source.

Full call-chain walkthrough (v8.1.5 sources, line by line)

Step 1 — Manager#processor_result returns the new Thread

# lib/sidekiq/manager.rb (v8.1.5)
def processor_result(processor, reason = nil)
  @plock.synchronize do
    @workers.delete(processor)
    if !@done && @count > @workers.size
      p = Processor.new(@config, &method(:processor_result))
      @workers << p
      p.start                # ← last expr of the `if`-body
    end                      # ← the `if` is itself the last expr of the `synchronize` block
  end                        # ← last expr of the method
end

p.start:

# lib/sidekiq/processor.rb
def start
  @thread ||= safe_thread("#{config.name}/processor", &method(:run))
end

@thread ||= Thread returns the freshly assigned Thread. So p.start → Thread (T_new). So synchronize → T_new. So processor_result → T_new.

Step 2 — The dying thread takes that value as its Thread#value

processor_result is invoked via @callback.call(...) from the dying thread T_old's Processor#run:

# lib/sidekiq/processor.rb
def run
  Thread.current[:sidekiq_capsule] = @capsule
  process_one until @done
  @callback.call(self)
rescue Sidekiq::Shutdown
  @callback.call(self)
rescue Exception => ex
  @callback.call(self, ex)   # ← last expr of the rescue
end

On job failure, @callback.call(self, ex) is the last expression of the rescue → run returns T_new.

run is itself the last expression of the block passed to Thread.new inside safe_thread:

# lib/sidekiq/component.rb
def safe_thread(name, priority: nil, &block)
  Thread.new do
    Thread.current.name = "sidekiq.#{name}"
    watchdog(name, &block)   # ← last expr of the Thread.new block
  end.tap { |t| t.priority = ... }
end

def watchdog(last_words)
  yield                       # ← returns whatever the block returns
rescue Exception => ex
  handle_exception(ex, {context: last_words})
  raise ex
end

watchdog returns the value of yield. So the Thread.new do … end block returns T_new. Therefore T_old.value = T_new.

Step 3 — The fix breaks the chain

   def processor_result(processor, reason = nil)
     @plock.synchronize do
       @workers.delete(processor)
       if !@done && @count > @workers.size
         p = Processor.new(@config, &method(:processor_result))
         @workers << p
         p.start
       end
     end
+    nil
   end

processor_result now returns nil. The chain becomes T_old.value = nil at every link. T_new is referenced only via the legitimate path Manager → @workers → P_new → @thread.

When the callback fires (and why the leak is narrower than "any callback")

From Processor#run (lib/sidekiq/processor.rb lines 78–82):

Situation	Triggers callback?
Job success	No — process_one until @done loop continues
Uncaught exception (re-raised by JobRetry)	Yes — rescue Exception => ex
Sidekiq::Shutdown	Yes — only on shutdown
@done = true (graceful exit)	Yes — only on shutdown

The leak is proportional to the rate of uncaught job exceptions (typically JobRetry re-raising after pushing the retry into Redis), not to overall throughput.

Even narrower: replacement only happens when the if !@done && @count > @workers.size branch in processor_result actually fires. On shutdown/graceful paths Manager's @done is already true, the if body is skipped, synchronize returns nil, and the chain is never built — even pre-fix.

Why Thread#value retains across GC (CRuby source)

Thread#value is persistent. CRuby stores the block's return value in the Thread C struct (th->value) when the block finishes, and Thread#value reads from that field forever after. From thread.c at v4.0.3:

// line 712 — block returned normally
th->value = result;
// line 753 — block raised
th->value = Qnil;
// line 1262 — Thread#value reads the field
static VALUE
thread_value(VALUE self)
{
    rb_thread_t *th = rb_thread_ptr(self);
    thread_join(th, Qnil, 0);
    if (UNDEF_P(th->value)) return Qnil;
    return th->value;
}

th->value is GC-marked by the standard Thread mark function, so as long as a dead Thread object is reachable, its #value strongly retains whatever the block returned — in the pre-fix Sidekiq case, the next live Thread.

Ruby-side investigation (unconvincing leads, included for completeness)

Read CRuby's thread-lifecycle code at v4.0.3 and surveyed commits between v3.4.0 and v4.0.0 (~330 commits in thread.c / thread_pthread.c / cont.c / vm.c / gc.c / ractor.c). No confirmed culprit. Recording what was checked so the next person can pick up from a known state:

• M:N scheduler zombie list: thread_pthread.c adds terminating threads to vm->ractor.sched.zombie_threads, and rb_thread_sched_mark_zombies GC-marks them so they survive collection until sched.finished is set. Path is gated by !th->sched.malloc_stack, and main-ractor threads take the dedicated-NT branch (native_thread_create_dedicated, malloc_stack = true). Sidekiq runs in the main ractor with the default RUBY_MN_THREADS, so the zombie mechanism shouldn't apply. Probably not the cause.
• Living-threads set: rb_ractor_living_threads_remove (ractor.c) does ccan_list_del on thread termination, so the VM's running-thread list isn't holding dead threads.
• thread_mark diff v3.4.0 → v4.0.0: only the removal of the unused stat_insn_usage field. The chain edge (rb_gc_mark(th->value)) is unchanged. Whatever regressed at runtime is not a new mark edge.
• 41-commit diff v4.0.3 → v4.0.5: one fiber/thread-lifecycle candidate — 2709a3ddd1 "Ensure fiber stack is freed in all cases, if the fiber is terminated" (Bug #21955) — but its repro is Fiber#transfer-based, not Thread-based, so not a clean match for Sidekiq.
• Two commits between v3.4.0 and v4.0.0 worth a closer look by someone with a working reproducer: 62430c19c9 (adds mutex_mark that marks the owning Thread while a Mutex is locked — relevant because Manager#processor_result runs inside @plock.synchronize) and 1d4822a175 (changes how value propagates through Ractor channels in shared thread code paths). Both are speculative.

A targeted git bisect would settle it, but it needs a binary predicate stable across reruns — neither the synthetic chain repro nor the self-contained Sidekiq repro (both in the appendices below) reproduces the production magnitude in my hands. At 5 000 failing jobs on the Sidekiq repro the RSS delta is in the 5–10 MB noise band across 3.4.9 ⇄ 4.0.3 × BUGGY ⇄ FIXED; at 50 000 jobs on 4.0.3 BUGGY the RSS actually decreased by 15 MB. The most actionable next step would be paired heap dumps from a leaking 4.0.3 production worker and a healthy 3.4 production worker — that would expose the actual retention edge that differs between versions, which a microbench cannot.

Repro scripts (the ones used to produce the table above)

repro_v2.rb — Sidekiq-shaped Manager/worker chain. Numbers in the table come from this one.

# Closer-to-Sidekiq reproduction of issue #7006.
#
# Mimics the Manager → Processor lifecycle:
#   - A "manager" holds the live worker in @workers.
#   - When the live worker dies, the worker's Thread block returns the
#     newly-spawned replacement Thread (the bug-shaped propagation).
#   - The manager swaps in the replacement and discards the dead worker.

require "objspace"

ITERATIONS = (ENV["ITERATIONS"] || "10000").to_i

class Manager
  attr_reader :worker
  def initialize
    @worker = spawn_worker
  end

  def spawn_worker
    Thread.new(self) do |mgr|
      # When this worker "dies", it does what Sidekiq's processor_result chain
      # does: spawn the replacement and return *that Thread* as the block's
      # last expression. (Replicates the missing trailing `nil`.)
      replacement = mgr.replace_self
      replacement   # ← last expression: the Thread.new block returns the new Thread
    end
  end

  def replace_self
    new_thread = spawn_worker
    @worker = new_thread  # legitimate reference path (analogous to @workers << p)
    new_thread
  end
end

3.times { GC.start(full_mark: true, immediate_sweep: true) }
baseline_threads = ObjectSpace.each_object(Thread).count
baseline_rss = `ps -o rss= -p #{Process.pid}`.strip.to_i

mgr = Manager.new
ITERATIONS.times do
  current = mgr.worker
  current.join
end
mgr.worker.join rescue nil
3.times { GC.start(full_mark: true, immediate_sweep: true) }

after_threads = ObjectSpace.each_object(Thread).count
after_rss = `ps -o rss= -p #{Process.pid}`.strip.to_i
dead = ObjectSpace.each_object(Thread).select { |t| !t.alive? && t != Thread.main }

puts "ruby:           #{RUBY_DESCRIPTION}"
puts "iterations:     #{ITERATIONS}"
puts "Threads (objs): baseline=#{baseline_threads}  after=#{after_threads}  delta=#{after_threads - baseline_threads}"
puts "RSS KB:         baseline=#{baseline_rss}      after=#{after_rss}     delta=#{after_rss - baseline_rss}"
puts "dead reachable: #{dead.size}"
puts "Thread.list:    size=#{Thread.list.size}"

Run with mise (or your Ruby manager of choice):

mise exec ruby@3.4.9 -- ruby repro_v2.rb
mise exec ruby@4.0.3 -- ruby repro_v2.rb
mise exec ruby@4.0.5 -- ruby repro_v2.rb

repro_v3.rb — same shape but with a FIXED=1 env switch to compare buggy vs patched return value under the same Ruby. (Numbers were within noise on 4.0.3, so this didn't end up in the table — included here so others can re-test.)

require "objspace"

ITERATIONS = (ENV["ITERATIONS"] || "10000").to_i
FIXED      = ENV["FIXED"] == "1"

class Manager
  attr_reader :worker

  def initialize
    @worker = spawn_worker
  end

  def spawn_worker
    Thread.new(self) { |mgr| mgr.replace_self }
  end

  # Buggy: returns the new Thread (matches Sidekiq pre-fix processor_result).
  # Fixed: returns nil (matches the 2ada6ab5 patch).
  def replace_self
    new_thread = spawn_worker
    @worker = new_thread
    FIXED ? nil : new_thread
  end
end

3.times { GC.start(full_mark: true, immediate_sweep: true) }
baseline_rss = `ps -o rss= -p #{Process.pid}`.strip.to_i

mgr = Manager.new
ITERATIONS.times { mgr.worker.join }
mgr.worker.join rescue nil
3.times { GC.start(full_mark: true, immediate_sweep: true) }

after_rss = `ps -o rss= -p #{Process.pid}`.strip.to_i
dead = ObjectSpace.each_object(Thread).select { |t| !t.alive? && t != Thread.main }.size

puts "ruby:           #{RUBY_DESCRIPTION}"
puts "mode:           #{FIXED ? 'FIXED (returns nil)' : 'BUGGY (returns new Thread)'}"
puts "iterations:     #{ITERATIONS}"
puts "RSS KB delta:   #{after_rss - baseline_rss}"
puts "dead reachable: #{dead}"

FIXED=0 mise exec ruby@4.0.3 -- ruby repro_v3.rb
FIXED=1 mise exec ruby@4.0.3 -- ruby repro_v3.rb

Heap-dump artifacts (the ones used to confirm the retention edge)

heap_dump_repro.rb — runs the chain pattern and dumps the heap via ObjectSpace.dump_all.

require "objspace"
require "json"

ITERATIONS = (ENV["ITERATIONS"] || "500").to_i

class Manager
  attr_reader :worker
  def initialize
    @worker = spawn_worker
  end

  def spawn_worker
    Thread.new(self) do |mgr|
      replacement = mgr.replace_self
      replacement # block last expr → next Thread (the leak shape)
    end
  end

  def replace_self
    new_thread = spawn_worker
    @worker = new_thread
    new_thread
  end
end

# Allocation tracing gives us file/line for every object's birth.
ObjectSpace.trace_object_allocations_start

mgr = Manager.new
ITERATIONS.times { mgr.worker.join }
mgr.worker.join rescue nil
3.times { GC.start(full_mark: true, immediate_sweep: true) }

ObjectSpace.trace_object_allocations_stop

out_path = "/tmp/sidekiq-preview/heap.#{RUBY_VERSION.tr(".", "_")}.json"
File.open(out_path, "w") { |f| ObjectSpace.dump_all(output: f) }
puts "ruby: #{RUBY_DESCRIPTION}"
puts "dump: #{out_path} (#{File.size(out_path) / 1024 / 1024} MB)"

analyze_heap.rb — indexes the dump, builds a reverse-reference map, categorises what each Thread points to and is pointed by.

require "json"
dump_path = ARGV[0] || abort("usage: ruby analyze_heap.rb <heap.json>")

by_addr = {}
File.foreach(dump_path) do |l|
  o = JSON.parse(l) rescue next
  by_addr[o["address"]] = o if o["address"]
end

referrers = Hash.new { |h, k| h[k] = [] }
by_addr.each do |addr, obj|
  Array(obj["references"]).each { |r| referrers[r] << addr }
end

# Threads are DATA objects with struct VM/thread.
threads = by_addr.values.select { |o| o["type"] == "DATA" && o["struct"] == "VM/thread" }
puts "Threads in heap: #{threads.size}"

threads.each_with_index do |t, i|
  refs = t["references"] || []
  cats = refs.map do |r|
    o = by_addr[r]
    o ? (o["struct"] ? "DATA/#{o["struct"]}" : o["type"]) : "(missing)"
  end
  alive = t["file"] ? "alloc=#{t["file"]}:#{t["line"]}" : "alloc=(none)"
  puts "T#{i + 1} #{t["address"]}  refs=#{refs.size}  refby=#{(referrers[t["address"]] || []).size}  #{alive}"
  puts "    outgoing kinds: #{cats.tally.inspect}"
end

Run:

mise exec ruby@4.0.3 -- ruby heap_dump_repro.rb
mise exec ruby@3.4.9 -- ruby heap_dump_repro.rb
ruby analyze_heap.rb /tmp/sidekiq-preview/heap.4_0_3.json
ruby analyze_heap.rb /tmp/sidekiq-preview/heap.3_4_9.json

The defining signal is that chain-link Threads emit exactly one DATA/VM/thread reference; non-chain Threads (VM system + a freshly-spawned Manager.@worker that hasn't run yet) emit zero.

Self-contained Sidekiq leak repro (Bundler.inline; doesn't yet produce the production-scale leak but is a starting point)

sidekiq_leak_repro.rb — single-file, stands up a real Sidekiq Capsule + Launcher + Manager + Processor on real Redis, pumps deliberately-failing jobs through it, and prints verdict=GOOD|BAD with a binary exit code suitable for git bisect run. In my hands at 500 / 5 000 / 50 000 iterations it does NOT measurably leak on Ruby 4.0.3 vs 3.4.9 — possibly because the chain forms transiently and gets swept before iteration overhead exposes it. Posting it here as a starting harness; someone with a production-like Rails app might be able to drive it past the measurement-noise floor.

Prerequisites:

redis-server --daemonize yes --port 6379

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"
  gem "sidekiq", "8.1.5"
  gem "redis-client"
end

require "sidekiq"
require "sidekiq/launcher"
require "sidekiq/capsule"
require "objspace"

ITERATIONS = (ENV["RUBY_LEAK_ITER"]      || "500").to_i
THRESHOLD  = (ENV["RUBY_LEAK_THRESHOLD"] || "8").to_i
APPLY_FIX  = ENV["RUBY_LEAK_FIXED"] == "1"

if APPLY_FIX
  require "sidekiq/manager"
  Sidekiq::Manager.prepend(
    Module.new do
      def processor_result(processor, reason = nil)
        super
        nil
      end
    end,
  )
end

QUEUE_NAME = "leak_repro_#{Process.pid}".freeze

config = Sidekiq.default_configuration
config.logger.level = Logger::WARN
config.redis = { url: "redis://127.0.0.1:6379/0" }
config.queues = [QUEUE_NAME]
config.concurrency = 1
config[:dead_max_jobs] = 50_000
config[:dead_timeout_in_seconds] = 60 * 60

Sidekiq.configure_client do |cfg|
  cfg.redis = { url: "redis://127.0.0.1:6379/0" }
end

class AlwaysFailingJob
  include Sidekiq::Job
  sidekiq_options queue: QUEUE_NAME, retry: 0
  def perform
    raise "boom"
  end
end

def rss_mb = `ps -o rss= -p #{Process.pid}`.strip.to_i / 1024.0

Sidekiq.redis { |r| r.call("DEL", "queue:#{QUEUE_NAME}") }
ITERATIONS.times { AlwaysFailingJob.perform_async }

launcher = Sidekiq::Launcher.new(Sidekiq.default_configuration)
3.times { GC.start(full_mark: true, immediate_sweep: true) }
baseline_rss = rss_mb
launcher.run(async_beat: false)

deadline = Time.now + 600
loop do
  remaining = Sidekiq.redis { |r| r.call("LLEN", "queue:#{QUEUE_NAME}").to_i }
  break if remaining.zero? || Time.now > deadline
  sleep 0.5
end
sleep 1
launcher.stop
3.times { GC.start(full_mark: true, immediate_sweep: true) }
final_rss = rss_mb
rss_delta = (final_rss - baseline_rss).round(1)

Sidekiq.redis { |r| r.call("DEL", "queue:#{QUEUE_NAME}"); r.call("DEL", "dead") }

verdict = rss_delta > THRESHOLD ? "BAD" : "GOOD"
puts "ruby=#{RUBY_VERSION} sidekiq=#{Sidekiq::VERSION} fixed=#{APPLY_FIX} " \
     "iter=#{ITERATIONS} rss_delta=#{rss_delta}MB verdict=#{verdict}"
exit(rss_delta > THRESHOLD ? 1 : 0)

# Buggy 8.1.5 on 4.0.3
RUBY_LEAK_ITER=5000 mise exec ruby@4.0.3 -- ruby sidekiq_leak_repro.rb
# Patched on 4.0.3
RUBY_LEAK_ITER=5000 RUBY_LEAK_FIXED=1 mise exec ruby@4.0.3 -- ruby sidekiq_leak_repro.rb
# Same on 3.4.9
RUBY_LEAK_ITER=5000 mise exec ruby@3.4.9 -- ruby sidekiq_leak_repro.rb

Limitations of this harness, in my hands:

• 500 / 5 000 / 50 000 iterations all show RSS deltas in the noise band (5–10 MB up or even down).
• The chain forms only transiently during a worker's mid-execute window and is swept once the iteration moves on; the production leak must depend on retention factors this harness doesn't capture (sustained mixed-allocation pressure under real job payloads, multi-capsule configs, long process uptime, RSS fragmentation that doesn't manifest as Ruby heap retention).

[Linuus]

Thanks for including and releasing the fix. It was me who reported and proposed the fix 🙏