Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resque can't cleanup heartbeats when it's killed before calling register_worker #1750

Closed
mtsmfm opened this issue Jun 27, 2021 · 0 comments · Fixed by #1751
Closed

Resque can't cleanup heartbeats when it's killed before calling register_worker #1750

mtsmfm opened this issue Jun 27, 2021 · 0 comments · Fixed by #1751

Comments

@mtsmfm
Copy link
Contributor

mtsmfm commented Jun 27, 2021

Currently, Resque::DataStore::HEARTBEAT_KEY and :workers are registered independently.
If a worker is killed in

resque/lib/resque/worker.rb

Lines 366 to 367 in be71bd1

prune_dead_workers
run_hook :before_first_fork
, Resque::DataStore::HEARTBEAT_KEY is registered but :workers isn't.

Resque::Worker#prune_dead_workers assumes Worker.all contains such a worker but they're missed.

all_workers = Worker.all

Here's an example to reproduce this problem:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'

  gem 'resque', '2.0.0'
  gem 'rspec', require: 'rspec/autorun'
end

Resque.redis = 'redis:6379'
Resque.redis.redis.flushall

RSpec.describe do
  it do
    Process.fork do
      Resque.before_first_fork do
        while Resque.redis.hlen(Resque::DataStore::HEARTBEAT_KEY) == 0
          sleep 0.1
        end

        Process.kill(:KILL, Process.pid)
      end

      worker = Resque::Worker.new('*')
      worker.prepare
      worker.work
    end

    Process.waitall

    expect(Resque.redis.hlen(Resque::DataStore::HEARTBEAT_KEY)).to eq 1
    expect(Resque.redis.smembers(:workers)).to eq []

    expect {
      worker = Resque::Worker.new('*')
      worker.prune_dead_workers
    }.to change { Resque.redis.hlen(Resque::DataStore::HEARTBEAT_KEY) }.from(1).to(0)
  end
end
$ ruby foo.rb
F

Failures:

  1) is expected to change `Resque.redis.hlen(Resque::DataStore::HEARTBEAT_KEY)` from 1 to 0
     Failure/Error:
       expect {
         worker = Resque::Worker.new('*')
         worker.prune_dead_workers
       }.to change { Resque.redis.hlen(Resque::DataStore::HEARTBEAT_KEY) }.from(1).to(0)
     
       expected `Resque.redis.hlen(Resque::DataStore::HEARTBEAT_KEY)` to have changed from 1 to 0, but did not change
     # foo.rb:34:in `block (2 levels) in <main>'

Finished in 0.1311 seconds (files took 0.06291 seconds to load)
1 example, 1 failure

Failed examples:

rspec foo.rb:14 # is expected to change `Resque.redis.hlen(Resque::DataStore::HEARTBEAT_KEY)` from 1 to 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant