concurrent-ruby 1.1.10 spikes volume of jobs

[ghost]

Describe the bug
When using concurrent-ruby 1.1.10, the gem does not behave as expected. For the application we are implementing in, after upgrading to sidekiq-unique-jobs 7.1.16 we started getting alerts about a higher than normal volume of calls going out to a service that was approaching our rate limit. Attempting the upgrade again with concurrent-ruby dropped to 1.1.9 had no issues. It looks like the fix applied in #688 for #697 has some unexpected consequences.

Expected behavior
Jobs would be unique, allowing for a consistent pattern of calls to hold.

Current behavior
Jobs spike in volume, suggesting some non-unique jobs are making it through.

Worker class

class MyWorker
  include Sidekiq::Worker
  sidekiq_options queue: :default,
    retry: false,
    backtrace: true,
    lock: :until_executed,
    on_conflict: :log,
    log_duplicate_payload: !Rails.env.test?

  def perform(args); end
end

Additional context
I noticed the upgrade contained separate logic based on the version of concurrent-ruby used, so it makes sense that downgrading it fixed it, I haven't looked deep enough to see what could be wrong though.

[mhenrixon]

I had a feeling it wouldn’t be so easy. Shame they removed the timeouts instead of using my version.

I’ll copy the previous version of the timer task and use that instead.

Thank you for reporting.

[mhenrixon]

#702 I wish they would have taken my suggestion and fixed the issue instead of making the issue permanent...

[mhenrixon]

Released as v7.1.19 https://github.com/mhenrixon/sidekiq-unique-jobs/releases/tag/v7.1.19

[ghost]

Something seems to still be wonky with the way this gem performs on the newest concurrent-ruby. On the below chart you can see our job patterns get really wonky and don't have the natural hourly drop in volume like they normally do when concurrent-ruby is at 1.1.10:

Through my testing this only happens if concurrent-ruby is 1.1.10 and sidekiq-unique-jobs is 7.1.16 or greater. Curiously it does not occur when sidekiq-unique-jobs 7.1.15 is paired with concurrent-ruby 1.1.10 or less.

[mhenrixon]

I hear you, I'll write my own timeout functionality then and hopefully it won't matter which concurrent version you are having.

Frigging dependencies aye?

[ghost]

Really appreciate that!

Frigging dependencies aye?

Haha yup

[mhenrixon]

@oneandonlymike see if v7.1.20 works better for you

[ghost]

v7.1.20 seems to have done the trick! Thanks!