Discarding a job of ContinuosJob when same job is already in the queue when using recurring.yml

[rajeevriitm]

I have a simple job in rails that's run every 15 minutes.I am using the default Solid queue setup for jobs.

class ContinuousSearchJob < ApplicationJob
  def perform
    keyword = Keyword.first
    if keyword
      keyword.insert_playlists
    end
  end
end

I have the recurring.yml to run it every 15 minutes. The issue I face is that , sometimes the job could take very long and longer than 15 minutes. And another ContinuousJob would again queue at that point which I dont want. If one ContinuousJob is in queue or running , I want to discard any following. There is an option to block such jobs for sometime. But there is no way to discard. How can I achieve this?

The only solution I could come up with is to destroy all in queue ContinuousJob at the end of a running job if any exist.

Something like this logic
SolidQueue.where(class: "ContinuousJob", status: "pending").discard
But I am unable to find any ways to get all the pending jobs and discard them. Any help would be appreciated.

[albertski]

Perhaps you can use Concurrency controls to prevent the issue.

class ContinuousSearchJob < ApplicationJob
  limits_concurrency to: 1, key: 'ContinuousSearchJob',  duration: 300

  def perform
    keyword = Keyword.first
    if keyword
      keyword.insert_playlists
    end
  end
end

[rajeevriitm]

@albertski But that would only delay the job until the existing job is executed right? What i want is to not queue a job at all when another one of the same class is already in queue or running.

[BenoitMC]

Hi,

I'm interested in the same feature.

I found this workaround:

class UniqueJob < ApplicationJob
  limits_concurrency key: name
  before_enqueue { throw :abort if SolidQueue::Job.where(concurrency_key:).any? }

  def perform
    # ...
  end
end

It works because I use SolidQueue.preserve_finished_jobs = false, but if you preserve jobs, you can probably simply add a where(finished_at: nil).

[rajeevriitm]

Hi @BenoitMC
This is what I was looking for as well. But one issue that propped up during testing is, this would count the Jobs that failed as well. Is there any way to just count jobs that are in queue or running? finished_at count errored as well.

[BenoitMC]

@rajeevriitm Try to add .where.missing(:failed_execution).

[rajeevriitm]

@BenoitMC
Thanks. I removed limits_concurrency and used this check which seems to do the job.
SolidQueue::Job.where(class_name: "ContinuousSearchJob",finished_at: nil).where.missing(:failed_execution).
Do you think there would be any issues with this?

[BenoitMC]

I think it would work just as well using class_name.

Keeping limits_concurrency might be useful if the way you enqueue jobs is subject to race conditions. Solid Queue will prevent the second job from running while the first job is still in progress if two jobs were enqueued accidentally.