Jobs disappears for 10 seconds when Worker is picking it up

[tangopium]

Ruby version: 3.2.2
Rails version: 7.1.2
Sidekiq / Pro / Enterprise version(s): 7.2.0

I'm facing the following issue. Whenever a Sidekiq Worker is picking up a Job from a queue, it take up to 10 seconds until it appears in the "Processing" queue. As the Job gets immediately removed from the queue, the Job basically disappears inbetween.

I noticed, that the Worker reports the status of the jobs with every heartbeat (here). Is there a way to trigger this update immediately?

The disappearance of the job causes several issues (e.g. this).

In order to reproduce the error, I set up a minimal rails app (here). This repo additionally contains sidekiq-unique-jobs, but the described issue appears even without it. To reproduce it, clone the repo and run the following command:

docker compose up app app_worker

Go to http://localhost:3000/sidekiq and then run the following command:

docker compose exec app rails runner "TestJob.perform_async"

You will see in the console, that the worker immediately picks up the job, but in the Web UI it takes up to 10 seconds. I checked the entries in the redis DB as well (as the Web UI is polling the server as well in an interval of about 5 seconds), and it takes as well about 10 seconds, before the job appears as processing there.

[mperham]

Yes, the heartbeat is asynchronous and not meant to be real-time or millisecond accurate. It pushes job status to Redis every 10 seconds.

[tangopium]

Is this a performance consideration? I had already several scenarios in which the wrong status info of a job leads to errors -specially if the job disappears completely. Wouldn't it make sense to update the job status in real-time?

[mperham]

Yes, for performance. If you are running 100 jobs/sec, that's 100 round trips to Redis. So the more work you do, the more it slows down. That's poor scalability.