Job Status

[jmazzi]

There are some jobs that can take a good bit of time to finish. Periodically, staff will want to check on the status of those jobs. We don't necessarily need to know how far it's progressed, just whether or not it's finished, running, or failed.

cc @tfwright

[mperham]

With Sidekiq Pro, you can create a batch of size 1 for that job and set up an email notification to the user. They will be emailed when it succeeds/fails.

Any ways I can improve that?

[jmazzi]

I should give a more specific example.

One of the things this application will be doing is creating a hosting account on a remote server that is running cPanel. This hosting account would belong to a customer who signed up for our services. The staff member would click an activate link that would process the job. The staff member will want to know when that is done so they can take the next action. Sending an email to the staff member is not ideal.

What would be most useful would be to have sidekiq give us a job/batch id when a job is queued. Then, in turn, a way to query that status of the job id. I envision it working a little bit like this.

job_id = Service.activate(service_id)

Sidekiq.job_status(job_id) #=> :queued

# Time passes 
Sidekiq.job_status(job_id) #=> :completed

# Or if something goes wrong
Sidekiq.job_status(job_id) #=> :failed

[mperham]

Ok, how does this look:

batch = Sidekiq::Batch.new
batch.jobs do
  LongWorker.perform_async
end
bid = batch.bid # save the BID somewhere, maybe in the session?

# now use that BID to check up on the job
batch = Sidekiq::Batch::Status.new(bid)
if !batch.complete?
  :queued
elsif batch.failures == 1
  :failed
else
  :completed
end

[jmazzi]

We're going to play with this syntax and see how it fits in.

Curious, would failures ever be more than one? If not, why not use failed??

[tfwright]

Would this mean that if we want to take advantage of batching for these jobs, we wouldn't be able to get the status of individual jobs?

[mperham]

@jmazzi I wrote that from memory, if failed? exists ya totally you can use it.

@tfwright Batches do track the jid and errors of any failed jobs so you can see why a given job failed. You don't know when a particular job has been run, just that the batch is complete or not. If the batch is complete and the jid is not in the batch's failures, you know that particular job succeeded.

[jmazzi]

Personally, I would prefer a way to get the status of a job outside of a batch. While batches seem to have some good uses, it doesn't really fit in line with everything we are doing.

Do you see something like this being added?

[tfwright]

@mperham can you check to see if a jid is in the batch's failures before the latter is complete? If so that should work fine.

[mperham]

@tfwright Yes, Sidekiq::Batch::Status has access to the failure info. I forgot to document it on the wiki page: https://github.com/mperham/sidekiq/wiki/Batches

[mperham]

@jmazzi I think that's a great feature request for Pro. I'll think about it over the next few days.

[jmazzi]

@mperham thank your prompt replies and consideration :)

[jmazzi]

@mperham what did you decide?

[mperham]

I'll add it to Pro sometime soon since I can see it being useful. No promised ETA but it should be pretty easy to implement.

[jmazzi]

😂 👏 😹

[andrusha]

There is a https://github.com/utgarda/sidekiq-status plugin which adds similar functionality, although it seem to be on a very early stage of development.

[mperham]

I've decided against tracking individual job status, in case you couldn't tell from the delay. Batches are Sidekiq's way to implement workflows, whether manual (via notifications) or automated (via success callbacks).