Fingerprint and Job Replacement

[pconstantinou]

I want to be able to Enqueue a job and then override it with a newer payload and an updated RunAt time.

It looks like, using the Fingerprint mechanism, I could define a Job as a replacement for another job and still update the payload. I can assign a Fingerprint so the hash of the Payload doesn't have to be the unique identifier for the job (great).

In the specific use case, I'd like to trigger a long-running remote process with some data. As soon as the data comes in, I trigger the job to start in 45 seconds, but if more data comes in, I want to update the payload and only have the most recent queued item handled.

Looking at the Postgres code, it looks like the first Job wins. I want the last job to win.

One solution would be to add a bool to the Job struct that would identify it as overriding and allow the INSERT to have an ON CONFLICT clause. This could be done without modifying the data storage. This may introduce some concurrency issues. I haven't read through all the code to be sure.

[acaloiaro]

Hi Philip, thanks for writing in. I've been meaning to overhaul the fingerprint mechanism due to some issues that I've personally had with it, so thanks for providing some additional feedback.

What you describe makes sense, and it combines well with something else I'd like to support: Job cancellation. What you describe, I consider a special form of job preemption, which I like.

Let me mull this one over an see what I can come up with. I'd like to support job preemption and cancellation based on the same underlying functionality.