Batching and Uniqueness - Faktory

[rocktavious]

Hello All, (This is a Faktory question and I wasn't sure if this is the right place for this kind of discussion so if i need to take it else where just let me know but this is all i could find)

I have a question about a job design / orchestration problem we are facing. So the high level problem is that we run a job type that is triggered off receiving a git webhook. We then enqueue a batch of jobs that could take a while to finish (in some cases up to 10-20 mins for the entire batch to complete all the sub jobs). What can happen during that time is we receive a git webhook again - maybe even 2 or 3 or 4. Since the jobs in question use the "default branch head" as the clone target we don't need to process the batch for webhooks 2 and 3 but we would like 4 to stay in the queue.

This presents a problem when enqueuing batches since they don't seem to support uniqueness (at least for Faktory) and we'd really like to not have to build something custom that can effectively "debounce" this kind of incoming event and were hoping that there was a way to design our jobs and batches with Faktory primitives to achieve this.

I'm open to any and all suggestions for how to re-architect this so we can limit the amount of work we do. When this scales out to the number of customers X git webhooks we receive it would be WAY too much work and not scale well if we don't debounce / deduplicate these events while the original batch is still being worked.

Please HALP! 😉

[mperham]

There is a Faktory report at contribsys/faktory. Do I not have discussions enabled? If not, open an issue there as this is not Sidekiq related.

[rocktavious]

@mperham - no discussions is not enabled on the contribsys/faktory repo. I've moved this over to here contribsys/faktory#450