How to use limiter and job#jobId together?

[michael-land]

Seems like bullmq always append limiter#groupKey to the job's id.

Is it possible to define a rate limiter based on group keys and custom job id?

For example

const jobData = [
  {groupKey:"group1", name:"item1"},
  {groupKey:"group2", name:"item2"},
  {groupKey:"group2", name:"item1"}
];


const queue = new Queue('queue1', {
  limiter: { groupKey: 'groupKey' },
})

const worker = new Worker('queue1', {
  concurrency: 20,
  limiter: { duration: 1000, max: 5, groupKey: 'groupKey' },
})

for (const item of jobData) {
  await queue.addJob('jobName', { jobId: item.name })
}

The above code snippet will product jobs with id

name1:group1
name2:group2
name1:group2

expected job id

name1
name2

My goal is each unique job id run only once regardless the group

[manast]

That would not be possible with current implementation that requires modifying the jobId in order to pass the group information to the workers. However you may be interested in the Pro version that implements groups in a different way: https://docs.bullmq.io/bullmq-pro/groups

[michael-land]

pricing for pro?

[manast]

Please refer to this page: https://docs.bullmq.io/bullmq-pro/introduction#pricing

[michael-land]

Thanks, that's too expensive for individual developers like me, I will solve it in application level tho.