Is there a way to retry failed jobs with priority instead of at the end of the queue?

[evilbuck]

Is there a way to make sure the retries are processed before the existing queue?
My use case is using a rate limited queue to smooth out traffic flow to rate limited api's. However, if a particular job fails and should be retried, it ends up at the end of the queue. I would like to prioritize the failures fifo to the original time they were added to the queue.

Update:
I found this related question, #1963 (reply in thread)
However, this is not what I'm observing.

Is there something I'm doing wrong for these expectations?

I created a simple test.

const Bull = require('bull');

const queue = new Bull('test', {
  limiter: {
    max: 1,
    duration: 1000
  }
});

queue.process(async job => {
  let { data } = job;
  let { fail } = data;

  console.log(`x: ${data.x} - ${job.id} - processing`);
  if (fail) {
    throw new Error('test error');
  }
  return { tada: job.id };
});
queue.on('failed', (job, error) => {
  console.log(`x: ${job.data.x} job ${job.id} failed ${error}`);
});


(async () => {
  let myQueue = queue;
  await myQueue.clean({ grace: 9000000 });
  myQueue.add({ fail: true, x: 200 }, { attempts: 2 });

  for (let i = 0; i < 10; i++) {
    myQueue.add({ x: i });
  }
})();

[manast]

Retried jobs and rate limited jobs become delayed jobs so they are kind of competing against each other on arriving at the head of the queue, I think that is the problem you are experiencing.

[evilbuck]

Thanks for the quick response.
That seems about right. Is there a pattern you would recommend short of putting a rate limited queue in front of the processor? I'm reluctant to add more complexity if there is an idiomatic way to handle this within the queue system itself.

[manast]

I do not think this can be resolved with current implementation. I am doing some proof-of-concept improvements implementing them as a native redis module which maybe I could port back to this library, but it is not a short term thing...

[evilbuck]

It would be desirable for my use case fwiw. Though, I'll solve it outside the queue for time being. thanks for the answers.