Order messed up?

[ghost]

When I am adding a limiter to the worker to process 4 jobs, it does that, and doesn't resume from the 5th job, it skips the 5th job and moves over to the sixth job, and this happens at everytime I attach a worker to the queue.
Here is the code

producer.ts
const sche = new QueueScheduler("contacts", {
		connection: redisConnection,
	});
	const contactsQueue = new Queue("contacts", {
		connection: redisConnection,
		defaultJobOptions: {
			removeOnComplete: false,
			removeOnFail: false,
		},
	});
	const sleep = promisify(setTimeout);
	let p = 0;
	for (let i = 0; i < 40; i++) {
		await contactsQueue.add("contact:add", { count: p++ });
		await sleep(200);
	}

consumer.ts
const worker = new Worker(
		"contacts",
		async (job: Job) => {},
		{
			connection: redisConnection,
			limiter: {
				max: 4,
				duration: 1000,
			},
		}
	);

	worker.on("completed", async (job: Job) => {
		console.log("[completed]", job.data, job.attemptsMade);
	});

	const failedJobs: Job[] = [];

	worker.on("failed", async (job: Job) => {
		console.log("[failed]", job.data, job.attemptsMade);
		failedJobs.push(job);
	});

	worker.on("drained", async () => {
		console.log("[queue drained]");
		await worker.close();
		failedJobs.forEach(async (job) => {
			await job.retry("failed");
		});
	});

And the output is,

completed] { count: 0 } 0
[completed] { count: 1 } 0
[completed] { count: 2 } 0
[completed] { count: 3 } 0
[queue drained]
[completed] { count: 5 } 0
[completed] { count: 6 } 0
[completed] { count: 7 } 0
[completed] { count: 8 } 0
[queue drained]
[completed] { count: 10 } 0
[completed] { count: 11 } 0
[completed] { count: 12 } 0
[completed] { count: 13 } 0
[queue drained]
[completed] { count: 15 } 0
[completed] { count: 16 } 0
[completed] { count: 17 } 0
[completed] { count: 18 } 0
[queue drained]
[completed] { count: 20 } 0
[completed] { count: 21 } 0
[completed] { count: 22 } 0
[completed] { count: 23 } 0
[queue drained]
[completed] { count: 25 } 0
[completed] { count: 26 } 0
[completed] { count: 27 } 0
[completed] { count: 28 } 0
[queue drained]
[completed] { count: 30 } 0
[completed] { count: 31 } 0
[completed] { count: 32 } 0
[completed] { count: 33 } 0
[queue drained]
[completed] { count: 35 } 0
[completed] { count: 36 } 0
[completed] { count: 37 } 0
[completed] { count: 38 } 0
[queue drained]
[completed] { count: 4 } 0
[completed] { count: 9 } 0
[completed] { count: 14 } 0
[completed] { count: 19 } 0
[queue drained]
[completed] { count: 29 } 0
[completed] { count: 34 } 0
[completed] { count: 39 } 0
[completed] { count: 24 } 0
[queue drained]

[manast]

When jobs are rate limited they are delayed, and if new jobs are arriving constantly it could happen that the order is not preserved. I have a plan to implement rate limiting in a different way that would avoid this behaviour, but currently it works as designed.

[ghost]

removeOnFail: false removes jobs when they fail, until explicitly it is specified job.retry(). Is this some sort of bug or I'm getting it wrong?

[manast]

failed jobs are not deleted by default.

[ghost]

#230 Please see this. No matter what I tried, the failed jobs are always removed. And there is some issues with async errors when using concurrency

[manast]

This passing test demonstrates that failed jobs are not always removed: https://github.com/taskforcesh/bullmq/blob/master/src/test/test_getters.ts#L111-L136

Please provide a code example that reproduces the issue and I will look into it.

[ghost]

According to the test, they are available in the Queue, should it manually be added to the queue later on?

[manast]

I do not understand what you mean.

[nguyenvudao]

Hey @manast, are there any possible ways to keep the order of the jobs?

[manast]

@nguyenvudao for rate limiter not guaranteed at the moment, best effort only.