Rate limit w/ groupKey is not respecting groupKey limit

[kylecannon]

I have a queue that has a concurrency of 10 on the worker but when i have a rate limit with a groupKey of 1, I am only getting a concurrency of 1 and every single job is delayed by 10 seconds. I have verified that the rateLimitKey is unique as well.

Essentially doing the following config below limits concurrency to 1 for a worker with 10 concurrency. Not sure why this is happening.

EX:
queue options:

    defaultJobOptions: {
        backoff: {
            type: 'exponential',
            delay: 2000,
        },
        attempts: 10,
        removeOnComplete: true,
        removeOnFail: true,
    },
    limiter: {
        groupKey: 'rateLimitKey',
    }

worker options:

{
        concurrency: 10,
        autorun: false,
        metrics: {
            maxDataPoints: MetricsTime.ONE_WEEK * 2,
        },
        limiter: {
            groupKey: 'rateLimitKey',
            max: 1,
            duration: 10000,
        },
    }

[manast]

@kylecannon do you have a complete test case that we can run?

[kylecannon]

I don't unfortunately at the moment.

[jamesjjk]

Any update on the above?

[manast]

The limiter based on the group key has its days counted. The implementation is too complex and makes it impossible to implement other optimizations that we want to do on the global rate limiter. On the other hand, we have complete group support in the Pro version, where the rate limit and other features work really well: https://docs.bullmq.io/bullmq-pro/groups

[manast]

Here some background information: https://blog.taskforce.sh/bullmq-3-0-release/

[Kylok]

I recently updated from BullMQ 1.54.1 to 2.4.0 (the latest 2.X) and the update introduced the issue described by @kylecannon above (large delays are being introduced that don't make sense given the limiting options). I see group-based limiting was explicitly removed in 3.0.0, but it appears at least somewhat broken in 2.X as well.

My queue and worker options are slightly more simple:

Queue options:

	defaultJobOptions: {
		removeOnComplete: true,
		removeOnFail: true
	},
	limiter: {
		groupKey: 'id'
	}

Worker options:

	concurrency: 20,
	limiter: {
		max: 1,
		duration: 250,
		groupKey: 'id'
	}

Reverting back to 1.91.1 (the latest 1.X) fixes the issue.

[manast]

@Kylok yes. The reason it was removed was precisely that the implementation was not solid enough. We are not going to spend time trying to fix the issues as there is no guarantee they can be fixed without a rewrite as we did on the Pro version.