Queue Throttling

[coreybutler]

Is your feature request related to a problem? Please describe.
Sometimes it is important to cap the number of operations executed within a period of time.

One use case is API operations. APIs often have rate limiting, which restricts how many API requests are allowed in a specific period of time. For example, an API may only allow 100 requests per minute. If 1000 requests need to be made, logic indicates it should take 10 minutes to run all the requests (100 per minute).

Describe the solution you'd like
I would like to configure throttling. The queue automatically batch tasks to fit within their time-restricted window or maximum
concurrent operations window. For example:

Rate Limiting

const tasks = new Queue()

tasks.ratelimit(100, 60 * 1000) // 100 requests, 60 seconds/60K milliseconds

for (i=0; i < 1000; i++) {
  tasks.add(() => ... do something...)
}

let plan = tasks.plan()
console.log(`Expected to take ${plan.executionDuration} milliseconds`)

tasks.on('complete', () => console.log('Done'))
tasks.run()

Sometimes limits aren't time sensitive, but only X number of operations can be performed at the same time (i.e. maximum simultaneous operations). This may look like:

Maximum Concurrency

const tasks = new Queue()

tasks.limit(10) // 10 operations at a time
...

The code above would only allow 10 simultaneous operations to work at the same time.

Describe alternatives you've considered
There are no alternatives in NGN. It must all be done by hand or with another library.

[gbdrummer]

This is a great idea! I have a couple of suggestions:

1. tasks.plan() would make more sense to me as a getter
2. tasks.limit() might be better named something like task.limitConcurrent() or something that indicates the concurrency aspect
3. I'd also love to see a way of batching tasks- if I could specify a number of tasks to run before checking if the the thread is idle, then run the next batch, etc... That would be awesome.

[gbdrummer]

After looking at this again, I actually think the limit function might work better as a getter/setter combo too, something like
maxConcurrent = 10. That makes it clearer to me what it is.

[coreybutler]

I suppose these could be getters/setters, with the exception of the rate limiter. These could also be configuration options for the Queue constructor, i.e.

const tasks = new Queue({
  ratelimit: [100, 60*1000],
  maxConcurrent: 10
})

// Update/reset rate limitations:
tasks.ratelimit(50, 60*1000)

// Update max concurrency
tasks.maxConcurrent = 30

// Remove ratelimit
tasks.ratelimit(null)

// Remove max concurrency limits
tasks.maxConcurrent = null

[gbdrummer]

What about adding a method like removeRateLimit()?

[coreybutler]

Both could work, i.e. removeRateLimit() and ratelimit(null). The first would just alias the second, i.e.

removeRateLimit () {
  this.ratelimit(null)
}

[coreybutler]

I added this to 1.0.0-alpha.11. The codepen has been updated.

The following now work:

const tasks = new Queue({
  rateLimit: [100, 60*1000],
  maxConcurrent: 10
})

• tasks.rate = [100, 60000]
• tasks.rate = null
• tasks.rateLimit(100, 60000)
• tasks.rateLimit(null)
• tasks.removeRateLimit()
• tasks.maxConcurrent = 10
• tasks.maxConcurrent = null

tasks.plan contains updated data (realtime):

{
  tasksCompleted: 100,
  tasksRemaining: 0,
  minimumDuration: 3000 // minimum amount of time (milliseconds) required to process the queue
}

The queue will also fire a new events when rate limiting is used:

1. batch.start represents the execution start of a new batch of tasks. The payload looks like

   {
       id: int, // A unique ID, within the queue, for the batch
       start: int, // (milliseconds)
       tasks: int // # of tasks in the batch
   }

2. batch.end represents the completion of the batch. The payload looks like

   {
       id: int, // A unique ID, within the queue, for the batch
       start: int, // (milliseconds)
       end: int, // (milliseconds)
       processingTime: int, // (milliseconds) time consumed by the collective tasks within the batch
       duration: int, // (milliseconds) time consumed since the queue execution began (i.e. total duration)
       tasks: int // # of tasks in the batch
   }

There are 39 passing unit tests and all syntax tests pass. The size of the library increased slightly, since there are two new classes (throttle.js and ratelimiter.js).

Rate limiting and throttling can be used simultaneously, but it may not have an effect in all circumstances. For example, a sequentially processed queue (i.e. non-parallel) may take longer than the minimum rate limit, so no limiting is necessary. Using both will not slow the process down though.

Overall, this addition should allow developers to run queues as fast as a service will allow, without exceeding caps. Be mindful that the timing in this library is very precise (nanoseconds). If a third party is less precise, developers should be careful to respect limits. For example, if a remote service says no more than 100 requests every minute, it may be worth using 61 seconds as the threshold instead of 60 seconds, to account for any potential clock issues.