Skip to content
Easily limit batch http(s) requests to a specified number of asynchronously managed, serially executed chains
JavaScript
Find file
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
tests
LICENSE
README.md
index.js
package.json

README.md

Limireq

Limireq is a Node.js module that throttles the number of concurrent active requests at a given time. This is useful when batch processing API data without overloading smaller servers and/or the client itself.

Installation

npm install limireq

Usage Example

// Require the module
var Limireq = require('limireq')

// Initialize a new instance
var lr = new Limireq(25) // max of 25 concurrent connections at any time

var usernames = [] // some array containing 50 usernames

// Push a URL or Request.js options object
for (var i = 0; i < 50; i += 2) {
  // Push a URL
  lr.push('http://api.hostname.com/users/' + usernames[i])

  // Push a request-module-compatible object
  lr.push({
    url: 'http://api.hostname.com/users/' + usernames[i],
    oauth: {
      consumer_key: 'YOUR_KEY_HERE',
      consumer_secret: 'YOUR_SECRET_HERE'
    }
  })
}

// Begin the processing
lr.start()

Event Handling

The responses and their body text will be emitted via the 'data' event:

lr.on('data', function(err, res, body) {
  try {
    var json = JSON.parse(console.log(body))
    doStuffToJsonInputFunction(json)
  } catch (e) {
    console.log(e)
  }
})

When all responses have been emitted, the throttle will emit an 'end' event and enable reuse of the limiter:

lr.on('end', function() {
  console.log('We finished')
})

Callbacks

If a specific request's response needs to be handled differently from the others, you can pass a callback function when you push the request to the pool. This means you will NOT receive a 'data' event for this request:

function callback(err, res, body) {
  // Do stuff
}

lr.push('http://api.hostname.com/users/' + usernames[i], callback)
lr.push({
  url: 'http://api.hostname.com/users/' + usernames[i],
  oauth: {
    consumer_key: 'YOUR_KEY_HERE',
    consumer_secret: 'YOUR_SECRET_HERE'
  }}, callback)

Reuse

Once the limiter has emitted it's 'end' event, it can reused immediately:

function startSecondJob() {
  lr.push()
  /// push more requests...
  lr.start()
  lr.on('data', function(err, res, body) {
    // do stuff with response
  }).on('end', function() {
    console.log('Completed two batch jobs')
  })
}

lr.on('end', startSecondJob)

Or it can be reinitialized with a new value for the maximum allowed simultaneous connections:

function startSecondJob() {
  lr.init(100) // raise parallel connection limit to 100
  lr.push()
  /// push more requests...
  lr.start()
  lr.on('data', function(err, res, body) {
    // do stuff with response
  }).on('end', function() {
    console.log('Completed two batch jobs')
  })
}

lr.on('end', startSecondJob)
Something went wrong with that request. Please try again.