Skip to content

Commit

Permalink
v4.0.0-alpha
Browse files Browse the repository at this point in the history
  • Loading branch information
stevenvachon committed May 1, 2017
1 parent 6ddd89e commit 67a8013
Show file tree
Hide file tree
Showing 23 changed files with 1,960 additions and 2,196 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
coverage/
node_modules/
.nyc_output/
6 changes: 3 additions & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
language: node_js
node_js:
- "4"
- "6"
script: npm test
- 6
- 7
script: npm run ci
93 changes: 43 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,23 @@
# limited-request-queue [![NPM Version][npm-image]][npm-url] [![Bower Version][bower-image]][bower-url] [![Build Status][travis-image]][travis-url] [![Dependency Status][david-image]][david-url]
# limited-request-queue [![NPM Version][npm-image]][npm-url] ![File Size][filesize-image] [![Build Status][travis-image]][travis-url] [![Coverage Status][coveralls-image]][coveralls-url] [![Dependency Status][david-image]][david-url]

> Interactively manage concurrency for outbound requests.
Features:
* Concurrency & rate limiting prevents overload on your server
* Per-Host concurrency limiting prevents overload on everyone else's servers
* Pause/Resume at any time
* Works in the browser (~4.4KB not gzipped)

* Concurrency & rate limiting prevents overload on your network.
* Per-Host concurrency limiting prevents overload on everyone else's network.
* Pause/Resume at any time.

```js
// Will work with any similar module, not just "request"
var request = require("request");
var RequestQueue = require("limited-request-queue");

var queue = new RequestQueue(null, {
item: function(input, done) {
request(input.url, function(error, response) {
done();
});
},
end: function() {
console.log("Queue completed!");
}
});

var urls = ["http://website.com/dir1/", "http://website.com/dir2/"];
urls.forEach(queue.enqueue, queue);
const RequestQueue = require('limited-request-queue');

const queue = new RequestQueue()
.on('item', function(url, data, done) {
yourRequestLib(url, () => done());
})
.on('end', () => console.log('Queue completed!'));

const urls = ['http://domain.com/dir1/', 'http://domain.com/dir2/'];
urls.forEach(url => queue.enqueue(new URL(url)));

setTimeout(queue.pause, 500);
setTimeout(queue.resume, 5000);
Expand All @@ -34,41 +26,41 @@ setTimeout(queue.resume, 5000);

## Installation

[Node.js](http://nodejs.org/) `>= 4` is required. To install, type this at the command line:
[Node.js](http://nodejs.org/) `>= 6` is required. To install, type this at the command line:
```shell
npm install limited-request-queue
```

Note: for use in a web browser, you will likely need [`Object.assign`](https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Object/assign) and [`URL`](https://developer.mozilla.org/en/docs/Web/API/URL/URL) polyfills for maximum coverage.
Note: for use in a web browser, you will likely need an ES2015/ES6 transpiler for maximum coverage.


## Constructor
```js
new RequestQueue(options, handlers);
new RequestQueue(options);
```


## Methods
## Methods & Properties

### `.dequeue(id)`
Removes a queue item from the queue. Use of this function is likely not needed as items are auto-dequeued when their turn is reached. Returns `true` on success or an `Error` on failure.
All methods from [`EventEmitter`](https://nodejs.org/api/events.html#events_class_eventemitter) are available.

### `.enqueue(input)`
Adds a URL to the queue. `input` can either be a URL `String` or a configuration `Object`. Returns a queue ID on success or an `Error` on failure.
### `.dequeue(id)`
Removes a queue item from the queue. Returns `true` if a queue item was removed and `false` if not. Use of this function is likely not needed as items are auto-dequeued when their turn is reached.

If `input` is an `Object`, it will accept the following keys:
### `.enqueue(url[, data, options])`
Adds a URL to the queue. Returns a queue item ID on success.

* `url`: a URL `String`, [`URL`](https://developer.mozilla.org/en/docs/Web/API/URL/) or [Node URL](https://nodejs.org/api/url.html#url_url_strings_and_url_objects)-compatible `Object`.
* `data`: additional data to be stored in the queue item.
* `id`: a unique ID (`String` or `Number`). If not defined, one will be generated.
* `url` *must* a [`URL`](https://developer.mozilla.org/en/docs/Web/API/URL/) instance.
* `data` is optional and can be of any type.
* `options` is an optional `Object` that overrides any defined options in the constructor (except for `maxSockets`).

### `.length()`
### `.length`
Returns the total number of items in the queue, active and inactive.

### `.numActive()`
### `.numActive`
Returns the number of items whose requests are currently in progress.

### `.numQueued()`
### `.numQueued`
Returns the number of items that have not yet made requests.

### `.pause()`
Expand All @@ -83,17 +75,17 @@ Resumes the queue.
### `options.ignorePorts`
Type: `Boolean`
Default value: `true`
Whether or not to treat identical hosts of different ports as a single concurrent group. **Example:** when `true`, http://mywebsite.com:80 and http://mywebsite.com:8080 may not have outgoing connections at the same time, but http://mywebsite.com:80 and http://yourwebsite.com:8080 will.
Whether or not to treat identical hosts of different ports as a single concurrent group. **Example:** when `true`, http://mydomain.com:80 and http://mydomain.com:8080 may not have outgoing connections at the same time, but http://mydomain.com:80 and http://yourdomain.com:8080 will.

### `options.ignoreSchemes`
### `options.ignoreProtocols`
Type: `Boolean`
Default value: `true`
Whether or not to treat identical hosts of different schemes/protocols as a single concurrent group. **Example:** when `true`, http://mywebsite.com and https://mywebsite.com may not have outgoing connections at the same time, but http://mywebsite.com and https://yourwebsite.com will.
Whether or not to treat identical hosts of different protocols as a single concurrent group. **Example:** when `true`, http://mydomain.com and https://mydomain.com may not have outgoing connections at the same time, but http://mydomain.com and https://yourdomain.com will.

### `options.ignoreSubdomains`
Type: `Boolean`
Default value: `true`
Whether or not to treat identical hosts of different subdomains as a single concurrent group. **Example:** when `true`, http://mywebsite.com and http://www.mywebsite.com may not have outgoing connections at the same time, but http://mywebsite.com and http://www.yourwebsite.com will.
Whether or not to treat identical domains of different subdomains as a single concurrent group. **Example:** when `true`, http://mydomain.com and http://www.mydomain.com may not have outgoing connections at the same time, but http://mydomain.com and http://www.yourdomain.com will.

This option is not available in the browser version (due to extreme file size).

Expand All @@ -104,7 +96,7 @@ The maximum number of connections allowed at any given time. A value of `0` will

### `options.maxSocketsPerHost`
Type: `Number`
Default value: `1`
Default value: `2`
The maximum number of connections per host allowed at any given time. A value of `0` will prevent anything from going out. A value of `Infinity` will provide no per-host concurrency limiting.

### `options.rateLimit`
Expand All @@ -113,20 +105,21 @@ Default value: `0`
The number of milliseconds to wait before each request. For a typical rate limiter, also set `maxSockets` to `1`.


## Handlers
## Events

### `handlers.end`
Called when the last item in the queue has been completed.
### `end`
Called when the last item in the queue has been completed/dequeued.

### `handlers.item`
Called when a queue item's turn has been reached. Arguments are: `input`, `done`.
### `item`
Called when a queue item's turn has been reached. Arguments are: `url`, `data`, `done`. Call the `done` function when your item's operations are complete.


[npm-image]: https://img.shields.io/npm/v/limited-request-queue.svg
[npm-url]: https://npmjs.org/package/limited-request-queue
[bower-image]: https://img.shields.io/bower/v/limited-request-queue.svg
[bower-url]: https://github.com/stevenvachon/limited-request-queue
[filesize-image]: https://img.shields.io/badge/size-3kB%20gzipped-blue.svg
[travis-image]: https://img.shields.io/travis/stevenvachon/limited-request-queue.svg
[travis-url]: https://travis-ci.org/stevenvachon/limited-request-queue
[coveralls-image]: https://img.shields.io/coveralls/stevenvachon/limited-request-queue.svg
[coveralls-url]: https://coveralls.io/github/stevenvachon/limited-request-queue
[david-image]: https://img.shields.io/david/stevenvachon/limited-request-queue.svg
[david-url]: https://david-dm.org/stevenvachon/limited-request-queue
28 changes: 0 additions & 28 deletions bower.json

This file was deleted.

1 change: 0 additions & 1 deletion browser/requestqueue.js

This file was deleted.

15 changes: 0 additions & 15 deletions lib/defaultOptions.js

This file was deleted.

90 changes: 17 additions & 73 deletions lib/getHostKey.js
Original file line number Diff line number Diff line change
@@ -1,105 +1,49 @@
"use strict";
var broquire = require("broquire")(require);
var isString = require("is-string");
var parseDomain = broquire("parse-domain", function(){ return null });
var URL = broquire("whatwg-url", "window").URL;
const defined = require("defined");
const parseDomain = require("parse-domain");



function getHostKey(url, options)
function getHostKey(url, options, optionOverrides={})
{
var key,port,protocol,urlDomain;

if (url == null)
{
return false;
}
else if (isString(url) === true)
{
try
{
url = new URL(url);
}
catch (error)
{
return false;
}
}

protocol = url.protocol;

// TODO :: remove support for node-js url.parse objects
if (isEmptyString(protocol)===true || isEmptyString(url.hostname)===true)
{
return false;
}

// Remove ":" suffix
if (protocol.indexOf(":") === protocol.length-1)
{
protocol = protocol.substr(0, protocol.length-1);
}

port = url.port;

// Get default port
// TODO :: remove support for node-js url.parse objects
if (isEmptyStringOrNumber(port)===true && options.defaultPorts[protocol]!==undefined)
{
port = options.defaultPorts[protocol];
}

key = "";
const ignorePorts = defined(optionOverrides.ignorePorts, options.ignorePorts);
const ignoreProtocols = defined(optionOverrides.ignoreProtocols, options.ignoreProtocols);
const ignoreSubdomains = defined(optionOverrides.ignoreSubdomains, options.ignoreSubdomains);

let key = "";

if (options.ignoreSchemes === false)
if (ignoreProtocols === false)
{
key += protocol + "://";
key += `${url.protocol}//`;
}

if (options.ignoreSubdomains === false)
if (ignoreSubdomains === false)
{
key += url.hostname;
}
else
{
urlDomain = parseDomain(url.hostname);
const hostname = parseDomain(url.hostname);

// If unknown top-level-domain (.com, etc)
// Or, if running in a browser
if (urlDomain === null)
// If unknown top-level domain or running in a browser
if (hostname === null)
{
key += url.hostname;
}
else
{
key += urlDomain.domain + "." + urlDomain.tld;
key += `${hostname.domain}.${hostname.tld}`;
}
}

if (options.ignorePorts===false && port!=null)
if (ignorePorts===false && url.port!=="")
{
key += ":" + port;
key += `:${url.port}`;
}

key += "/";

return key;
}



function isEmptyString(value)
{
return value==="" || value==null || isString(value)===false;
}



function isEmptyStringOrNumber(value)
{
return value==="" || value==null || isNaN(value)===true;
}



module.exports = getHostKey;

0 comments on commit 67a8013

Please sign in to comment.