Skip to content

Commit

Permalink
v4.0.0
Browse files Browse the repository at this point in the history
  • Loading branch information
stevenvachon committed Jul 12, 2019
1 parent 6ddd89e commit 069afcc
Show file tree
Hide file tree
Showing 24 changed files with 2,069 additions and 2,284 deletions.
4 changes: 4 additions & 0 deletions .gitignore
@@ -1 +1,5 @@
coverage/
lib-es5/
node_modules/
.nyc_output/
package-lock.json
7 changes: 4 additions & 3 deletions .travis.yml
@@ -1,5 +1,6 @@
language: node_js
node_js:
- "4"
- "6"
script: npm test
- 10
- 12
- node
script: npm run ci
98 changes: 46 additions & 52 deletions README.md
@@ -1,31 +1,23 @@
# limited-request-queue [![NPM Version][npm-image]][npm-url] [![Bower Version][bower-image]][bower-url] [![Build Status][travis-image]][travis-url] [![Dependency Status][david-image]][david-url]
# limited-request-queue [![NPM Version][npm-image]][npm-url] ![File Size][filesize-image] [![Build Status][travis-image]][travis-url] [![Coverage Status][coveralls-image]][coveralls-url] [![Dependency Monitor][greenkeeper-image]][greenkeeper-url]

> Interactively manage concurrency for outbound requests.
Features:
* Concurrency & rate limiting prevents overload on your server
* Per-Host concurrency limiting prevents overload on everyone else's servers
* Pause/Resume at any time
* Works in the browser (~4.4KB not gzipped)

* Concurrency & rate limiting prevents overload on your network.
* Per-Host concurrency limiting prevents overload on your target network(s).
* Pause/Resume at any time.

```js
// Will work with any similar module, not just "request"
var request = require("request");
var RequestQueue = require("limited-request-queue");

var queue = new RequestQueue(null, {
item: function(input, done) {
request(input.url, function(error, response) {
done();
});
},
end: function() {
console.log("Queue completed!");
}
});

var urls = ["http://website.com/dir1/", "http://website.com/dir2/"];
urls.forEach(queue.enqueue, queue);
const RequestQueue = require('limited-request-queue');

const queue = new RequestQueue()
.on('item', (url, data, done) => {
yourRequestLib(url, () => done());
})
.on('end', () => console.log('Queue completed!'));

const urls = ['http://domain.com/dir1/', 'http://domain.com/dir2/'];
urls.forEach(url => queue.enqueue(new URL(url)));

setTimeout(queue.pause, 500);
setTimeout(queue.resume, 5000);
Expand All @@ -34,41 +26,42 @@ setTimeout(queue.resume, 5000);

## Installation

[Node.js](http://nodejs.org/) `>= 4` is required. To install, type this at the command line:
[Node.js](http://nodejs.org) `>= 10` is required. To install, type this at the command line:
```shell
npm install limited-request-queue
```

Note: for use in a web browser, you will likely need [`Object.assign`](https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Object/assign) and [`URL`](https://developer.mozilla.org/en/docs/Web/API/URL/URL) polyfills for maximum coverage.


## Constructor
```js
new RequestQueue(options, handlers);
new RequestQueue(options);
```


## Methods
## Methods & Properties

All methods from [`EventEmitter`](https://nodejs.org/api/events.html#events_class_eventemitter) are available.

### `.dequeue(id)`
Removes a queue item from the queue. Use of this function is likely not needed as items are auto-dequeued when their turn is reached. Returns `true` on success or an `Error` on failure.
Removes a queue item from the queue. Returns `true` if a queue item was removed and `false` if not. Use of this function is likely not needed as items are auto-dequeued when their turn is reached.

### `.enqueue(input)`
Adds a URL to the queue. `input` can either be a URL `String` or a configuration `Object`. Returns a queue ID on success or an `Error` on failure.
### `.enqueue(url[, data, options])`
Adds a URL to the queue. Returns a queue item ID on success.

If `input` is an `Object`, it will accept the following keys:
* `url` *must* a [`URL`](https://developer.mozilla.org/en/docs/Web/API/URL/) instance.
* `data` is optional and can be of any type.
* `options` is an optional `Object` that overrides any defined options in the constructor (except for `maxSockets`).

* `url`: a URL `String`, [`URL`](https://developer.mozilla.org/en/docs/Web/API/URL/) or [Node URL](https://nodejs.org/api/url.html#url_url_strings_and_url_objects)-compatible `Object`.
* `data`: additional data to be stored in the queue item.
* `id`: a unique ID (`String` or `Number`). If not defined, one will be generated.
### `.isPaused`
Returns `true` if the queue is currently paused and `false` if not.

### `.length()`
### `.length`
Returns the total number of items in the queue, active and inactive.

### `.numActive()`
### `.numActive`
Returns the number of items whose requests are currently in progress.

### `.numQueued()`
### `.numQueued`
Returns the number of items that have not yet made requests.

### `.pause()`
Expand All @@ -83,17 +76,17 @@ Resumes the queue.
### `options.ignorePorts`
Type: `Boolean`
Default value: `true`
Whether or not to treat identical hosts of different ports as a single concurrent group. **Example:** when `true`, http://mywebsite.com:80 and http://mywebsite.com:8080 may not have outgoing connections at the same time, but http://mywebsite.com:80 and http://yourwebsite.com:8080 will.
Whether or not to treat identical hosts of different ports as a single concurrent group. **Example:** when `true`, http://mydomain.com:80 and http://mydomain.com:8080 may not have outgoing connections at the same time, but http://mydomain.com:80 and http://yourdomain.com:8080 will.

### `options.ignoreSchemes`
### `options.ignoreProtocols`
Type: `Boolean`
Default value: `true`
Whether or not to treat identical hosts of different schemes/protocols as a single concurrent group. **Example:** when `true`, http://mywebsite.com and https://mywebsite.com may not have outgoing connections at the same time, but http://mywebsite.com and https://yourwebsite.com will.
Whether or not to treat identical hosts of different protocols as a single concurrent group. **Example:** when `true`, http://mydomain.com and https://mydomain.com may not have outgoing connections at the same time, but http://mydomain.com and https://yourdomain.com will.

### `options.ignoreSubdomains`
Type: `Boolean`
Default value: `true`
Whether or not to treat identical hosts of different subdomains as a single concurrent group. **Example:** when `true`, http://mywebsite.com and http://www.mywebsite.com may not have outgoing connections at the same time, but http://mywebsite.com and http://www.yourwebsite.com will.
Whether or not to treat identical domains of different subdomains as a single concurrent group. **Example:** when `true`, http://mydomain.com and http://www.mydomain.com may not have outgoing connections at the same time, but http://mydomain.com and http://www.yourdomain.com will.

This option is not available in the browser version (due to extreme file size).

Expand All @@ -104,7 +97,7 @@ The maximum number of connections allowed at any given time. A value of `0` will

### `options.maxSocketsPerHost`
Type: `Number`
Default value: `1`
Default value: `2`
The maximum number of connections per host allowed at any given time. A value of `0` will prevent anything from going out. A value of `Infinity` will provide no per-host concurrency limiting.

### `options.rateLimit`
Expand All @@ -113,20 +106,21 @@ Default value: `0`
The number of milliseconds to wait before each request. For a typical rate limiter, also set `maxSockets` to `1`.


## Handlers
## Events

### `handlers.end`
Called when the last item in the queue has been completed.
### `end`
Called when the last item in the queue has been completed/dequeued.

### `handlers.item`
Called when a queue item's turn has been reached. Arguments are: `input`, `done`.
### `item`
Called when a queue item's turn has been reached. Arguments are: `url`, `data`, `done`. Call the `done` function when your item's operations are complete.


[npm-image]: https://img.shields.io/npm/v/limited-request-queue.svg
[npm-url]: https://npmjs.org/package/limited-request-queue
[bower-image]: https://img.shields.io/bower/v/limited-request-queue.svg
[bower-url]: https://github.com/stevenvachon/limited-request-queue
[filesize-image]: https://img.shields.io/badge/size-4.5kB%20gzipped-blue.svg
[travis-image]: https://img.shields.io/travis/stevenvachon/limited-request-queue.svg
[travis-url]: https://travis-ci.org/stevenvachon/limited-request-queue
[david-image]: https://img.shields.io/david/stevenvachon/limited-request-queue.svg
[david-url]: https://david-dm.org/stevenvachon/limited-request-queue
[coveralls-image]: https://img.shields.io/coveralls/stevenvachon/limited-request-queue.svg
[coveralls-url]: https://coveralls.io/github/stevenvachon/limited-request-queue
[greenkeeper-image]: https://badges.greenkeeper.io/stevenvachon/limited-request-queue.svg
[greenkeeper-url]: https://greenkeeper.io/
28 changes: 0 additions & 28 deletions bower.json

This file was deleted.

1 change: 0 additions & 1 deletion browser/requestqueue.js

This file was deleted.

15 changes: 0 additions & 15 deletions lib/defaultOptions.js

This file was deleted.

106 changes: 22 additions & 84 deletions lib/getHostKey.js
@@ -1,105 +1,43 @@
"use strict";
var broquire = require("broquire")(require);
var isString = require("is-string");
var parseDomain = broquire("parse-domain", function(){ return null });
var URL = broquire("whatwg-url", "window").URL;
import parseDomain from "parse-domain";



function getHostKey(url, options)
export default (url, options, optionOverrides) =>
{
var key,port,protocol,urlDomain;

if (url == null)
{
return false;
}
else if (isString(url) === true)
{
try
{
url = new URL(url);
}
catch (error)
{
return false;
}
}

protocol = url.protocol;

// TODO :: remove support for node-js url.parse objects
if (isEmptyString(protocol)===true || isEmptyString(url.hostname)===true)
{
return false;
}

// Remove ":" suffix
if (protocol.indexOf(":") === protocol.length-1)
{
protocol = protocol.substr(0, protocol.length-1);
}

port = url.port;

// Get default port
// TODO :: remove support for node-js url.parse objects
if (isEmptyStringOrNumber(port)===true && options.defaultPorts[protocol]!==undefined)
{
port = options.defaultPorts[protocol];
}

key = "";

if (options.ignoreSchemes === false)
const ignorePorts = optionOverrides?.ignorePorts ?? options.ignorePorts;
const ignoreProtocols = optionOverrides?.ignoreProtocols ?? options.ignoreProtocols;
const ignoreSubdomains = optionOverrides?.ignoreSubdomains ?? options.ignoreSubdomains;

let key = "";

if (!ignoreProtocols)
{
key += protocol + "://";
key += `${url.protocol}//`;
}
if (options.ignoreSubdomains === false)

if (!ignoreSubdomains)
{
key += url.hostname;
}
else
{
urlDomain = parseDomain(url.hostname);

// If unknown top-level-domain (.com, etc)
// Or, if running in a browser
if (urlDomain === null)
const hostname = parseDomain(url.hostname);

// If unknown top-level domain or running in a browser
if (hostname === null)
{
key += url.hostname;
}
else
{
key += urlDomain.domain + "." + urlDomain.tld;
key += `${hostname.domain}.${hostname.tld}`;
}
}
if (options.ignorePorts===false && port!=null)

if (!ignorePorts && url.port!=="")
{
key += ":" + port;
key += `:${url.port}`;
}

key += "/";

return key;
}



function isEmptyString(value)
{
return value==="" || value==null || isString(value)===false;
}



function isEmptyStringOrNumber(value)
{
return value==="" || value==null || isNaN(value)===true;
}



module.exports = getHostKey;
return key;
};

0 comments on commit 069afcc

Please sign in to comment.