Promise based JavaScript library that enables you to make numerous HTTP requests without bumping into the rate limits
- Make parallel/concurrent HTTP requests by splitting your requests into batches
- Make sequential HTTP requests
- Put some delay between requests
- Retry failed requests again until reaching the specified retry limits
Using npm:
$ npm install reqqueUsing yarn:
$ yarn add reqquereqque(requests, requestTemplate, config)
reqque has a very simple usage. You just need to pass 3 arguments to start the machine;
- requests (required): The list of request items that will be passed to requestTemplate
- requestTemplate (required): The template that takes request item as parameter and makes the HTTP call by using the passed request item
- config (optional): custom reqque config
There are two different examples that are prepared to guide you on how to use reqque; these are basic.js and advanced.js.
In both examples, Axios is used to make HTTP requests easily configurable. You can replace Axios with any HTTP client that you prefer.
PS: You can access the executable example files under the examples folder.
Basic usage of reqque
import axios from "axios";
import reqque from "reqque";
(async () => {
const requests = [
"https://httpstat.us/200?sleep=100",
/*
"...",
"...",
*/
];
const requestTemplate = async (url) => axios.get(url);
const config = {
maxRetries: 2,
batch: {
size: {
limit: 5,
},
},
delay: {
duration: {
limit: 1000,
},
},
};
const results = await reqque(requests, requestTemplate, config);
})();This example shows you to how you can disguise your requests while making HTTP calls with reqque.
import axios from "axios";
import UserAgent from "user-agents";
import HttpsProxyAgent from "https-proxy-agent";
import reqque from "reqque";
(async () => {
const requests = [
{
method: "GET",
code: 200,
sleep: 100,
},
/*
{...},
{...},
*/
];
const proxyList = [
"110.164.253.85:8080",
/*
"...",
"...",
*/
];
const requestTemplate = async ({ method, code, sleep }) => {
const randomProxyIndex = Math.floor(Math.random() * proxyList.length);
const randomProxy = proxyList[randomProxyIndex];
const httpsAgent = new HttpsProxyAgent(`http://${randomProxy}`);
const userAgent = new UserAgent();
return axios({
url: `http://httpstat.us/${code}`,
query: { sleep },
headers: {
"User-Agent": userAgent,
},
method,
httpsAgent,
});
};
const config = {
maxRetries: 5,
batch: {
size: {
limit: 5,
},
},
};
const results = await reqque(requests, requestTemplate, config);
})();These are the available options of reqque config. All of them are optional. It will use the default options unless you specify custom values
{
// `maxRetries` defines the maximum number of retries.
// If set to 0, the failed requests won't be retried.
maxRetries: 5, // default
// `batch` defines the options of batch request feature.
batch: {
// `active` indicates whether or not the batch request feature is active
// If set to true, the requests will be splitted into batches and the requests in those batches will be made concurrently
// If set to false, the requests will be made sequentially one by one
active: true, // default
// `size` defines the size options of batch request feature.
size: {
// `limit` defines the number of requests that are placed in each batch
limit: 20, // default
// `random` indicates whether or not the batch size is random
// If set to true, a random batch size that is generated between 0 and limit value will be used in each iteration
// If set to false, a fixed batch size will be used in each iteration
random: false // default
}
},
// `delay` defines the options of delay feature.
delay: {
// `active` indicates whether or not the delay feature is active
// If set to true, it will be put some delay between each iterations
// If set to false, it won't be put any delay between iterations
active: true, // default
// `duration` defines the duration options of delay feature.
duration: {
// `limit` defines the amount of delays that are put between each iterations
limit: 20, // default
// `random` indicates whether or not the delay duration is random
// If set to true, a random delay duration that is generated between 0 and limit value will be used in each iteration
// If set to false, a fixed delay duration will be used in each iteration
random: false // default
}
}
}The result for a reqque request contains the following information.
[
{
// request item that was passed to request template
request: {},
// response body that was provided by the server
response: {},
// status of the request
status: "successful",
// the total number of request attempts
tryCount: 1,
},
/*
{...},
{...},
{...}
*/
];