ak-fetch

tldr;

a "batch and queue" HTTP client for making bulk POST requests to an API endpoint:

🌍 Overview

ak-fetch is a powerful and flexible tool designed for making bulk POST requests to an API endpoint. It provides a simple interface to batch and queue requests which send data over the network; it adds oft needed features like concurrency control, retries, fire-and-forget making it a robust solution for handling large data transfer operations with ease.

i built this module when creating hello-mixpanel as i needed a way to send large amounts of data to various SaaS analytics APIs in a reliable and efficient manner. it was useful enough for me, that i made it into a proper package.

🚀 Installation

To get started with ak-fetch, install the module using npm:

npm install ak-fetch

you may also use npx to run the CLI without installing the package:

npx ak-fetch --help

🖥️ Usage

Use ak-fetch as a module directly in your node script:

const akFetch = require('ak-fetch');
const config = {
        url: 'https://api.example.com',
        data: [...],
		batchSize: 10,
		concurrency: 5,
		delay: 1000,
        // ... other configurations
};

try {
	const responses = await akFetch(config);
	console.log('API Responses:', responses);
} catch (error) {
	console.error('Error:', error);
}

or via the command line:

npx ak-fetch --url https://api.example.com './payloads.json' --batchSize 10 --concurrency 5

Use --help to see all options

🛠️ Configuration

The ak-fetch module can be configured with a variety of options to suit your needs:

Option	Type	Description
url	string	The URL of the API endpoint.
data	Object[]	An array of data objects to be sent in the requests.
batchSize	number	# records per batch; Use 0 for no batching.
concurrency	number	The level of concurrency for the requests.
delay	number	The delay between requests in milliseconds.
searchParams	Object	The search parameters to be appended to the URL.
bodyParams	Object	The body parameters to be sent in the request.
headers	Object	The headers to be sent in the request.
verbose	boolean	Whether to log progress of the requests.
dryRun	boolean	If true, no actual requests are made.
logFile	string	File path where responses will be saved.
retries	number	Number of retries for failed requests; use null for fire-and-forget.
retryDelay	number	Delay between retries in milliseconds.
retryOn	number[]	HTTP status codes to retry on.

(note that with the CLI you may use camelCase or snake_case for options)

🧩 Contributing

Contributions to ak-fetch are welcome! Feel free to open issues or submit pull requests.

📝 License

ak-fetch is ISC licensed; use it however you wish.