hardened-fetch

Hardened Fetch is a tiny wrapper for global.fetch adding request timeouts, request throttling, retries with backoff and rate limit support, pagination, and descriptive errors. It makes working with APIs without SDKs and web scraping easier.

import { HardenedFetch } from 'hardened-fetch'

const client = new HardenedFetch({
  baseUrl: 'https://swapi.dev/api/',
})

const response = await client.fetch('/species/1/')

Installation

This is a Node.js module available through the npm registry. Node.js 18 or higher is required.

$ npm install hardened-fetch

Features

• Request timeouts
• Throttle concurrent requests and request rate
• Retries failed requests with exponential back off
• Retries rate-limited requests according to reset time
• Easily step-through paginated API responses

API

new HardenedFetch([options] = {})

Creates a new Hardened Fetch client.

Constructor Options:

Name	Type	Description
baseUrl	string	A base URL to prepend to each request, optional.
defaultHeaders	HeadersInit	Default headers to add to each request, optional.
maxConcurrency	number	How many requests can be running at the same time.
minRequestTime	number	How long to wait after launching a request before launching another one.
maxRetries	number	Number of retry attempts for failed requests.
doNotRetry	number[]	List of HTTP status codes that will not trigger a retry attempt.
rateLimitHeader	string	The name of the rate limit reset header, usually one of "Retry-After", "X-RateLimit-Reset", or"X-Rate-Limit-Reset"
resetFormat	string	The format of the rate limit reset header, must be one of "datetime", "seconds" or "milliseconds".

All of the options and their defaults are shown below:

const client = new HardenedFetch({
  // Request options
  baseUrl: undefined,
  defaultHeaders: undefined,
  // Throttle options
  maxConcurrency: 10,
  minRequestTime: 0,
  // Retry options
  maxRetries: 3,
  doNotRetry: [400, 401, 403, 404, 422, 451],
  // Rate limit options
  rateLimitHeader: 'Retry-After',
  resetFormat: 'seconds',
})

client.fetch(url, [init] = {}, [timeout] = 30000)

Expects a url to the resource that you wish to fetch and optionally custom settings to apply to the request, and a timeout in milliseconds. Returns a promise which will resolve with the Response object when successful. Rejects with a relevant HTTP error on failure.

const response = await client.fetch('https://swapi.dev/api/species/1/')
const json = await response.json()

client.paginatedFetch(url, [options] = {}, [timeout] = 30000)

Expects a url to the resource that you wish to fetch and optionally custom settings to apply to the request, and a timeout in milliseconds. Returns an AsyncIterator which will resolve with a Response object and done property on each successful iteration. Rejects with a relevant HTTP error on failure.

const pages = client.paginatedFetch('https://swapi.dev/api/species')

for await (const { response, done } of pages) {
  const json = await response.json()

  if (done) {
    console.log('Done!')
  }
}

License

This package is MIT licensed.