Page-exctractor is function to retrieve data from page-like API, page by page until all data is retrieved.
😇 Zero depends
✨ Simple
🔥 Fast
Install Page-exctractor using npm
npm install page-exctractor
Import module:
const pageExctractor = require('page-exctractor');
or
import pageExctractor from 'page-exctractor';
Let`s getting started retrieve data from api (an example Steam API):
// totalData will contain data from all pages
const totalData = await pageExctractor(
// fetchFn will request data page by page until it reaches total
// For example, if the page size is 100 and the data is 1000, then 10 queries will be executed.
async (offset) => {
const response = await fetch(
`https://steamcommunity.com/market/search/render/?query=&start=${offset}count=100&appid=753&norender=1`,
);
const { results, total_count } = await response.json();
return {
data: results,
total: total_count,
};
},
);
pageExctractor(fetchFn, [, options]);
Where
- fetchFn is callback function to retrieve data. It must return an object with the number of records (total) and data.
- options is an optional options object
Possible option values
- pageSize single page size, defaults 100
- pageLimit limit of pages, defaults infinite
- totalLimit limit on the total amount of data for all pages, defaults infinite
- orderRequired whether the data order is mandatory, defaults false