Process large CSV files in batches without backpressure
The CSV is streamed in batches and the stream is paused while you perform async operations on each batch to prevent backpressure building up
Uses csv-parser
under the hood
$ npm install batch-stream-csv
const batch = require('batch-stream-csv');
batch('./file.csv', batchHandler, options).then(() => console.log('All done!'))
async function batchHandler (batch, progress) => {
console.log(batch) // batch of rows parsed as JSON, e.g. [{ a: 1, b: 2 }, { a: 2, b: 3 }]
await db.batchInsert('table', batch) // the stream is paused while you do your inserts
console.log(`${progress * 100}% complete`) // progress value can be fed into a progress bar
}
batchSize
: number of rows to process per batch (default = 5000)- ...see https://github.com/mafintosh/ for other options