Home

stream-csv-as-json

stream-csv-as-json is a micro-library of Node.js stream components for creating custom CSV processing pipelines with a minimal memory footprint. It can parse CSV files far exceeding available memory by streaming individual primitives using a SAX-inspired token API.

It is a companion library for stream-json and stream-chain, fully compatible with them. All components are factory functions returning flushable closures, composed via stream-chain.

Features

• Parse CSV files compliant with RFC 4180 — quoted fields, embedded newlines, \r\n and \n line endings.
• Customizable field separator (comma, tab, pipe, etc.).
• Stream individual field values piece-wise for minimal memory use.
• Convert rows to object token streams using the first row as field names.
• Convert token streams back to CSV text.
• Token protocol compatible with stream-json — use its filters, streamers, and utilities downstream.
• TypeScript declarations included.

Install

npm i stream-csv-as-json

Quick start

const chain = require('stream-chain');
const {parser} = require('stream-csv-as-json');
const asObjects = require('stream-csv-as-json/as-objects.js');
const fs = require('node:fs');

// a,b,c
// 1,2,3

const pipeline = chain([fs.createReadStream('data.csv'), parser(), asObjects()]);

pipeline.on('data', token => console.log(token));
// {name: 'startObject'}
// {name: 'keyValue', value: 'a'}
// {name: 'stringValue', value: '1'}
// ...
// {name: 'endObject'}

Components

Component	Import path	Description
parser	stream-csv-as-json/parser.js	Streaming CSV parser producing {name, value} tokens
as-objects	stream-csv-as-json/as-objects.js	Header row → object token stream
stringer	stream-csv-as-json/stringer.js	Token stream → CSV text
utils/with-parser	stream-csv-as-json/utils/with-parser.js	CSV-specific withParser utility
Main module	stream-csv-as-json	Parser + emit() for event-based processing

Each component is a factory function: call it to get a flushable function for use in chain(), or use .asStream() to get a Duplex stream.

Token protocol

The parser represents CSV as a stream of {name, value} tokens:

Token	Value	Meaning
startArray	—	Start of a CSV row
endArray	—	End of a CSV row
startString	—	Start of a field value
stringChunk	string	Piece of a field value
endString	—	End of a field value
stringValue	string	Complete field value (packed)

By default both streamed (startString/stringChunk/endString) and packed (stringValue) tokens are emitted. This is controlled by packing and streaming options — see parser for details.

After as-objects, additional tokens appear:

Token	Value	Meaning
startObject	—	Start of a data row
endObject	—	End of a data row
startKey	—	Start of field name
stringChunk	string	Piece of field name
endKey	—	End of field name
keyValue	string	Complete field name (packed)

Common patterns

CSV round-trip

const stringer = require('stream-csv-as-json/stringer.js');

chain([fs.createReadStream('input.csv'), parser(), stringer(), fs.createWriteStream('output.csv')]);

Compressed CSV

const zlib = require('node:zlib');

chain([fs.createReadStream('data.csv.gz'), zlib.createGunzip(), parser(), asObjects()]);

Custom separator (TSV)

chain([fs.createReadStream('data.tsv'), parser({separator: '\t'}), asObjects()]);

Event-based processing

const make = require('stream-csv-as-json');

const stream = make();
stream.on('startArray', () => console.log('row started'));
stream.on('stringValue', val => console.log('field:', val));
fs.createReadStream('data.csv').pipe(stream);

Performance

See Performance for tips on optimizing pipelines.

Migrating from 1.x

See Migration from 1.x to 2.x for a detailed guide on upgrading.

V1 documentation

Documentation for the 1.x API is preserved with a V1 prefix: V1-Home · V1-Parser · V1-AsObjects · V1-Stringer · V1-Main-module · V1-Performance

Credits

The test file tests/sample.csv.gz is Master.csv from Lahman's Baseball Database 2012. The file is copyrighted by Sean Lahman. It is used here under a Creative Commons Attribution-ShareAlike 3.0 Unported License. In order to test all features of the CSV parser, the file was minimally modified: row #1000 has a CRLF inserted in a value, row #1001 has a double quote inserted in a value, then the file was compressed by gzip.