Skip to content

Home

Jump to bottom
Eugene Lazutkin edited this page Jul 28, 2026 · 17 revisions

stream-csv-as-json

Node.js CI NPM version

Search

🔍 Search this wiki — ranked, deep-linked search via wiki-search; install the bookmarklet to search in place. Fallback: GitHub wiki search.

Documentation

Start here

  • Main module — the Node entry point: a parser stream with emit() event sugar. Read it when you want events rather than a token pipeline.

Components

  • parser — the streaming CSV parser and its packing / streaming / separator options. Read it first: everything else consumes its tokens.
  • as-objects — turns the header row into field names and the rows that follow into object token streams.
  • stringer — the inverse: a token stream back to CSV text, with quoting and row-terminator control.
  • utils/with-parser — the CSV-specific withParser helper for combined pipelines.
  • file components — Node-only file-edge stages: parseFile() reads a path into tokens, stringerToFile() writes tokens to a path.

Tuning

  • Performance — where the throughput goes and which options move it. Read it when the pipeline is slower than expected.

Reference

V1 documentation (legacy)

  • V1-Home — the 1.x landing page; the other V1- pages hang off it.

About

stream-csv-as-json is a micro-library of stream components for building custom CSV processing pipelines with a minimal memory footprint, on Node.js or Web Streams. It can parse CSV files far exceeding available memory by streaming individual primitives using a SAX-inspired token API.

It is a companion library for stream-json and stream-chain, fully compatible with them. All components are substrate-free factory functions returning flushable closures, composed via stream-chain; each carries .asStream() (Node.js Duplex) and .asWebStream() (Web TransformStream-shaped pair). ESM-only, Node.js 22+.

The parser produces {name, value} tokens that are fully compatible with the stream-json token protocol. This means you can use stream-json utilities downstream — filters, streamers (like streamValues), and other components work seamlessly with CSV token streams.

Features

  • Parse CSV files compliant with RFC 4180 — quoted fields, embedded newlines. Reads CRLF, LF, and bare CR row terminators; strips a leading UTF-8 BOM.
  • Customizable field separator (comma, tab, pipe, etc.).
  • Stream individual field values piece-wise for minimal memory use.
  • Convert rows to object token streams using the first row as field names.
  • Convert token streams back to CSV text.
  • Token protocol compatible with stream-json — use its filters, streamers, and utilities downstream.
  • TypeScript declarations included.

Install

npm i stream-csv-as-json

Quick start

The package is ESM-only and requires Node.js 22+. The example below uses the Node.js entry; for browsers and Web Streams runtimes see Modules below.

import fs from 'node:fs';
import {parser} from 'stream-csv-as-json';
import asObjects from 'stream-csv-as-json/as-objects.js';
import chain from 'stream-chain';

// a,b,c
// 1,2,3

const pipeline = chain([fs.createReadStream('data.csv'), parser(), asObjects()]);

pipeline.on('data', token => console.log(token));
// {name: 'startObject'}
// {name: 'keyValue', value: 'a'}
// {name: 'stringValue', value: '1'}
// ...
// {name: 'endObject'}

Modules

ESM-only (Node.js 22+). Three substrate flavors are exposed through the exports map:

Node.js (. and per-component subpaths) — factories carry both .asStream (Node Duplex) and .asWebStream (Web pair):

import {parser} from 'stream-csv-as-json';
import asObjects from 'stream-csv-as-json/as-objects.js';
import stringer from 'stream-csv-as-json/stringer.js';

Web Streams (/web and /web/*) — browser-safe, no node:*; factories carry only .asWebStream:

import {chain} from 'stream-chain/web';
import parser from 'stream-csv-as-json/web/parser.js';
import asObjects from 'stream-csv-as-json/web/as-objects.js';

const pipeline = chain([response.body.pipeThrough(new TextDecoderStream()), parser(), asObjects()]);
for await (const token of pipeline.readable) console.log(token);

Substrate-free (/core/*) — the bare factory, no stream adapters attached:

import parser from 'stream-csv-as-json/core/parser.js';

Components

Component Import path Description
parser stream-csv-as-json/parser.js Streaming CSV parser producing {name, value} tokens
as-objects stream-csv-as-json/as-objects.js Header row → object token stream
stringer stream-csv-as-json/stringer.js Token stream → CSV text
utils/with-parser stream-csv-as-json/utils/with-parser.js CSV-specific withParser utility
Main module stream-csv-as-json Parser + emit() for event-based processing
file components stream-csv-as-json/file/parser.js Node-only: file path → tokens → file

Each component is a factory function: call it to get a flushable function for use in chain(), use .asStream() for a Node.js Duplex, or .asWebStream() for a Web {readable, writable} pair.

Token protocol

The parser represents CSV as a stream of {name, value} tokens:

Token Value Meaning
startArray — Start of a CSV row
endArray — End of a CSV row
startString — Start of a field value
stringChunk string Piece of a field value
endString — End of a field value
stringValue string Complete field value (packed)

By default both streamed (startString/stringChunk/endString) and packed (stringValue) tokens are emitted. This is controlled by packing and streaming options — see parser for details.

After as-objects, additional tokens appear:

Token Value Meaning
startObject — Start of a data row
endObject — End of a data row
startKey — Start of field name
stringChunk string Piece of field name
endKey — End of field name
keyValue string Complete field name (packed)

Common patterns

CSV round-trip

import fs from 'node:fs';
import stringer from 'stream-csv-as-json/stringer.js';

chain([fs.createReadStream('input.csv'), parser(), stringer(), fs.createWriteStream('output.csv')]);

Compressed CSV

import fs from 'node:fs';
import zlib from 'node:zlib';

chain([fs.createReadStream('data.csv.gz'), zlib.createGunzip(), parser(), asObjects()]);

Custom separator (TSV)

import fs from 'node:fs';

chain([fs.createReadStream('data.tsv'), parser({separator: '\t'}), asObjects()]);

Event-based processing

import fs from 'node:fs';
import makeParser from 'stream-csv-as-json';

const stream = makeParser();
stream.on('startArray', () => console.log('row started'));
stream.on('stringValue', val => console.log('field:', val));
fs.createReadStream('data.csv').pipe(stream);

Ecosystem integration

Because the parser produces tokens compatible with stream-json, you can use stream-json utilities downstream:

import fs from 'node:fs';
import chain from 'stream-chain';
import {parser} from 'stream-csv-as-json';
import asObjects from 'stream-csv-as-json/as-objects.js';
import streamValues from 'stream-json/streamers/stream-values.js';

// Convert CSV rows to actual JavaScript objects
chain([
  fs.createReadStream('data.csv'),
  parser(),
  asObjects(),
  streamValues(),
  ({value}) => console.log(value) // {name: 'John', age: '30', ...}
]);

Available stream-json utilities that work with CSV token streams:

  • streamers: streamValues — assembles tokens into complete values
  • filters: Filter, Pick, Ignore — filter the token stream
  • utils: emit — emits named events for each token type (used by the main module)

Performance

See Performance for tips on optimizing pipelines.

Migrating

V1 documentation

Documentation for the 1.x API is preserved with a V1 prefix: V1-Home · V1-Parser · V1-AsObjects · V1-Stringer · V1-Main-module · V1-Performance

Credits

The test file tests/sample.csv.gz is Master.csv from Lahman's Baseball Database 2012. The file is copyrighted by Sean Lahman. It is used here under a Creative Commons Attribution-ShareAlike 3.0 Unported License. In order to test all features of the CSV parser, the file was minimally modified: row #1000 has a CRLF inserted in a value, row #1001 has a double quote inserted in a value, then the file was compressed by gzip.

🔍 Search the wiki

stream-csv-as-json

Start here

Components

Tuning

Reference

stream-csv-as-json 1.x (legacy)

Built on stream-chain and stream-json

Clone this wiki locally