Skip to content

A null line ends the stream and silently drops every record after it #6

Description

@MildlyMeticulous

Summary

A line containing null ends the stream. Every record after it is silently dropped, and the stream emits stream.push() after EOF instead of anything that points at the cause.

null is valid JSON and valid NDJSON, so this is data loss on well-formed input.

Reproduction

import { Readable } from "node:stream";
import ndjson from "ndjson";

const input = '{"id":1}\n{"id":2}\nnull\n{"id":3}\n{"id":4}\n{"id":5}\n';
const rows = [];

Readable.from([input])
  .pipe(ndjson.parse())
  .on("data", (d) => rows.push(d))
  .on("error", (e) => console.log("error :", e.message))
  .on("end", () => console.log("rows  :", rows.length));
error : stream.push() after EOF
rows  : 2

Five valid records in, two out. {"id":3}, {"id":4} and {"id":5} are gone.

Cause

parseRow returns whatever JSON.parse produced, and split2 pushes that value downstream:

function parseRow (row) {
  try {
    if (row) return JSON.parse(row)
  } catch (e) {
    ...
  }
}

return split(parseRow, opts)

For the row null, JSON.parse returns null — and in Node object-mode streams push(null) is the end-of-stream signal. So the stream terminates early, and the next row that arrives triggers stream.push() after EOF.

The empty-string case is already guarded by if (row); null just isn't covered by it.

Suggested fix

null carries no record, so skipping it matches how blank lines are already handled. Returning undefined makes split2 skip the row:

function parseRow (row) {
  try {
    if (row) {
      const parsed = JSON.parse(row)
      return parsed === null ? undefined : parsed
    }
  } catch (e) {
    if (opts.strict) {
      this.emit('error', new Error('Could not parse row ' + row.slice(0, 50) + '...'))
    }
  }
}

With that, the repro above emits all five records.

If you'd rather surface it than skip it, emitting a parse-style error under opts.strict would work too — either is better than truncating the stream. Happy to send a PR for whichever you prefer.

Two smaller things found alongside

Not data loss, but same area, in case they're useful:

  • A whitespace-only line (" ") throws Could not parse row, while a truly empty line is skipped. Files written with trailing spaces hit this.
  • A leading UTF-8 BOM makes the first row fail to parse. Common with files produced by Windows tooling.

Tested against ndjson@2.0.0 on Node 24.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions