Skip to content

Home

Jump to bottom
Eugene Lazutkin edited this page Jun 26, 2026 · 8 revisions

Dashboard

Node.js CI NPM version

About

stream-sorting is a streaming sort plus a suite of operations on sorted object-mode Readable streams. The headline operation is a disk-backed external merge sort designed to sort a billion records on a laptop without OOMing. On top of the sort, the package builds key-based join, set operations (union / intersection / difference), and k-way sorted merge.

Scope of the package: anything that requires or preserves sortedness. Combinators that know nothing about sortedness live in stream-join; 1→N split / route / filter lives in stream-fork. Built on stream-chain and stream-join — its only runtime dependencies. Node-only (server / CLI); browsers are out of scope. ESM only. Distributed under the New BSD license.

Status

Early development. All components below are implemented — the object-stream wrapper protocol, both sort algorithms (sort, polyphaseSort), the join family, aggregate, the key-based filters, and the set operations. The package has not yet published its first npm release.

Documentation

Sorting

  • sort — disk-backed external (k-way) merge sort. The headline operation.
  • polyphaseSort — polyphase merge sort; fixed file budget, for bounded or heterogeneous storage.
  • sortJsonl — text-JSONL convenience: parse lines → sort → stringify.

Storage

  • ObjectStreamWrapper — the storage abstraction the algorithms read and write through (also documents ItemWriter and consume).
  • MemoryWrapper — in-memory wrapper for tests, small data, and benchmarks.
  • LocalFileWrapper — local-file wrapper (line-delimited; JSON by default).

Joins and aggregation

  • join — inner join of two or more sorted streams by key (Cartesian product on equal keys; per-input optional for outer shapes).
  • leftJoin — left outer join; the first input is required, the rest null-filled.
  • fullJoin — full outer join; a row emits for any key on any side.
  • aggregate — fold child streams under a master (or a key-derived group) by key; SQL-style group/aggregate.

Filters and set operations

  • matching / unmatched — key-based filters: rows of the primary stream whose key is present / absent in the other.
  • merge — k-way sorted merge, duplicates preserved; the suite's foundational op.
  • union — sorted merge with cross-stream deduplication.
  • intersection — values present in all input streams.
  • difference — values in the first stream not present in the rest.

Other

Installation

npm i stream-sorting

Clone this wiki locally