highWaterMark

highWaterMark (hwm) is the queue-size threshold at which a stream signals backpressure to its producer. When the internal queue holds at least hwm items, the producer is told to slow down so the consumer can catch up. Properly-written stream code reacts to that signal; ignoring it lets the queue grow unboundedly, which costs memory and risks OOM on large streams.

This page covers what hwm is, how Node streams and Web Streams differ in their defaults and semantics, and how to set it through chain(), asStream(), and asWebStream().

Why hwm matters

Streams are about flow control between a producer and a consumer that may run at different speeds. The internal queue between them absorbs short-term mismatches; hwm decides "how short is short."

• Too small → frequent throttling round-trips between producer and consumer; more scheduling overhead per item.
• Too large → more buffered items; more memory; bursts may complete entirely before backpressure ever kicks in.
• Unbounded (the buggy case) → producer ignores backpressure, queue grows with input size, eventual OOM.

The right hwm depends on item size, consumer latency variance, and how much memory you can afford to buffer.

Node streams: Duplex / Transform / Writable

Defaults:

• Object-mode streams (objectMode: true): highWaterMark = 16 (items)
• Byte-mode streams: highWaterMark = 16384 (bytes)

Backpressure mechanism: stream.write(chunk) returns false when the writable buffer has at least hwm items queued. Writers should pause and resume on the 'drain' event. Similarly, stream.push(value) inside a Transform/Duplex returns false when the readable side is full; the implementation should stop pushing and resume from _read().

Configurable per-side: Node Duplex accepts separate readableHighWaterMark and writableHighWaterMark in addition to the unified highWaterMark. See Node's Duplex docs.

Web Streams: ReadableStream / WritableStream / {readable, writable} duplex pair

Defaults:

• Default queuing strategy for object streams: {highWaterMark: 1, size: () => 1} — yes, 1.
• For byte streams: defaults are based on byte length (ByteLengthQueuingStrategy).

Backpressure mechanism: controller.desiredSize reports (highWaterMark - queueSize). When <= 0, the readable's pull() callback stops firing; the writable's writer.write(chunk) returns a Promise that resolves only when downstream catches up. controller.enqueue() itself is non-throwing even when over capacity — it's the producer's job to check desiredSize and not over-enqueue.

Configurable via queuing strategy: the second argument to new ReadableStream(source, strategy) / new WritableStream(sink, strategy). Strategy shape: {highWaterMark: number, size?: (chunk) => number}. Built-ins: CountQueuingStrategy({highWaterMark}), ByteLengthQueuingStrategy({highWaterMark}).

Note: Web Streams' default hwm of 1 is much smaller than Node's 16. For many workloads this means more throttling round-trips on Web. See the perf notes below.

Setting options in this project

chain(fns, options) — Node side (stream-chain / stream-chain/node)

options is spread into the underlying Duplex constructor. Pass any Node DuplexOptions (including highWaterMark, readableHighWaterMark, writableHighWaterMark):

import chain from 'stream-chain';

const pipe = chain([source, fn1, fn2, sink], {
  highWaterMark: 64, // both sides
  // or:
  readableHighWaterMark: 100, // readable only
  writableHighWaterMark: 32 // writable only
});

If you pass functions wrapped with asStream() inside the chain, those wrappers were created with their OWN options. The chain doesn't reconfigure pre-built streams — only the Duplex it creates around them.

asStream(fn, options) — Node-side wrapper

Same shape as chain(). Spread into Duplex:

import asStream from 'stream-chain/asStream.js';

const s = asStream(myFn, {highWaterMark: 64});

chain(fns, options) — Web side (stream-chain/web)

options uses Web Streams' standard QueuingStrategy (the same {highWaterMark, size} shape accepted by new ReadableStream(..., strategy) / new WritableStream(..., strategy)). It's forwarded to every asWebStream stage the chain creates:

import chain from 'stream-chain/web';

const c = chain([source, fn1, fn2, sink], {
  strategy: {highWaterMark: 64}, // shorthand: both sides of each new stage

  // or per-side:
  readableStrategy: {highWaterMark: 100},
  writableStrategy: {highWaterMark: 32},

  // or full built-in strategy:
  readableStrategy: new CountQueuingStrategy({highWaterMark: 100})
});

Per-side strategies override the shorthand strategy. Pre-built {readable, writable} items in fns keep their own settings; only newly-wrapped stages get the chain's options.

asWebStream(fn, options) — Web-side wrapper

Same fields as the /web chain():

import asWebStream from 'stream-chain/asWebStream.js';

const ts = asWebStream(myFn, {strategy: {highWaterMark: 64}});

Perf implications

The bench file bench/raw-streams.js compares Node Duplex vs Web {readable, writable} at default hwm with proper per-item backpressure. With the project's asStream / asWebStream executors:

• Node-side chain() with default hwm=16 amortizes the per-item Promise overhead well — push is cheap, drain events are batched.
• Web-side chain() with default hwm=1 pays per-item Promise resolution cost on every push. Increasing hwm reduces the wait-for-drain frequency but increases steady-state buffer size.

Rough rules of thumb when tuning:

• For stream-of-many-small-items workloads (objects passing through a pipeline), Web users see meaningful wins from highWaterMark: 16 or 64 on the readable side — the chain spends less time round-tripping with the consumer.
• For stream-of-large-items workloads (big buffers, big objects), keep hwm modest — each buffered item is expensive in memory.
• For bursty-source workloads (one input that expands to many outputs via a generator), the per-item backpressure pattern keeps the queue at hwm regardless of burst size. Increasing hwm changes the steady-state queue size, not the maximum.

The asStream / asWebStream executors honor backpressure per push — no matter how large the burst, the queue never exceeds hwm. This was an explicit design fix in v4.

What hwm does NOT change

• It doesn't bound the total memory used by the stream over its lifetime — that's a function of how many in-flight chunks the consumer is also holding (event listeners that retain values, etc.).
• It doesn't change the semantic order of values flowing through the chain — all items are still processed exactly once, in order.
• It doesn't affect the chain's error-propagation behavior or the stop/finalValue/many/none semantics.

Related

• asStream — Node-side executor.
• asWebStream — Web-side executor.
• chain() — pipeline factory.
• Node Stream backpressure docs
• Web Streams queuing strategies