Why is Readable.read(size) always returning the whole readable data?

[ericmorand]

I am obviously missing something obvious, but I can't figure what.

Consider the following code:

import {Readable} from "node:stream";

const readable = Readable.from('1'.repeat(1000000));
const partialData = readable.read(16);

console.log(partialData.length);

The last command always output "1000000" - that is the size of the length of the string that the Readable was created from. I expect the output size to be much lower since I explicitely ask the Readable to read only 16 bytes.

What am I missing?

How can we read partial data from a Readable?

[Crackle2K]

Readable.from() when passed a plain string treats the whole thing as a single chunk, so by the time you call .read(16) the internal buffer already has one 1,000,000-byte chunk in it and Node returns that whole chunk rather than slicing it.

The simplest fix is to slice the result after reading:

const partialData = (readable.read(16) ?? '').slice(0, 16);
console.log(partialData.length); // 16

Or if you want a stream that genuinely yields data in controlled chunk sizes from the start, use a custom Readable that pushes in pieces:

import { Readable } from "node:stream";

function chunkedReadable(str, chunkSize) {
  let offset = 0;
  return new Readable({
    read() {
      if (offset >= str.length) return this.push(null);
      this.push(str.slice(offset, offset + chunkSize));
      offset += chunkSize;
    }
  });
}

const readable = chunkedReadable('1'.repeat(1000000), 16);
const partialData = readable.read(16);
console.log(partialData.length); // 16

[ericmorand]

the internal buffer already has one 1,000,000-byte chunk in it and Node returns that whole chunk rather than slicing it.

This is the point that I don't understand. The Node.js documentation states the following about the read function:

The optional size argument specifies a specific number of bytes to read. If size bytes are not available to be read, null will be returned unless the stream has ended, in which case all of the data remaining in the internal buffer will be returned.

If the size argument is not specified, all of the data contained in the internal buffer will be returned.

The second part removes every possible ambiguity: only if the size is not specified is the whole internal buffer data returned.

So, why is the read function returning the whole chunk when we pass a size argument?

EDIT: I get it now. This part matches the behaviour that you explained:

null will be returned unless the stream has ended, in which case all of the data remaining in the internal buffer will be returned.

Thank you, it is clear now.