Fetch highWaterMark

[ronag]

What should we use as highWaterMark for the fetch api? I think browsers usually have 1m+ while Node has ~16k. Should we try matching the browsers? I think this has some implications on functionality when using Response.clone.

[szmarczak]

Can it be updated dynamically? E.g when the response is not cloned, then it can be 64kb. If someone reads from the clone then increase it again by 64kb if it's been read.

[ronag]

I guess it could.

[ronag]

But if it's only 64k I think we could just have it as a default. It's more relevant if we have something like > 128k.

[ronag]

node-fetch/node-fetch#151
node-fetch/node-fetch#248

[szmarczak]

But if it's only 64k I think we could just have it as a default. It's more relevant if we have something like > 128k.

Performance-wise I don't think there's a difference.

because we are not reading from cloned stream at the time, its internal buffer will fill up, eventually, causing both stream to pause and hence what we called backpressure.

I mean if the response is cloned, then every time it's consumed bit by bit increase the highWaterMark so it can flow.

Example:

1. A new response is created.
2. It's being cloned.
3. 64k is read from the clone. In order not to pause the stream, increase the highWaterMark by 64k.
4. The highWaterMark is now 128k.
5. Another 64k is read from the clone. In order not to pause the stream, increase the highWaterMark by 64kb.
6. The highWaterMark is now 192k.
7. 192k is read from the original response. In order not to pause the stream, increase highWaterMark again by 64k.
8. The highWaterMark is now 256k.

[ronag]

We need a "max" highWaterMark then.

[szmarczak]

Ah. Then I think 1MB should be more than enough.

[ronag]

I wonder if it is possible to implement a custom stream controller? @jasnell

[mcollina]

Take a look at https://www.npmjs.com/package/cloneable-readable on how I clone Node.js streams. In our world I did that by setting the highwaterMark of the clone, and then the pipe mechanism did the rest.

[ronag]

This is unfortunately for web streams, not node streams.