stream: support readable/writableHighWaterMark for Duplex streams #14636

guymguym · 2017-08-04T23:34:05Z

Fixes: #14555

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

stream

gibfahn · 2017-08-05T09:52:48Z

cc/ @nodejs/streams

mcollina

Good work! There are only some nits to address!

mcollina · 2017-08-05T11:12:14Z

test/parallel/test-stream-transform-split-highwatermark.js

+// NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+// DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+// USE OR OTHER DEALINGS IN THE SOFTWARE.


the copyright header is not needed for new files

mcollina · 2017-08-05T11:12:38Z

test/parallel/test-stream-transform-split-highwatermark.js

+const DEFAULT = 16 * 1024;
+
+function test_transform(expected_readable_hwm, expected_writable_hwm, options) {
+  console.log(options);


Can you remove this console.log?

mcollina · 2017-08-05T11:14:00Z

test/parallel/test-stream-transform-split-highwatermark.js

+const Writable = stream.Writable;
+const DEFAULT = 16 * 1024;
+
+function test_transform(expected_readable_hwm, expected_writable_hwm, options) {


we use camelCase in general. You can use shorter names if you want.

guymguym · 2017-08-05T21:58:42Z

@mcollina Thanks for the review! I fixed the comments and squashed.

lpinca · 2017-08-06T08:29:31Z

CI: https://ci.nodejs.org/job/node-test-pull-request/9502/

guymguym · 2017-08-06T11:45:51Z

@lpinca I'm trying to understand if the failure is related to my change, and I suspect it doesn't.
Let me know if I'm missing anything.

lpinca · 2017-08-06T13:33:20Z

@guymguym you are correct, it's not related.

cjihrig

LGTM with a few comments.

cjihrig · 2017-08-07T15:32:51Z

test/parallel/test-stream-transform-split-highwatermark.js

+const assert = require('assert');
+const stream = require('stream');
+
+const Transform = stream.Transform;


Can you grab these 3 things using destructuring?

cjihrig · 2017-08-07T15:38:43Z

test/parallel/test-stream-transform-split-highwatermark.js

+testTransform(DEFAULT, DEFAULT, { writableHighWaterMark: NaN });
+
+// test highWaterMark = undefined, null, NaN
+testTransform(666, DEFAULT, {


You could reduce duplication by doing something like this throughout the test:

[undefined, null, NaN].forEach((highWaterMark) => { testTransform(666, DEFAULT, { highWaterMark, readableHighWaterMark: 666 }); });

Fixes: nodejs#14555

guymguym · 2017-08-07T18:42:11Z

@cjihrig Thanks, fixed and squashed.

mcollina

LGTM

mcollina · 2017-08-08T09:35:28Z

Landed as c3c045a.

mcollina · 2017-08-08T09:35:42Z

Congratulations for your first contribution to Node.js!

mcollina · 2017-08-08T09:36:14Z

I've added the semver-minor flag as it's a new feature.

This commits adds support for readableHighWaterMark and writableHighWaterMark in Duplex stream, so that they can be set without accessing the internal state. Fixes: #14555 PR-URL: #14636 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

refack

A few comments a bit late. If we get some agreement I'll open a PR.

refack · 2017-08-08T17:01:27Z

doc/api/stream.md

@@ -1752,6 +1752,10 @@ constructor and implement *both* the `readable._read()` and
  * `writableObjectMode` {boolean} Defaults to `false`. Sets `objectMode`
    for writable side of the stream. Has no effect if `objectMode`
    is `true`.
+  * `readableHighWaterMark` {number} Sets `highWaterMark` for the readable side


Suggestions:
Add a link to [#buffering]
Also in the first * (options) add links to [#constructor-new-streamwritableoptions] and [#new-streamreadableoptions]

refack · 2017-08-08T17:19:04Z

doc/api/stream.md

@@ -1752,6 +1752,10 @@ constructor and implement *both* the `readable._read()` and
  * `writableObjectMode` {boolean} Defaults to `false`. Sets `objectMode`
    for writable side of the stream. Has no effect if `objectMode`
    is `true`.
+  * `readableHighWaterMark` {number} Sets `highWaterMark` for the readable side
+    of the stream. Has no effect if `highWaterMark` is provided.


I'd phrase it as The general `highWaterMark` option overrides this one. IMHO a positive-active sentence is a little bit clearer than a negative-passive one.

I agree that positive is better. I just copied this format from readable/writableObjectMode.

refack · 2017-08-08T19:42:54Z

lib/_stream_readable.js

+  // However, some cases require setting options to different
+  // values for the readable and the writable sides of the duplex stream.
+  // These options can be provided separately as readableXXX and writableXXX.
+  var isDuplex = stream instanceof Stream.Duplex;


@mcollina isn't instanceof discouraged?

yes, but this was already there, this PR just moves it.

In my opinion it is best to allow it also for Readable/Writable without checking for Duplex, because this is an inherited behavior that behaves different on the base class vs the child class, and this is more confusing than helping. Not sure if this will be agreed by any developer reading the docs though. I wonder what you guys think.

refack · 2017-08-08T19:46:27Z

test/parallel/test-stream-transform-split-highwatermark.js

+});
+
+// test undefined, null, NaN
+[undefined, null, NaN].forEach((v) => {


what happens with ['', 'foo', ()=>{}, Symbol('frog'), {}, []]?

@refack The behavior will be unspecified in these cases, since highWaterMark is expected to be a number, but only checked to be a truthy value or 0.

This was also the case before my PR, though I would prefer it to be a specified behavior instead.

Perhaps the behavior should be to ignore non number values or else throw TypeError as a breaking change...

mcollina · 2017-08-08T20:39:01Z

@refack go ahead and fire a PR.

This commits adds support for readableHighWaterMark and writableHighWaterMark in Duplex stream, so that they can be set without accessing the internal state. Fixes: #14555 PR-URL: #14636 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

Ref: nodejs#14636

Ref: #14636 PR-URL: #14810 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

Notable changes * **Inspector** * `require()` is available in the inspector console now. [#8837](#8837) * **N-API** * New APIs for creating number values have been introduced. [#14573](#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](#14558) PR-URL: #14811

Notable changes * **HTTP2** * Experimental support for the built-in `http2` has been added via the `--expose-http2` flag. [#14239](#14239) * **Inspector** * `require()` is available in the inspector console now. [#8837](#8837) * Multiple contexts, as created by the `vm` module, are supported now. [#14465](#14465) * **N-API** * New APIs for creating number values have been introduced. [#14573](#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](#14558) PR-URL: #14811

Notable changes * **HTTP2** * Experimental support for the built-in `http2` has been added via the `--expose-http2` flag. [#14239](nodejs/node#14239) * **Inspector** * `require()` is available in the inspector console now. [#8837](nodejs/node#8837) * Multiple contexts, as created by the `vm` module, are supported now. [#14465](nodejs/node#14465) * **N-API** * New APIs for creating number values have been introduced. [#14573](nodejs/node#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](nodejs/node#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](nodejs/node#14558) PR-URL: nodejs/node#14811

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Aug 4, 2017

mcollina reviewed Aug 5, 2017

View reviewed changes

guymguym force-pushed the stream-issue-14555 branch from 771f20e to 078b4ee Compare August 5, 2017 21:56

cjihrig approved these changes Aug 7, 2017

View reviewed changes

stream: support readable/writableHighWaterMark for Duplex streams

13a8f7b

Fixes: nodejs#14555

guymguym force-pushed the stream-issue-14555 branch from 078b4ee to 13a8f7b Compare August 7, 2017 18:40

mcollina approved these changes Aug 8, 2017

View reviewed changes

mcollina closed this Aug 8, 2017

mcollina added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 8, 2017

refack reviewed Aug 8, 2017

View reviewed changes

addaleax added a commit to addaleax/node that referenced this pull request Aug 13, 2017

doc: add missing changes: metadata for streams

04c0387

Ref: nodejs#14636

addaleax mentioned this pull request Aug 13, 2017

doc: add missing changes: metadata for recent PRs #14810

Closed

addaleax added a commit that referenced this pull request Aug 13, 2017

doc: add missing changes: metadata for streams

7e0c2ea

Ref: #14636 PR-URL: #14810 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

addaleax added a commit that referenced this pull request Aug 13, 2017

doc: add missing changes: metadata for streams

4811fea

Ref: #14636 PR-URL: #14810 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

addaleax mentioned this pull request Aug 13, 2017

v8.4.0 proposal #14811

Merged

MylesBorins mentioned this pull request Jan 15, 2018

Semver Minor Tracking nodejs/Release#298

Closed

everhardt mentioned this pull request Dec 9, 2019

[node] stream: support readable/writableHWM for Duplex DefinitelyTyped/DefinitelyTyped#40843

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: support readable/writableHighWaterMark for Duplex streams #14636

stream: support readable/writableHighWaterMark for Duplex streams #14636

guymguym commented Aug 4, 2017

gibfahn commented Aug 5, 2017

mcollina left a comment

mcollina Aug 5, 2017

guymguym Aug 5, 2017

mcollina Aug 5, 2017

guymguym Aug 5, 2017

mcollina Aug 5, 2017

guymguym Aug 5, 2017

guymguym commented Aug 5, 2017

lpinca commented Aug 6, 2017

guymguym commented Aug 6, 2017

lpinca commented Aug 6, 2017

cjihrig left a comment

cjihrig Aug 7, 2017

guymguym Aug 7, 2017

cjihrig Aug 7, 2017

guymguym Aug 7, 2017

guymguym commented Aug 7, 2017

mcollina left a comment

mcollina commented Aug 8, 2017

mcollina commented Aug 8, 2017

mcollina commented Aug 8, 2017

refack left a comment

refack Aug 8, 2017

refack Aug 8, 2017

guymguym Sep 10, 2017

refack Aug 8, 2017

mcollina Aug 8, 2017

guymguym Sep 10, 2017

refack Aug 8, 2017

guymguym Sep 10, 2017

mcollina commented Aug 8, 2017

stream: support readable/writableHighWaterMark for Duplex streams #14636

stream: support readable/writableHighWaterMark for Duplex streams #14636

Conversation

guymguym commented Aug 4, 2017

Checklist

Affected core subsystem(s)

gibfahn commented Aug 5, 2017

mcollina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guymguym commented Aug 5, 2017

lpinca commented Aug 6, 2017

guymguym commented Aug 6, 2017

lpinca commented Aug 6, 2017

cjihrig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guymguym commented Aug 7, 2017

mcollina left a comment

Choose a reason for hiding this comment

mcollina commented Aug 8, 2017

mcollina commented Aug 8, 2017

mcollina commented Aug 8, 2017

refack left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Aug 8, 2017