lib: implement various stream utils for webstreams #39517

ghost · 2021-07-25T16:58:40Z

Implements the finished, addAbortSignal, and pipeline functions for webstreams, and adds the documentation and tests for each function. The promisified versions of finished and pipeline are also added.

Refs: #39316

…g tests

Implements the finished, addAbortSignal, and pipeline functions for webstreams, and adds the documentation and tests for each function. The promisified versions of finished and pipeline are also added. Refs: #39316

VoltrexKeyva · 2021-07-25T17:12:36Z

lib/internal/webstreams/finished.js

+
+  throw new ERR_INVALID_ARG_TYPE(
+    'stream',
+    'ReadableStream|WritableStream|TransformStream',


Suggested change

'ReadableStream|WritableStream|TransformStream',

['ReadableStream', 'WritableStream', 'TransformStream'],

VoltrexKeyva · 2021-07-25T17:13:15Z

lib/internal/webstreams/pipeline.js

+  if (streams.some(cannotWriteToStream)) {
+    throw new ERR_INVALID_ARG_TYPE(
+      'streams',
+      'WritableStream|TransformStream',


Suggested change

'WritableStream|TransformStream',

['WritableStream', 'TransformStream'],

VoltrexKeyva · 2021-07-25T17:14:23Z

lib/internal/webstreams/pipeline.js

+  if (streams.some(cannotReadFromStream)) {
+    throw new ERR_INVALID_ARG_TYPE(
+      'streams',
+      'ReadableStream|TransformStream',


Suggested change

'ReadableStream|TransformStream',

['ReadableStream', 'TransformStream'],

VoltrexKeyva · 2021-07-25T17:15:23Z

A lot of unrelated style changes in lib/internal/webstreams/readablestream.js?

VoltrexKeyva · 2021-07-25T17:18:20Z

lib/internal/webstreams/readablestream.js

+
+function readableStreamAddFinishedCallback(stream, callback) {
+  assert(isReadableStream(stream));
+  stream[kState].finishedCallbacks.push(callback);


Suggested change

stream[kState].finishedCallbacks.push(callback);

ArrayPrototypePush(stream[kState].finishedCallbacks, callback);

VoltrexKeyva · 2021-07-25T17:20:42Z

lib/internal/webstreams/writablestream.js

      controller[kState].started = true;
      writableStreamDealWithRejection(stream, error);
    });
 }

+function writableStreamAddFinishedCallback(stream, callback) {
+  assert(isWritableStream(stream));
+  stream[kState].finishedCallbacks.push(callback);


Suggested change

stream[kState].finishedCallbacks.push(callback);

ArrayPrototypePush(stream[kState].finishedCallbacks, callback);

aduh95

The code contains utility functions that are only used once (such as signalIsAborted, validateInputs, throwErrorIfNotWebstream, addAbortCallbackToSignal, etc.), which makes the code hard to follow. Could you try to remove them? Instead you can use comments to explain what the code is doing.

aduh95 · 2021-07-25T22:02:16Z

doc/api/webstreams.md

@@ -1267,6 +1270,135 @@ added: REPLACEME
 -->

 * Type: {WritableStream}
+### `webstream.finished(stream, callback)`


Suggested change

### `webstream.finished(stream, callback)`

### `webstream.finished(stream, callback)`

aduh95 · 2021-07-25T22:02:36Z

doc/api/webstreams.md

+```
+The `finished` API also provides promise version:


Suggested change

```

The `finished` API also provides promise version:

```

The `finished` API also provides promise version:

ronag

We should just add support for web streams in existing stream functions. Not implement new ones.

ghost · 2021-07-26T06:58:31Z

A lot of unrelated style changes in lib/internal/webstreams/readablestream.js?

@VoltrexMaster The only formatting tool I used was eslint with the given config file in the repo, so I'm unsure as to why this has happened. Let me know if I'm missing something, but otherwise I'll see if I can sort this out.

ghost · 2021-07-26T07:03:44Z

We should just add support for web streams in existing stream functions. Not implement new ones.

@ronag Understood. Is it satisfactory to add a conditional flow in the existing functions which calls the functions added in this request?

ronag · 2021-07-26T07:25:16Z

lib/internal/webstreams/add-abort-signal.js

+  validateInputs(signal, stream);
+
+  const onAbort = () => {
+    stream.abort(new AbortError());


abort doesn't exist on web streams? Do they?

Only on WritableStream. On ReadableStream it needs to be cancel().

There is another fundamental problem with this, however... if the ReadableStream has a Reader, or the WritableStream has a Writer, their cancel() and abort() methods cannot be called directly... and there's no public API for knowing if either is locked. If they are, there's no public API for getting the Reader or the Writer and those have to be used for doing the canceling/aborting.

Instead, in order for us to do this, we need to rely on the internal functions and cannot rely on the public API. This also means that these will only work for the core implementations of web streams (userland implementations would not be supported).

@jasnell do you think this finished for webstreams is still a worthwhile development given that this is the case?

ronag · 2021-07-26T07:27:05Z

We should just add support for web streams in existing stream functions. Not implement new ones.

@ronag Understood. Is it satisfactory to add a conditional flow in the existing functions which calls the functions added in this request?

I think for addAbortSignal it's ok.

For pipeline I prefer #39519.

For finished I'm unsure. You are just converting it into a stream.Readable. Isn't there a better way? If that's all you do then you might as well do this as a one liner (if val is ReadableStream then val = val.readable) in stream.finished.

jasnell · 2021-07-26T14:25:09Z

doc/api/webstreams.md

@@ -33,6 +33,9 @@ There are three primary types of objects
 * `WritableStream` - Represents a destination for streaming data.
 * `TransformStream` - Represents an algorithm for transforming streaming data.

+Additionally, this module includes the utility functions
+`pipeline()`, `finished()`, and `addAbortSignal()`.


I'm -1 on having this also exported on stream/web. Just have the stream exports support the additional types.

jasnell · 2021-07-26T14:33:22Z

lib/internal/webstreams/readablestream.js

-const {
-  DOMException,
-} = internalBinding('messaging');
+const { DOMException } = internalBinding('messaging');


Unrelated style changes. Please leave the require statements formatted as they are.

jasnell · 2021-07-26T14:34:06Z

lib/internal/webstreams/readablestream.js

@@ -216,7 +207,8 @@ class ReadableStream {
      setupReadableByteStreamControllerFromSource(
        this,
        source,
-        extractHighWaterMark(highWaterMark, 0));
+        extractHighWaterMark(highWaterMark, 0)
+      );


All of the unrelated style changes in here make this difficult to review.

jasnell · 2021-07-26T14:34:48Z

lib/internal/webstreams/readablestream.js

+   */
+  abort(reason = undefined) {
+    this.cancel(reason);
+  }


We cannot add methods to the standard class.

jasnell · 2021-07-26T14:36:25Z

lib/internal/webstreams/transformstream.js

+  */
+  abort(reason = undefined) {
+    this?.readable?.abort?.(reason);
+  }


We cannot add methods to the standard API

jasnell · 2021-07-26T14:40:35Z

lib/internal/webstreams/writablestream.js

+  stream[kState].finishedCallbacks?.forEach((cb) => {
+    cb(stream[kState].storedError);
+  });
+}


I think this is the wrong approach. the WritableStream and ReadableStream APIs already have mechanisms for monitoring when each are closed via the closed promises on the Writer and Reader.

jasnell

I appreciate the work here but this has quite a few reasons why it's the wrong approach.

First, there are way too many unrelated style changes that make it impossible to effectively review. All of the purely style changes should be removed.

Second, this adds non-standard methods to the ReadableStream and TransportStream APIs that we just can't add.

Third, the abort controller implementation fails to account for locked streams.

Fourth, this should modify the existing require('stream').pipeline|finished|addAbortSignal() methods in place rather than creating new versions of them, creating new exports off stream/web, or introducing the new stream/web/promises.

Fifth, the way this adds the finish callbacks is problematic and definitely not ideal.

mcollina

I'm -1 for landing this. Support for webstreams should be added to the utilities we already have.

Please always tag @nodejs/streams when discussing streams PRs.

ghost · 2021-07-26T16:19:39Z

Thanks for all the feedback and sorry for it being a bit of a mess! Looks like I have some work to do...

Seeing as though it'll be a good few changes, I'm going to close this PR for now.

zwhitchcox · 2021-08-25T17:16:17Z

Any updates on this @sam-stanford ? Is this still in progress?

ghost · 2021-08-25T19:37:23Z

Any updates on this @sam-stanford ? Is this still in progress?

@zwhitchcox

Yep! Been away since I last took a look at this, so a little behind where is expected, but I'm making progress.

I'm still learning the ropes for node and open source as a whole, so if this is something which needs doing quickly, feel free to take the reigns :)

zwhitchcox · 2021-08-25T23:12:02Z

All right, I'll take a stab at it... Great work so far though!

sam-stanford added 2 commits July 25, 2021 17:20

add pipeline, addAbortSignal, and finished for webstreams with passin…

2e23211

…g tests

lib: implement various stream utils for webstreams

ca11064

Implements the finished, addAbortSignal, and pipeline functions for webstreams, and adds the documentation and tests for each function. The promisified versions of finished and pipeline are also added. Refs: #39316

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jul 25, 2021

VoltrexKeyva reviewed Jul 25, 2021

View reviewed changes

aduh95 reviewed Jul 25, 2021

View reviewed changes

ronag requested changes Jul 26, 2021

View reviewed changes

ronag added stream Issues and PRs related to the stream subsystem. web streams and removed lib / src Issues and PRs related to general changes in the lib or src directory. labels Jul 26, 2021

ronag reviewed Jul 26, 2021

View reviewed changes

jasnell reviewed Jul 26, 2021

View reviewed changes

jasnell requested changes Jul 26, 2021

View reviewed changes

mcollina requested changes Jul 26, 2021

View reviewed changes

ghost closed this Jul 29, 2021

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lib: implement various stream utils for webstreams #39517

lib: implement various stream utils for webstreams #39517

ghost commented Jul 25, 2021

VoltrexKeyva Jul 25, 2021

VoltrexKeyva Jul 25, 2021

VoltrexKeyva Jul 25, 2021

VoltrexKeyva commented Jul 25, 2021

VoltrexKeyva Jul 25, 2021

VoltrexKeyva Jul 25, 2021

aduh95 left a comment

aduh95 Jul 25, 2021

aduh95 Jul 25, 2021

ronag left a comment

ghost commented Jul 26, 2021

ghost commented Jul 26, 2021

ronag Jul 26, 2021

jasnell Jul 26, 2021

ghost Jul 29, 2021

ronag commented Jul 26, 2021 •

edited

Loading

jasnell Jul 26, 2021

jasnell Jul 26, 2021

jasnell Jul 26, 2021

jasnell Jul 26, 2021

jasnell Jul 26, 2021

jasnell Jul 26, 2021

jasnell left a comment

mcollina left a comment

ghost commented Jul 26, 2021 •

edited by ghost

Loading

zwhitchcox commented Aug 25, 2021 •

edited

Loading

ghost commented Aug 25, 2021

zwhitchcox commented Aug 25, 2021

	'ReadableStream\|WritableStream\|TransformStream',
	['ReadableStream', 'WritableStream', 'TransformStream'],

	'WritableStream\|TransformStream',
	['WritableStream', 'TransformStream'],

	'ReadableStream\|TransformStream',
	['ReadableStream', 'TransformStream'],

	stream[kState].finishedCallbacks.push(callback);
	ArrayPrototypePush(stream[kState].finishedCallbacks, callback);

	### `webstream.finished(stream, callback)`

	### `webstream.finished(stream, callback)`

lib: implement various stream utils for webstreams #39517

lib: implement various stream utils for webstreams #39517

Conversation

ghost commented Jul 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VoltrexKeyva commented Jul 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aduh95 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ronag left a comment

Choose a reason for hiding this comment

ghost commented Jul 26, 2021

ghost commented Jul 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ronag commented Jul 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell left a comment

Choose a reason for hiding this comment

mcollina left a comment

Choose a reason for hiding this comment

ghost commented Jul 26, 2021 • edited by ghost Loading

zwhitchcox commented Aug 25, 2021 • edited Loading

ghost commented Aug 25, 2021

zwhitchcox commented Aug 25, 2021

ronag commented Jul 26, 2021 •

edited

Loading

ghost commented Jul 26, 2021 •

edited by ghost

Loading

zwhitchcox commented Aug 25, 2021 •

edited

Loading