Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

streams: implement Readable.from async iterator utility #27660

Closed
wants to merge 1 commit into from

Conversation

@guybedford
Copy link
Contributor

commented May 12, 2019

This implements a Readable.from utility method for easily converting async iterators into readable streams.

The implementation and approach are taken directly from the toReadable implementation in https://github.com/mcollina/stream-iterators-utils.

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
@nodejs-github-bot

This comment has been minimized.

@guybedford guybedford requested a review from mcollina May 12, 2019

@guybedford guybedford force-pushed the guybedford:readable-from branch from e951386 to 41a88b1 May 12, 2019

@devsnek

This comment has been minimized.

Copy link
Member

commented May 12, 2019

fun fact: web streams are also async iterators :P

Show resolved Hide resolved lib/_stream_readable.js Outdated
Show resolved Hide resolved lib/_stream_readable.js Outdated
@benjamingr
Copy link
Member

left a comment

Added some comments, generally LGTM and 👍

@benjamingr

This comment has been minimized.

Copy link
Member

commented May 12, 2019

fun fact: web streams are also async iterators :P

@devsnek oh cool that means you ended up following through and making that change?

Show resolved Hide resolved doc/api/stream.md Outdated
Show resolved Hide resolved lib/_stream_readable.js Outdated
Show resolved Hide resolved lib/_stream_readable.js Outdated
Show resolved Hide resolved doc/api/stream.md
Show resolved Hide resolved doc/api/stream.md Outdated
@targos

targos approved these changes May 13, 2019

@mcollina
Copy link
Member

left a comment

LGTM, awesome work!

@benjamingr
Copy link
Member

left a comment

This is great, thanks for this and for the changes 🙇

I think this, its counterpart of readable streams being async iterable, once promisify, its counterpart callbackify and the promise warnings mean we finally have a good async story for Node.js

I'm excited :D

Show resolved Hide resolved doc/api/stream.md
Show resolved Hide resolved doc/api/stream.md Outdated
@nodejs-github-bot

This comment has been minimized.

@guybedford guybedford force-pushed the guybedford:readable-from branch from fdc7d57 to a7a09a6 May 16, 2019

@nodejs-github-bot

This comment has been minimized.

@guybedford

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

I keep getting really odd CI errors for free-bsd only. Have retried it twice now and everytime it's a different error - https://ci.nodejs.org/job/node-test-commit-freebsd/nodes=freebsd11-x64/lastCompletedBuild/testReport/(root)/test/parallel_test_cli_node_options/.

Is this a known flake?

@nodejs-github-bot

This comment has been minimized.

@guybedford

This comment has been minimized.

Copy link
Contributor Author

commented May 18, 2019

Landed in 030fa2e.

@guybedford guybedford closed this May 18, 2019

guybedford added a commit that referenced this pull request May 18, 2019

stream: implement Readable.from async iterator utility
PR-URL: #27660
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>

targos added a commit that referenced this pull request May 18, 2019

stream: implement Readable.from async iterator utility
PR-URL: #27660
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>

BridgeAR added a commit that referenced this pull request May 21, 2019

2019-05-21, Version 12.3.0 (Current)
Notable changes:

* process:
  * Log errors using `util.inspect` in case of fatal exceptions
    (Ruben Bridgewater) #27243
* repl:
  * Add `process.on('uncaughtException')` support (Ruben Bridgewater)
    #27151
* stream:
  * Implemented `Readable.from` async iterator utility (Guy Bedford)
    #27660
* tls:
  * Expose built-in root certificates (Ben Noordhuis)
    #26415
  * Support `net.Server` options (Luigi Pinca)
    #27665
  * Expose `keylog` event on TLSSocket (Alba Mendez)
    #27654
* worker:
  * Added the ability to unshift messages from the `MessagePort`
    (Anna Henningsen) #27294

@BridgeAR BridgeAR referenced this pull request May 21, 2019

Merged

v12.3.0 proposal #27799

4 of 4 tasks complete

BridgeAR added a commit that referenced this pull request May 21, 2019

2019-05-21, Version 12.3.0 (Current)
Notable changes:

* esm:
  * Added the `--experimental-wasm-modules` flag to support
    WebAssembly modules (Myles Borins & Guy Bedford)
    #27659
* process:
  * Log errors using `util.inspect` in case of fatal exceptions
    (Ruben Bridgewater) #27243
* repl:
  * Add `process.on('uncaughtException')` support (Ruben Bridgewater)
    #27151
* stream:
  * Implemented `Readable.from` async iterator utility (Guy Bedford)
    #27660
* tls:
  * Expose built-in root certificates (Ben Noordhuis)
    #26415
  * Support `net.Server` options (Luigi Pinca)
    #27665
  * Expose `keylog` event on TLSSocket (Alba Mendez)
    #27654
* worker:
  * Added the ability to unshift messages from the `MessagePort`
    (Anna Henningsen) #27294

PR-URL: #27799

BridgeAR added a commit that referenced this pull request May 21, 2019

2019-05-21, Version 12.3.0 (Current)
Notable changes:

* esm:
  * Added the `--experimental-wasm-modules` flag to support
    WebAssembly modules (Myles Borins & Guy Bedford)
    #27659
* process:
  * Log errors using `util.inspect` in case of fatal exceptions
    (Ruben Bridgewater) #27243
* repl:
  * Add `process.on('uncaughtException')` support (Ruben Bridgewater)
    #27151
* stream:
  * Implemented `Readable.from` async iterator utility (Guy Bedford)
    #27660
* tls:
  * Expose built-in root certificates (Ben Noordhuis)
    #26415
  * Support `net.Server` options (Luigi Pinca)
    #27665
  * Expose `keylog` event on TLSSocket (Alba Mendez)
    #27654
* worker:
  * Added the ability to unshift messages from the `MessagePort`
    (Anna Henningsen) #27294

PR-URL: #27799
(async function() {
for await (const chunk of iterator) {
// Handle backpressure on write
if (!writeable.write(value))

This comment has been minimized.

Copy link
@felixfbecker

felixfbecker May 21, 2019

Contributor

where does value come from?

This comment has been minimized.

Copy link
@lpinca

lpinca May 21, 2019

Member

Should be chunk, feel free to open a PR.

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

@imcotton imcotton referenced this pull request Jun 11, 2019

Closed

chore(node): add Readable.from utility #36106

6 of 16 tasks complete

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

imcotton added a commit to imcotton/DefinitelyTyped that referenced this pull request Jun 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.