stream: simplify Transform stream implementation #32763

ronag · 2020-04-10T15:19:04Z

Significantly simplified Transform stream implementation by
using mostly standard stream code.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

ronag · 2020-04-10T15:27:13Z

lib/_stream_transform.js

-
-function done(stream, er, data) {
-  if (er)
-    return stream.emit('error', er);


emitting 'error' was a bug

nodejs-github-bot · 2020-04-10T15:31:31Z

CI: https://ci.nodejs.org/job/node-test-pull-request/30617/

Significantly simplified Transform stream implementation by using mostly standard stream code.

ronag · 2020-04-10T15:54:38Z

Some problems in zlib.

nodejs-github-bot · 2020-04-10T16:14:11Z

CI: https://ci.nodejs.org/job/node-test-pull-request/30618/

nodejs-github-bot · 2020-04-10T16:39:46Z

CI: https://ci.nodejs.org/job/node-test-pull-request/30620/

nodejs-github-bot · 2020-04-10T16:54:18Z

CI: https://ci.nodejs.org/job/node-test-pull-request/30621/

ronag · 2020-04-10T17:07:55Z

Marking as semver major since minor differences in behavior probably exists.

test/parallel/test-zlib-flush-drain.js

test/parallel/test-stream2-transform.js

nodejs-github-bot · 2020-04-10T17:32:14Z

CI: https://ci.nodejs.org/job/node-test-pull-request/30623/

_transformState is no longer used since Transform was simplified. Refs: nodejs#32763

jasnell · 2020-04-28T14:54:48Z

This really should have done a runtime deprecation for _transformState rather than removing it entirely. It's relatively low impact given that, as a semver-major, this won't be pulled in to any release until 15.0.0 but for 14.x we should add a proper deprecation for _transformState.

_transformState is no longer used since Transform was simplified. Refs: #32763 PR-URL: #33105 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

Transform._transformState is removed in future version as part of a refactoring. Refs: nodejs#32763 Refs: nodejs#33105 (comment)

Transform._transformState is removed in future version as part of a refactoring. Refs: #32763 Refs: #33105 (comment) Backport-PR-URL: #33126 PR-URL: #32763

Notable changes: build: * (SEMVER-MINOR) reset embedder string to "-node.0" (Michaël Zasso) #33376 cli: * (SEMVER-MINOR) add alias for report-directory to make it consistent (AshCripps) #33587 crypto: * (SEMVER-MINOR) allow KeyObjects in postMessage (Tobias Nießen) #33360 deps: * (SEMVER-MINOR) V8: cherry-pick 0d6debcc5f08 (Michaël Zasso) #33376 * (SEMVER-MINOR) update V8 to 8.3.110.9 (Michaël Zasso) #33376 dgram: * (SEMVER-MINOR) allow typed arrays in .send() (Sarat Addepalli) #22413 events: * (SEMVER-MINOR) initial implementation of experimental EventTarget (James M Snell) #33556 fs: * (SEMVER-MINOR) implement lutimes (Maël Nison) #33399 http: * (SEMVER-MINOR) expose host and protocol on ClientRequest (wenningplus) #33803 * (SEMVER-MINOR) add maxTotalSockets to agent class (rickyes) #33617 * (SEMVER-MINOR) return this from OutgoingMessage#destroy() (Colin Ihrig) #32789 * (SEMVER-MINOR) return this from ClientRequest#destroy() (Colin Ihrig) #32789 * (SEMVER-MINOR) return this from IncomingMessage#destroy() (Colin Ihrig) #32789 * (SEMVER-MINOR) added scheduling option to http agent (delvedor) #33278 http2: * (SEMVER-MINOR) return this for Http2ServerRequest#setTimeout (Pranshu Srivastava) #33994 * (SEMVER-MINOR) do not modify explicity set date headers (Pranshu Srivastava) #33160 process: * (SEMVER-MINOR) add unhandled-rejection throw and warn-with-error-code (Dan Fabulich) #33475 src: * (SEMVER-MINOR) store key data in separate class (Tobias Nießen) #33360 * (SEMVER-MINOR) add NativeKeyObject base class (Tobias Nießen) #33360 * (SEMVER-MINOR) rename internal key handles to KeyObjectHandle (Tobias Nießen) #33360 * (SEMVER-MINOR) add equality operators for BaseObjectPtr (Anna Henningsen) #33772 * (SEMVER-MINOR) introduce BaseObject base FunctionTemplate (Anna Henningsen) #33772 * (SEMVER-MINOR) add public APIs to manage v8::TracingController (Anna Henningsen) #33850 stream*: * runtime deprecate Transform._transformState (Robert Nagy) #32763 win: * (SEMVER-MINOR) allow skipping the supported platform check (João Reis) #33176 worker: * (SEMVER-MINOR) add public method for marking objects as untransferable (Anna Henningsen) #33979 * (SEMVER-MINOR) emit `'messagerror'` events for failed deserialization (Anna Henningsen) #33772 * (SEMVER-MINOR) allow passing JS wrapper objects via postMessage (Anna Henningsen) #33772 * (SEMVER-MINOR) allow transferring/cloning generic BaseObjects (Anna Henningsen) #33772 worker,fs: * (SEMVER-MINOR) make FileHandle transferable (Anna Henningsen) #33772 zlib: * (SEMVER-MINOR) add `maxOutputLength` option (unknown) #33516 PR-URL: #34093

Transform._transformState is removed in future version as part of a refactoring. Refs: #32763 Refs: #33105 (comment) Backport-PR-URL: #33126 PR-URL: #32763

Notable changes: build: * (SEMVER-MINOR) reset embedder string to "-node.0" (Michaël Zasso) #33376 cli: * (SEMVER-MINOR) add alias for report-directory to make it consistent (AshCripps) #33587 crypto: * (SEMVER-MINOR) allow KeyObjects in postMessage (Tobias Nießen) #33360 deps: * (SEMVER-MINOR) V8: cherry-pick 0d6debcc5f08 (Michaël Zasso) #33376 * (SEMVER-MINOR) update V8 to 8.3.110.9 (Michaël Zasso) #33376 dgram: * (SEMVER-MINOR) allow typed arrays in .send() (Sarat Addepalli) #22413 events: * (SEMVER-MINOR) initial implementation of experimental EventTarget (James M Snell) #33556 fs: * (SEMVER-MINOR) implement lutimes (Maël Nison) #33399 http: * (SEMVER-MINOR) expose host and protocol on ClientRequest (wenningplus) #33803 * (SEMVER-MINOR) add maxTotalSockets to agent class (rickyes) #33617 * (SEMVER-MINOR) return this from OutgoingMessage#destroy() (Colin Ihrig) #32789 * (SEMVER-MINOR) return this from ClientRequest#destroy() (Colin Ihrig) #32789 * (SEMVER-MINOR) return this from IncomingMessage#destroy() (Colin Ihrig) #32789 * (SEMVER-MINOR) added scheduling option to http agent (delvedor) #33278 http2: * (SEMVER-MINOR) return this for Http2ServerRequest#setTimeout (Pranshu Srivastava) #33994 * (SEMVER-MINOR) do not modify explicity set date headers (Pranshu Srivastava) #33160 process: * (SEMVER-MINOR) add unhandled-rejection throw and warn-with-error-code (Dan Fabulich) #33475 src: * (SEMVER-MINOR) store key data in separate class (Tobias Nießen) #33360 * (SEMVER-MINOR) add NativeKeyObject base class (Tobias Nießen) #33360 * (SEMVER-MINOR) rename internal key handles to KeyObjectHandle (Tobias Nießen) #33360 * (SEMVER-MINOR) add equality operators for BaseObjectPtr (Anna Henningsen) #33772 * (SEMVER-MINOR) introduce BaseObject base FunctionTemplate (Anna Henningsen) #33772 * (SEMVER-MINOR) add public APIs to manage v8::TracingController (Anna Henningsen) #33850 stream*: * runtime deprecate Transform._transformState (Robert Nagy) #32763 win: * (SEMVER-MINOR) allow skipping the supported platform check (João Reis) #33176 worker: * (SEMVER-MINOR) add public method for marking objects as untransferable (Anna Henningsen) #33979 * (SEMVER-MINOR) emit `'messagerror'` events for failed deserialization (Anna Henningsen) #33772 * (SEMVER-MINOR) allow passing JS wrapper objects via postMessage (Anna Henningsen) #33772 * (SEMVER-MINOR) allow transferring/cloning generic BaseObjects (Anna Henningsen) #33772 worker,fs: * (SEMVER-MINOR) make FileHandle transferable (Anna Henningsen) #33772 zlib: * (SEMVER-MINOR) add `maxOutputLength` option (unknown) #33516 PR-URL: #34093

Notable changes: Deprecations and Removals: - **build**: remove --build-v8-with-gn configure option (Yang Guo) (#27576) - **build**: drop support for VS2017 (Michaël Zasso) (#33694) - **doc**: move DEP0018 to End-of-Life (Rich Trott) (#35316) - **fs**: deprecation warning on recursive rmdir (Ian Sutherland) (#35562) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **net**: remove long deprecated server.connections property (James M Snell) (#33647) - **repl**: remove deprecated repl.memory function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.turnOffEditorMode() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.parseREPLKeyword() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated bufferedCommand property (Ruben Bridgewater) (#33286) - **repl**: remove deprecated .rli (Ruben Bridgewater) (#33286) - **src**: remove deprecated node debug command (James M Snell) (#33648) - **timers**: introduce timers/promises (James M Snell) (#33950) - **util**: change default value of `maxStringLength` to 10000 (unknown) (#32744) - **wasi**: drop --experimental-wasm-bigint requirement (Colin Ihrig) (#35415) npm 7 (#35631): Node.js 15 comes with a new major release of npm, npm 7. npm 7 comes with many new features - including npm workspaces and a new package-lock.json format. npm 7 also includes yarn.lock file support. One of the big changes in npm 7 is that peer dependencies are now installed by default. Throw On Unhandled Rejections (#33021): As of Node.js 15, the default mode for `unhandledRejection` is changed to `throw` (from `warn`). In `throw` mode, if an `unhandledRejection` hook is not set, the `unhandledRejection` is raised as an uncaught exception. Users that have an `unhandledRejection` hook should see no change in behavior, and it’s still possible to switch modes using the `--unhandled-rejections=mode` process flag. QUIC (#32379): Node.js 15 comes with experimental support QUIC, which can be enabled by compiling Node.js with the `--experimental-quic` configuration flag. The Node.js QUIC implementation is exposed by the core `net` module. V8 8.6 (#35415): The V8 JavaScript engine has been updated to V8 8.6 (V8 8.4 is the latest available in Node.js 14). Along with performance tweaks and improvements the V8 update also brings the following language features: * `Promise.any()` (from V8 8.5) * `AggregateError` (from V8 8.5) * `String.prototype.replaceAll()` (from V8 8.5) * Logical assignment operators `&&=`, `||=`, and `??=` (from V8 8.5) Other Notable Changes: - **assert**: add `assert/strict` alias module (ExE Boss) (#34001) - **dns**: add dns/promises alias (shisama) (#32953) - **fs**: reimplement read and write streams using stream.construct (Robert Nagy) (#29656) - **http2**: allow Host in HTTP/2 requests (Alba Mendez) (#34664) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **lib**: unflag AbortController (James M Snell) (#33527) - **lib**: initial experimental AbortController implementation (James M Snell) (#33527) - **net**: autoDestroy Socket (Robert Nagy) (#31806) - **src**: disallow JS execution inside FreeEnvironment (Anna Henningsen) (#33874) - **stream**: construct (Robert Nagy) (#29656) - **worker**: make MessageEvent class more Web-compatible (Anna Henningsen) (#35496) Semver-Major Commits: - **assert**: add `assert/strict` alias module (ExE Boss) (#34001) - **build**: reset embedder string to "-node.0" (Michaël Zasso) (#35415) - **build**: remove --build-v8-with-gn configure option (Yang Guo) (#27576) - **build**: drop support for VS2017 (Michaël Zasso) (#33694) - **crypto**: refactoring internals, add WebCrypto (James M Snell) (#35093) - **crypto**: move node\_crypto files to src/crypto (James M Snell) (#35093) - **deps**: V8: cherry-pick d76abfed3512 (Michaël Zasso) (#35415) - **deps**: V8: cherry-pick 717543bbf0ef (Michaël Zasso) (#35415) - **deps**: V8: cherry-pick 6be2f6e26e8d (Michaël Zasso) (#35415) - **deps**: fix V8 build issue with inline methods (Jiawen Geng) (#35415) - **deps**: fix platform-embedded-file-writer-win for ARM64 (Michaël Zasso) (#35415) - **deps**: update V8 postmortem metadata script (Colin Ihrig) (#35415) - **deps**: update V8 to 8.6.395 (Michaël Zasso) (#35415) - **deps**: upgrade npm to 7.0.0 (Myles Borins) (#35631) - **deps**: update npm to 7.0.0-rc.3 (Myles Borins) (#35474) - **deps**: V8: cherry-pick 0d6debcc5f08 (Gus Caplan) (#33600) - **dns**: add dns/promises alias (shisama) (#32953) - **doc**: move DEP0018 to End-of-Life (Rich Trott) (#35316) - **doc**: update support macos version for 15.x (Ash Cripps) (#35022) - **fs**: deprecation warning on recursive rmdir (Ian Sutherland) (#35562) - **fs**: reimplement read and write streams using stream.construct (Robert Nagy) (#29656) - **http**: fixed socket.setEncoding fatal error (iskore) (#33405) - **http**: emit 'error' on aborted server request (Robert Nagy) (#33172) - **http**: cleanup end argument handling (Robert Nagy) (#31818) - **http2**: allow Host in HTTP/2 requests (Alba Mendez) (#34664) - **http2**: add `invalidheaders` test (Pranshu Srivastava) (#33161) - **http2**: refactor state code validation for the http2Stream class (rickyes) (#33535) - **http2**: header field valid checks (Pranshu Srivastava) (#33193) - **lib**: add EventTarget-related browser globals (Anna Henningsen) (#35496) - **lib**: remove ERR\_INVALID\_OPT\_VALUE and ERR\_INVALID\_OPT\_VALUE\_ENCODING (Denys Otrishko) (#34682) - **lib**: handle one of args case in ERR\_MISSING\_ARGS (Denys Otrishko) (#34022) - **lib**: remove NodeError from the prototype of errors with code (Michaël Zasso) (#33857) - **lib**: unflag AbortController (James M Snell) (#33527) - **lib**: initial experimental AbortController implementation (James M Snell) (#33527) - **net**: check args in net.connect() and socket.connect() calls (Denys Otrishko) (#34022) - **net**: remove long deprecated server.connections property (James M Snell) (#33647) - **net**: autoDestroy Socket (Robert Nagy) (#31806) - **process**: update v8 fast api calls usage (Maya Lekova) (#35415) - **process**: change default --unhandled-rejections=throw (Dan Fabulich) (#33021) - **process**: use v8 fast api calls for hrtime (Gus Caplan) (#33600) - **process**: delay throwing an error using `throwDeprecation` (Ruben Bridgewater) (#32312) - **repl**: remove deprecated repl.memory function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.turnOffEditorMode() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated repl.parseREPLKeyword() function (Ruben Bridgewater) (#33286) - **repl**: remove deprecated bufferedCommand property (Ruben Bridgewater) (#33286) - **repl**: remove deprecated .rli (Ruben Bridgewater) (#33286) - **src**: implement NodePlatform::PostJob (Clemens Backes) (#35415) - **src**: update NODE\_MODULE\_VERSION to 88 (Michaël Zasso) (#35415) - **src**: error reporting on CPUUsage (Yash Ladha) (#34762) - **src**: use node:moduleName as builtin module filename (Michaël Zasso) (#35498) - **src**: enable wasm trap handler on windows (Gus Caplan) (#35033) - **src**: update NODE\_MODULE\_VERSION to 86 (Michaël Zasso) (#33579) - **src**: disallow JS execution inside FreeEnvironment (Anna Henningsen) (#33874) - **src**: remove \_third\_party\_main support (Anna Henningsen) (#33971) - **src**: remove deprecated node debug command (James M Snell) (#33648) - **src**: remove unused CancelPendingDelayedTasks (Anna Henningsen) (#32859) - **stream**: try to wait for flush to complete before 'finish' (Robert Nagy) (#34314) - **stream**: cleanup and fix Readable.wrap (Robert Nagy) (#34204) - **stream**: add promises version to utility functions (rickyes) (#33991) - **stream**: fix writable.end callback behavior (Robert Nagy) (#34101) - **stream**: construct (Robert Nagy) (#29656) - **stream**: write should throw on unknown encoding (Robert Nagy) (#33075) - **stream**: fix \_final and 'prefinish' timing (Robert Nagy) (#32780) - **stream**: simplify Transform stream implementation (Robert Nagy) (#32763) - **stream**: use callback to properly propagate error (Robert Nagy) (#29179) - **test**: update tests after increasing typed array size to 4GB (Kim-Anh Tran) (#35415) - **test**: fix tests for npm 7.0.0 (Myles Borins) (#35631) - **test**: fix test suite to work with npm 7 (Myles Borins) (#35474) - **test**: update WPT harness and tests (Michaël Zasso) (#33770) - **timers**: introduce timers/promises (James M Snell) (#33950) - **tools**: disable x86 safe exception handlers in V8 (Michaël Zasso) (#35415) - **tools**: update V8 gypfiles for 8.6 (Ujjwal Sharma) (#35415) - **tools**: update V8 gypfiles for 8.5 (Ujjwal Sharma) (#35415) - **url**: file URL path normalization (Daijiro Wachi) (#35477) - **url**: verify domain is not empty after "ToASCII" (Michaël Zasso) (#33770) - **url**: remove U+0000 case in the fragment state (Michaël Zasso) (#33770) - **url**: remove gopher from special schemes (Michaël Zasso) (#33325) - **url**: forbid lt and gt in url host code point (Yash Ladha) (#33328) - **util**: change default value of `maxStringLength` to 10000 (unknown) (#32744) - **wasi**: drop --experimental-wasm-bigint requirement (Colin Ihrig) (#35415) - **win, child_process**: sanitize env variables (Bartosz Sosnowski) (#35210) - **worker**: make MessageEvent class more Web-compatible (Anna Henningsen) (#35496) - **worker**: set trackUnmanagedFds to true by default (Anna Henningsen) (#34394) - **worker**: rename error code to be more accurate (Anna Henningsen) (#33872) PR-URL: #35014

ronag force-pushed the transform-simplify branch from aade81a to a934d7f Compare April 10, 2020 15:19

nodejs-github-bot added errors Issues and PRs related to JavaScript errors originated in Node.js core. stream Issues and PRs related to the stream subsystem. labels Apr 10, 2020

ronag force-pushed the transform-simplify branch from a934d7f to ea3509c Compare April 10, 2020 15:23

ronag commented Apr 10, 2020

View reviewed changes

ronag force-pushed the transform-simplify branch from ea3509c to 88e8a20 Compare April 10, 2020 15:29

ronag added the wip Issues and PRs that are still a work in progress. label Apr 10, 2020

ronag force-pushed the transform-simplify branch from 88e8a20 to f84621d Compare April 10, 2020 15:51

stream: simplify Transform stream implementation

6099382

Significantly simplified Transform stream implementation by using mostly standard stream code.

ronag force-pushed the transform-simplify branch from f84621d to 6099382 Compare April 10, 2020 15:52

ronag added 2 commits April 10, 2020 18:08

fixup: zlib

8a54b1f

fixup: zlib

8cf4a58

ronag added 2 commits April 10, 2020 18:14

fixup: test

2f5da16

fixup: cleanup

e12e6b4

ronag removed the wip Issues and PRs that are still a work in progress. label Apr 10, 2020

fixup: lint

5cd52b7

ronag requested review from addaleax, mcollina and lpinca April 10, 2020 17:07

ronag added the semver-major PRs that contain breaking changes and should be released in the next major version. label Apr 10, 2020

ronag commented Apr 10, 2020

View reviewed changes

test/parallel/test-zlib-flush-drain.js Show resolved Hide resolved

ronag commented Apr 10, 2020

View reviewed changes

test/parallel/test-stream2-transform.js Show resolved Hide resolved

fixup

dce04fb

ronag added a commit to nxtedition/node that referenced this pull request Apr 27, 2020

stream: remove unused _transformState

5b92a54

_transformState is no longer used since Transform was simplified. Refs: nodejs#32763

ronag mentioned this pull request Apr 27, 2020

stream: remove unused _transformState #33105

Closed

4 tasks

ronag mentioned this pull request Apr 28, 2020

[v14.x] stream: runtime deprecate Transform._transformState #33126

Closed

4 tasks

ronag added a commit to nxtedition/node that referenced this pull request Jun 28, 2020

stream: runtime deprecate Transform._transformState

b3a32c2

Transform._transformState is removed in future version as part of a refactoring. Refs: nodejs#32763 Refs: nodejs#33105 (comment)

codebytere pushed a commit that referenced this pull request Jun 30, 2020

stream: runtime deprecate Transform._transformState

db2d1ca

Transform._transformState is removed in future version as part of a refactoring. Refs: #32763 Refs: #33105 (comment) Backport-PR-URL: #33126 PR-URL: #32763

codebytere mentioned this pull request Jun 30, 2020

v14.5.0 proposal #34093

Merged

codebytere pushed a commit that referenced this pull request Jun 30, 2020

stream: runtime deprecate Transform._transformState

9637ee6

Transform._transformState is removed in future version as part of a refactoring. Refs: #32763 Refs: #33105 (comment) Backport-PR-URL: #33126 PR-URL: #32763

BethGriggs mentioned this pull request Sep 1, 2020

2020-10-20, Version 15.0.0 (Current) #35014

Merged

targos added the backported-to-v14.x label Apr 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: simplify Transform stream implementation #32763

stream: simplify Transform stream implementation #32763

ronag commented Apr 10, 2020 •

edited

ronag Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

ronag commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

ronag commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

jasnell commented Apr 28, 2020

stream: simplify Transform stream implementation #32763

stream: simplify Transform stream implementation #32763

Conversation

ronag commented Apr 10, 2020 • edited

Checklist

ronag Apr 10, 2020

Choose a reason for hiding this comment

nodejs-github-bot commented Apr 10, 2020

ronag commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

ronag commented Apr 10, 2020

nodejs-github-bot commented Apr 10, 2020

jasnell commented Apr 28, 2020

ronag commented Apr 10, 2020 •

edited