New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream: always defer 'readable' with nextTick #17979

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@mcollina
Member

mcollina commented Jan 4, 2018

Fixes: #3203

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

stream

@mcollina

This comment has been minimized.

Member

mcollina commented Jan 4, 2018

This solves a recursive push() -> _read() - > push() cycle when multiple push() happen asynchronously. I'm not 100% sure this is something we want to do at this point, nor if this is the right way to achieve this.

It's tagged as semver-major because I feel it might break some edge cases. We might want to revert this quickly if that is the case.

cc @nodejs/streams @nodejs/tsc

@jasnell

jasnell approved these changes Jan 4, 2018

@addaleax

Seems pretty reasonable to me

The dicer CITGM failures appear to be real (or are pretty new at least), /cc @mscdex

if (chunk.length === 32 * 1024) { // first chunk
readable.push(Buffer.alloc(34 * 1024)); // above hwm
// We should check if awaitDrain counter is increased in the next
// tick, because when the 'data' event is fired

This comment has been minimized.

@addaleax

addaleax Jan 4, 2018

Member

This sentence seems like it’s missing something?

process.nextTick(common.mustCall(() => {
readable.push(null);
}));
}), 10);

This comment has been minimized.

@addaleax

addaleax Jan 4, 2018

Member

Isn’t setImmediate() enough here?

asyncReadable.push(null);
}));
assert.strictEqual(asyncReadable._readableState.needReadable, false);
}), 10);

This comment has been minimized.

@addaleax

addaleax Jan 4, 2018

Member

Ditto … using setTimeout() tends to introduce race conditions because they might expire before the current event loop turn is over, but setImmediate() always defers (and doesn’t keep test running longer than they need to)

@mcollina

This comment has been minimized.

Member

mcollina commented Jan 5, 2018

Fixed nits.
The dicer regression is legit, I'll dig into it.

@mcollina

This comment has been minimized.

@benjamingr

This comment has been minimized.

Member

benjamingr commented Jan 5, 2018

Benchmarks?

@addaleax

This comment has been minimized.

Member

addaleax commented Jan 5, 2018

}
if (data.length === 0) {
console.log('pushing null');

This comment has been minimized.

@jasnell

jasnell Jan 5, 2018

Member

would prefer to omit the console.log :-)

});
readable.on('end', common.mustCall(() => {
assert.strictEqual(i, (Math.floor(MAX / BATCH) + 1) * BATCH);

This comment has been minimized.

@jasnell

jasnell Jan 5, 2018

Member

Please add a comment that explains the calculation

@jasnell

This comment has been minimized.

Member

jasnell commented Jan 5, 2018

Still looks good to me. Couple of nits.

@mcollina

This comment has been minimized.

Member

mcollina commented Jan 5, 2018

                                                   improvement confidence   p.value

streams/readable-bigread.js n=1000 0.40 % 0.7330756
streams/readable-bigunevenread.js n=1000 -0.31 % 0.7630930
streams/readable-boundaryread.js type="buffer" n=2000 -1.21 % 0.2752480
streams/readable-boundaryread.js type="string" n=2000 -0.85 % 0.4784480
streams/readable-readall.js n=5000 -1.52 % 0.1030397
streams/readable-unevenread.js n=1000 0.57 % 0.1487255
streams/transform-creation.js n=1000000 3.56 % 0.1932558
streams/writable-manywrites.js n=2000000 -2.82 % 0.2757697

No perf regression!

@mcollina

This comment has been minimized.

Member

mcollina commented Jan 9, 2018

Added docs, PTAL

@jasnell

This comment has been minimized.

Member

jasnell commented Jan 10, 2018

docs LGTM

@mcollina

This comment has been minimized.

Member

mcollina commented Jan 10, 2018

Landed as 1e0f331.

@mcollina mcollina closed this Jan 10, 2018

mcollina added a commit that referenced this pull request Jan 10, 2018

stream: always defer 'readable' with nextTick
Emit 'readable' always in the next tick, resulting in a single
call to _read() per microtick. This removes the need for the
user to implement buffering if they wanted to call this.push()
multiple times in an asynchronous fashion, as this.push() triggers
this._read() call.

PR-URL: #17979
Fixes: #3203
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

@mafintosh mafintosh referenced this pull request Jan 25, 2018

Closed

stream: fix readable being emitted when nothing to do #18372

4 of 4 tasks complete

mcollina added a commit that referenced this pull request Jan 29, 2018

stream: do not emit readable if the stream ended
Fixes a regression introduced by the once-per-microtick 'readable'
event emission.

See: #17979
PR-URL: #18372
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

kcaulfield94 added a commit to kcaulfield94/node that referenced this pull request Feb 2, 2018

stream: do not emit readable if the stream ended
Fixes a regression introduced by the once-per-microtick 'readable'
event emission.

See: nodejs#17979
PR-URL: nodejs#18372
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

msoechting added a commit to hpicgs/node that referenced this pull request Feb 5, 2018

stream: always defer 'readable' with nextTick
Emit 'readable' always in the next tick, resulting in a single
call to _read() per microtick. This removes the need for the
user to implement buffering if they wanted to call this.push()
multiple times in an asynchronous fashion, as this.push() triggers
this._read() call.

PR-URL: nodejs#17979
Fixes: nodejs#3203
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

msoechting added a commit to hpicgs/node that referenced this pull request Feb 5, 2018

stream: do not emit readable if the stream ended
Fixes a regression introduced by the once-per-microtick 'readable'
event emission.

See: nodejs#17979
PR-URL: nodejs#18372
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

@mcollina mcollina deleted the mcollina:stream-next-tick-readable branch Feb 7, 2018

msoechting added a commit to hpicgs/node that referenced this pull request Feb 7, 2018

stream: always defer 'readable' with nextTick
Emit 'readable' always in the next tick, resulting in a single
call to _read() per microtick. This removes the need for the
user to implement buffering if they wanted to call this.push()
multiple times in an asynchronous fashion, as this.push() triggers
this._read() call.

PR-URL: nodejs#17979
Fixes: nodejs#3203
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

msoechting added a commit to hpicgs/node that referenced this pull request Feb 7, 2018

stream: do not emit readable if the stream ended
Fixes a regression introduced by the once-per-microtick 'readable'
event emission.

See: nodejs#17979
PR-URL: nodejs#18372
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

MayaLekova added a commit to MayaLekova/node that referenced this pull request May 8, 2018

stream: do not emit readable if the stream ended
Fixes a regression introduced by the once-per-microtick 'readable'
event emission.

See: nodejs#17979
PR-URL: nodejs#18372
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
@nevercast

This comment has been minimized.

nevercast commented May 16, 2018

This landed is Node.js v10.1 yes? That is how I should interpret the linked commits?

@mcollina

This comment has been minimized.

Member

mcollina commented May 16, 2018

Yes this is in 10.1

squaremo added a commit to squaremo/amqp.node that referenced this pull request Jul 16, 2018

Reorder mux test to account for async reads
The behaviour of the `'readable'` event changed, or was tightened up,
in

    nodejs/node#17979

such that it is _always_ called on the next tick. This change appears
first in Node.JS v10.0).

Since we rely on `'readable' in the multiplexing, it means that we
have to be more careful about when we wait for the next event loop to
come around in tests.

The tests tend to be much more brittle than live code with respect to
the order things happen, since they attempt to nail down precise
states or orderings (e.g., "the `unpipe` happened exactly between
these writes").

squaremo added a commit to squaremo/amqp.node that referenced this pull request Aug 3, 2018

Reorder mux test because 'readable' event deferred
The behaviour of the `'readable'` event changed, or was tightened up,
in

    nodejs/node#17979

such that it is _always_ called on the next tick. This change appears
first in Node.JS v10.0).

Since we rely on `'readable' in the multiplexing, it means that we
have to be more careful about when we wait for the next event loop to
come around in tests.

The tests tend to be much more brittle than live code with respect to
the order things happen, since they attempt to nail down precise
states or orderings (e.g., "the `unpipe` happened exactly between
these writes").

@mcollina mcollina referenced this pull request Aug 30, 2018

Closed

stream-whatwg: add whatwg streams #22352

4 of 4 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment