flaky: test-stream-pipeline-http2 #24456
Labels
flaky-test
Issues and PRs related to the tests with unstable failures on the CI.
stream
Issues and PRs related to the stream subsystem.
windows
Issues and PRs related to the Windows platform.
Projects
Comments
refack
added
flaky-test
4 tasks
|
Replicable locally. Move to $ tools/test.py --repeat 192 -j 96 test/parallel/test-stream-pipeline.js
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
=== release test-stream-pipeline ===
Path: parallel/test-stream-pipeline
Command: out/Release/node /Users/trott/io.js/test/parallel/test-stream-pipeline.js
--- TIMEOUT ---
[02:19|% 100|+ 168|- 24]: Done
$ |
|
Does running with |
Yes. $ tools/test.py --repeat 192 -j 1 test/parallel/test-stream-pipeline.js
[00:58|% 100|+ 192|- 0]: Done
$ |
|
Since this was obsered on Windows2008R2 Seems like "sequential"izing won't solve this. |
3 tasks
|
Job: https://ci.nodejs.org/job/node-test-binary-windows/21717/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=1 15:42:10 not ok 409 parallel/test-stream-pipeline
15:42:10 ---
15:42:10 duration_ms: 120.88
15:42:10 severity: fail
15:42:10 exitcode: 1
15:42:10 stack: |-
15:42:10 timeout
15:42:10 ... |
4 tasks
|
Some more context from the 11:04:29 not ok 1 parallel/test-stream-pipeline
11:04:29 ---
11:04:29 duration_ms: 120.160
11:04:29 severity: fail
11:04:29 exitcode: 1
11:04:29 stack: |-
11:04:29 timeout
11:04:29 ...
11:04:29 2 OK: 0 NOT OK: 2 TOTAL: 200
11:04:30 TAP version 13
11:04:30 1..1
11:04:30 ok 1 parallel/test-stream-pipeline
11:04:30 ---
11:04:30 duration_ms: 0.492
11:04:30 ...Seems like a race condition? |
Trott
added a commit
to Trott/io.js
that referenced
this issue
Nov 24, 2018
Splitting out the http2 portion of the test has a few benfits: * We don't skip the rest of the tests if `node` is compiled without crypto. * We can find out if the http2 portion of the test is responsible for the timeouts reported in issue 24456. Refs: nodejs#24456
3 tasks
|
#24631 is an attempt to diagnose this a bit by splitting the |
3 tasks
Trott
added a commit
to Trott/io.js
that referenced
this issue
Nov 25, 2018
Splitting out the http2 portion of the test has a few benfits: * We don't skip the rest of the tests if `node` is compiled without crypto. * We can find out if the http2 portion of the test is responsible for the timeouts reported in issue 24456. Refs: nodejs#24456 PR-URL: nodejs#24631 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
|
Now that #24631 has landed, the problematic part of the test is in its own test file ( |
targos
pushed a commit
that referenced
this issue
Nov 25, 2018
Splitting out the http2 portion of the test has a few benfits: * We don't skip the rest of the tests if `node` is compiled without crypto. * We can find out if the http2 portion of the test is responsible for the timeouts reported in issue 24456. Refs: #24456 PR-URL: #24631 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
rvagg
pushed a commit
that referenced
this issue
Nov 28, 2018
Splitting out the http2 portion of the test has a few benfits: * We don't skip the rest of the tests if `node` is compiled without crypto. * We can find out if the http2 portion of the test is responsible for the timeouts reported in issue 24456. Refs: #24456 PR-URL: #24631 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
|
I am also seeing this on macOS |
'use strict';
const common = require('../common');
+const assert = require('assert');
+
if (!common.hasCrypto)
common.skip('missing crypto');
const { Readable, pipeline } = require('stream');
@@ -14,8 +16,14 @@ const http2 = require('http2');
server.listen(0, () => {
const url = `http://localhost:${server.address().port}`;
const client = http2.connect(url);
+ client.on('error', (err) => {
+ console.log(`'error':`, err);
+ assert.ifError(err);
+ });
+
const req = client.request({ ':method': 'POST' });
+ let cnt = 10;
const rs = new Readable({
read() {
rs.push('hello');
@@ -23,14 +31,21 @@ const http2 = require('http2');
});
pipeline(rs, req, common.mustCall((err) => {
+ common.expectsError({
+ code: 'ERR_STREAM_PREMATURE_CLOSE'
+ })(err);
+ console.log('tearing down server and client');
server.close();
client.close();
}));
- let cnt = 10;
req.on('data', (data) => {
+ console.log(`'data' event: ${cnt}, ${data.length}`);
cnt--;
- if (cnt === 0) rs.destroy();
+ if (cnt === 0) {
+ console.log('destroying rs');
+ rs.destroy();
+ }
});
});
}I am able to produce this locally. Looks like somehow the callback of |
with NODE_DEBUG=http2 enabled |
|
With NODE_DEBUG=* enabled, this should be what happens after see logFor comparison this is what happens when the test does not time out see log |
|
cc @nodejs/streams any ideas what may cause the callback of |
|
I do not see any net activity whatsoever when it times out. This is weird and probably the source of the issue. |
|
I did a When that commit was introduced, this test file did not exist. This test was pulled out of Pulling this test out of that file and into its own |
|
@addaleax ^^^^^ |
Trott
added a commit
to Trott/io.js
that referenced
this issue
Dec 8, 2018
This reverts commit d3f02d0. Fixes: nodejs#24456
2 tasks
addaleax
added a commit
to addaleax/node
that referenced
this issue
Dec 9, 2018
HTTP/2 streams call `.end()` on themselves from their `.destroy()` method, which might be queued (e.g. due to network congestion) and not processed before the stream itself is destroyed. In that case, the `_writableState.ended` property could be set before the stream emits its `'close'` event, and never actually emits the `'finished'` event, confusing the end-of-stream implementation so that it wouldn’t call its callback. This can be fixed by watching for the end events themselves using the existing `'finish'` and `'end'` listeners rather than relying on the `.ended` properties of the `_...State` objects. These properties still need to be checked to know whether stream closure was premature – My understanding is that ideally, streams should not emit `'close'` before `'end'` and/or `'finished'`, so this might be another bug, but changing this would require modifying tests and almost certainly be a breaking change. Fixes: nodejs#24456
3 tasks
|
This should be addressed by #24926 :) |
Trott
pushed a commit
to Trott/io.js
that referenced
this issue
Dec 11, 2018
HTTP/2 streams call `.end()` on themselves from their `.destroy()` method, which might be queued (e.g. due to network congestion) and not processed before the stream itself is destroyed. In that case, the `_writableState.ended` property could be set before the stream emits its `'close'` event, and never actually emits the `'finished'` event, confusing the end-of-stream implementation so that it wouldn’t call its callback. This can be fixed by watching for the end events themselves using the existing `'finish'` and `'end'` listeners rather than relying on the `.ended` properties of the `_...State` objects. These properties still need to be checked to know whether stream closure was premature – My understanding is that ideally, streams should not emit `'close'` before `'end'` and/or `'finished'`, so this might be another bug, but changing this would require modifying tests and almost certainly be a breaking change. Fixes: nodejs#24456 PR-URL: nodejs#24926 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
|
Fixed in 83ec33b |
BethGriggs
pushed a commit
that referenced
this issue
Dec 17, 2018
HTTP/2 streams call `.end()` on themselves from their `.destroy()` method, which might be queued (e.g. due to network congestion) and not processed before the stream itself is destroyed. In that case, the `_writableState.ended` property could be set before the stream emits its `'close'` event, and never actually emits the `'finished'` event, confusing the end-of-stream implementation so that it wouldn’t call its callback. This can be fixed by watching for the end events themselves using the existing `'finish'` and `'end'` listeners rather than relying on the `.ended` properties of the `_...State` objects. These properties still need to be checked to know whether stream closure was premature – My understanding is that ideally, streams should not emit `'close'` before `'end'` and/or `'finished'`, so this might be another bug, but changing this would require modifying tests and almost certainly be a breaking change. Fixes: #24456 PR-URL: #24926 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
refack
pushed a commit
to refack/node
that referenced
this issue
Jan 14, 2019
Splitting out the http2 portion of the test has a few benfits: * We don't skip the rest of the tests if `node` is compiled without crypto. * We can find out if the http2 portion of the test is responsible for the timeouts reported in issue 24456. Refs: nodejs#24456 PR-URL: nodejs#24631 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
refack
pushed a commit
to refack/node
that referenced
this issue
Jan 14, 2019
HTTP/2 streams call `.end()` on themselves from their `.destroy()` method, which might be queued (e.g. due to network congestion) and not processed before the stream itself is destroyed. In that case, the `_writableState.ended` property could be set before the stream emits its `'close'` event, and never actually emits the `'finished'` event, confusing the end-of-stream implementation so that it wouldn’t call its callback. This can be fixed by watching for the end events themselves using the existing `'finish'` and `'end'` listeners rather than relying on the `.ended` properties of the `_...State` objects. These properties still need to be checked to know whether stream closure was premature – My understanding is that ideally, streams should not emit `'close'` before `'end'` and/or `'finished'`, so this might be another bug, but changing this would require modifying tests and almost certainly be a breaking change. Fixes: nodejs#24456 PR-URL: nodejs#24926 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
BethGriggs
pushed a commit
that referenced
this issue
Feb 11, 2019
Splitting out the http2 portion of the test has a few benfits: * We don't skip the rest of the tests if `node` is compiled without crypto. * We can find out if the http2 portion of the test is responsible for the timeouts reported in issue 24456. Refs: #24456 PR-URL: #24631 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
BethGriggs
pushed a commit
that referenced
this issue
Feb 12, 2019
HTTP/2 streams call `.end()` on themselves from their `.destroy()` method, which might be queued (e.g. due to network congestion) and not processed before the stream itself is destroyed. In that case, the `_writableState.ended` property could be set before the stream emits its `'close'` event, and never actually emits the `'finished'` event, confusing the end-of-stream implementation so that it wouldn’t call its callback. This can be fixed by watching for the end events themselves using the existing `'finish'` and `'end'` listeners rather than relying on the `.ended` properties of the `_...State` objects. These properties still need to be checked to know whether stream closure was premature – My understanding is that ideally, streams should not emit `'close'` before `'end'` and/or `'finished'`, so this might be another bug, but changing this would require modifying tests and almost certainly be a breaking change. Fixes: #24456 PR-URL: #24926 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
BethGriggs
pushed a commit
that referenced
this issue
Feb 20, 2019
HTTP/2 streams call `.end()` on themselves from their `.destroy()` method, which might be queued (e.g. due to network congestion) and not processed before the stream itself is destroyed. In that case, the `_writableState.ended` property could be set before the stream emits its `'close'` event, and never actually emits the `'finished'` event, confusing the end-of-stream implementation so that it wouldn’t call its callback. This can be fixed by watching for the end events themselves using the existing `'finish'` and `'end'` listeners rather than relying on the `.ended` properties of the `_...State` objects. These properties still need to be checked to know whether stream closure was premature – My understanding is that ideally, streams should not emit `'close'` before `'end'` and/or `'finished'`, so this might be another bug, but changing this would require modifying tests and almost certainly be a breaking change. Fixes: #24456 PR-URL: #24926 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
rvagg
pushed a commit
that referenced
this issue
Feb 28, 2019
Splitting out the http2 portion of the test has a few benfits: * We don't skip the rest of the tests if `node` is compiled without crypto. * We can find out if the http2 portion of the test is responsible for the timeouts reported in issue 24456. Refs: #24456 PR-URL: #24631 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
rvagg
pushed a commit
that referenced
this issue
Feb 28, 2019
HTTP/2 streams call `.end()` on themselves from their `.destroy()` method, which might be queued (e.g. due to network congestion) and not processed before the stream itself is destroyed. In that case, the `_writableState.ended` property could be set before the stream emits its `'close'` event, and never actually emits the `'finished'` event, confusing the end-of-stream implementation so that it wouldn’t call its callback. This can be fixed by watching for the end events themselves using the existing `'finish'` and `'end'` listeners rather than relying on the `.ended` properties of the `_...State` objects. These properties still need to be checked to know whether stream closure was premature – My understanding is that ideally, streams should not emit `'close'` before `'end'` and/or `'finished'`, so this might be another bug, but changing this would require modifying tests and almost certainly be a breaking change. Fixes: #24456 PR-URL: #24926 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
masterThe text was updated successfully, but these errors were encountered: