Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flake: sequential/test-inspector-stop-profile-after-done #16772

Closed
refack opened this issue Nov 5, 2017 · 22 comments
Closed

flake: sequential/test-inspector-stop-profile-after-done #16772

refack opened this issue Nov 5, 2017 · 22 comments
Labels
arm Issues and PRs related to the ARM platform. flaky-test Issues and PRs related to the tests with unstable failures on the CI. inspector Issues and PRs related to the V8 inspector protocol test Issues and PRs related to the tests. windows Issues and PRs related to the Windows platform.

Comments

@refack
Copy link
Contributor

refack commented Nov 5, 2017 •
edited
Loading

  • Version: master
  • Platform: Windows
  • Subsystem: inspector,test

https://ci.nodejs.org/job/node-test-binary-windows/12525/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=3/console

not ok 506 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 124.922
  severity: fail
  stack: |-
    timeout
  ...

Refs: #14507
@refack refack added flaky-test Issues and PRs related to the tests with unstable failures on the CI. inspector Issues and PRs related to the V8 inspector protocol test Issues and PRs related to the tests. labels Nov 5, 2017
@refack
Copy link
Contributor Author

refack commented Nov 5, 2017

/cc @nodejs/v8-inspector

@Trott
Copy link
Member

Trott commented Nov 6, 2017

Again today: https://ci.nodejs.org/job/node-test-binary-windows/12549/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=3/console

not ok 506 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 128.785
  severity: fail
  stack: |-
    timeout
``

@Trott
Copy link
Member

Trott commented Nov 8, 2017

Now on Raspberry Pi too:

https://ci.nodejs.org/job/node-test-binary-arm/RUN_SUBSET=6,label=pi1-raspbian-wheezy/11617/console:

not ok 278 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 360.77
  severity: fail
  stack: |-
    timeout

Looks like this resulted in the device failing to build multiple times subsequently. @rvagg indicated it was due to a zombie process. So maybe it came about due to this test failure?

@Trott
Copy link
Member

Trott commented Nov 8, 2017

Another windows one. Looks like it only ever happens on win2016 vs2017 hosts in CI.

https://ci.nodejs.org/job/node-test-binary-windows/12634/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=3/console

not ok 507 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 123.544
  severity: fail
  stack: |-
    timeout

@refack
Copy link
Contributor Author

refack commented Nov 8, 2017

In the reports I see 3/3 VS2017 fails, so this might be related to #15558

@refack
Copy link
Contributor Author

refack commented Nov 8, 2017

See if we can repro (single test x 100 repeats)
win2012r2-mp-vs2015 - https://ci.nodejs.org/job/node-stress-single-test/1548/
win2012r2-mp-vs2017 - https://ci.nodejs.org/job/node-stress-single-test/1549/

@Trott
Copy link
Member

Trott commented Nov 17, 2017

Today on Raspberry Pi:

https://ci.nodejs.org/job/node-test-binary-arm/11870/RUN_SUBSET=4,label=pi1-raspbian-wheezy/console

20:26:43 not ok 282 sequential/test-inspector-stop-profile-after-done
20:26:43   ---
20:26:43   duration_ms: 360.111
20:26:43   severity: fail
20:26:43   stack: |-
20:26:43     timeout

@mmarchini
Copy link
Contributor

Today on Raspbery Pi again:

https://ci.nodejs.org/job/node-test-binary-arm/11987/RUN_SUBSET=3,label=pi1-raspbian-wheezy/console

11:57:59 not ok 283 sequential/test-inspector-stop-profile-after-done
11:57:59   ---
11:57:59   duration_ms: 360.66
11:57:59   severity: fail
11:57:59   stack: |-
11:57:59     timeout

@Trott
Copy link
Member

Trott commented Dec 2, 2017

WIndows again but this time an assertion failure rather than a timeout:

https://ci.nodejs.org/job/node-test-binary-windows/13307/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=2/console

not ok 518 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 0.680
  severity: fail
  stack: |-
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] Debugger listening on ws://127.0.0.1:61080/4b6bfeec-65aa-4129-b2c3-2cd3c829c1dc
    [err] For help see https://nodejs.org/en/docs/inspector
    [err] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [out] {}
    [out] 
    [err] Debugger attached.
    [err] Waiting for the debugger to disconnect...
    [err] 
    { AssertionError [ERR_ASSERTION]: 0 strictEqual 3221225477
        at runTests (c:\workspace\node-test-binary-windows\test\sequential\test-inspector-stop-profile-after-done.js:26:10)
        at <anonymous>
        at process._tickCallback (internal/process/next_tick.js:165:7)
      generatedMessage: true,
      name: 'AssertionError [ERR_ASSERTION]',
      code: 'ERR_ASSERTION',
      actual: 0,
      expected: 3221225477,
      operator: 'strictEqual' }
    1

@Trott
Copy link
Member

Trott commented Dec 23, 2017

Back to a timeout on Windows:

https://ci.nodejs.org/job/node-test-binary-windows/13846/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=1/console

not ok 524 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 127.567
  severity: fail
  stack: |-
    timeout
  ...

@Trott
Copy link
Member

Trott commented Dec 23, 2017 •
edited
Loading

It does appear that this test takes an unusually large amount of time to run:

ok 524 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 35.635
  ...

In comparison, the test takes less than a second to run on (for example) our CentOS 7 test machine:

ok 2029 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 0.620

That's quite a difference....

@Trott
Copy link
Member

Trott commented Dec 23, 2017

If we could get the output of the test when it times out, that would let us know how far along it got. (One reason I'm not super-excited about the tendency to remove console.log()/console.error() statements from tests.)

@refack Any chance you can run the test locally and get an idea where it seems to do al its waiting?

@Trott
Copy link
Member

Trott commented Dec 23, 2017

@nodejs/platform-windows

@Trott
Copy link
Member

Trott commented Dec 28, 2017

Timeout again today.

https://ci.nodejs.org/job/node-test-binary-windows/13917/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=2/console

not ok 525 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 127.467
  severity: fail
  stack: |-
    timeout
  ...

@Trott
Copy link
Member

Trott commented Dec 28, 2017

@jBarz @nodejs/testing

@Trott
Copy link
Member

Trott commented Jan 12, 2018

Back on Raspberry Pi 1 devices:

https://ci.nodejs.org/job/node-test-binary-arm/12846/RUN_SUBSET=4,label=pi1-raspbian-wheezy/console

03:41:04 not ok 282 sequential/test-inspector-stop-profile-after-done
03:41:04   ---
03:41:04   duration_ms: 360.84
03:41:04   severity: fail
03:41:04   stack: |-
03:41:04     timeout

@Trott
Copy link
Member

Trott commented Jan 12, 2018

Doesn't take much time when successful, so some sort of race condition or something must be happening. Here's a successful run on a Pi 2:

03:21:29 ok 282 sequential/test-inspector-stop-profile-after-done
03:21:29   ---
03:21:29   duration_ms: 2.837

@Trott
Copy link
Member

Trott commented Jan 12, 2018 •
edited
Loading

Replicated on macOS after 1700+ runs, but this time we have some output so we might have an idea of how/where it's getting stuck. Failure:

=== release test-inspector-stop-profile-after-done ===                     
Path: sequential/test-inspector-stop-profile-after-done
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:49485/23586fe0-dfa1-453b-8d7e-6acffa7c211b
[err] For help see https://nodejs.org/en/docs/inspector
[err] 
[err] Debugger attached.
[err] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[err] Waiting for the debugger to disconnect...
[err]
Command: out/Release/node /Users/trott/io.js/test/sequential/test-inspector-stop-profile-after-done.js
--- TIMEOUT ---

For comparison, here's success:

[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:51256/d8a1ee2c-1189-49f1-8b3b-127177e8693f
[err] For help see https://nodejs.org/en/docs/inspector
[err] 
[err] Debugger attached.
[err] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[out] {}
[out] 
[err] Waiting for the debugger to disconnect...
[err]

Uhhh....no real difference in output....so I guess it's the exit/cleanup code?

@Trott Trott mentioned this issue Jan 12, 2018
Closed
3 tasks
@Trott
Copy link
Member

Trott commented Jan 16, 2018

Another one today...

https://ci.nodejs.org/job/node-test-binary-arm/12900/RUN_SUBSET=3,label=pi1-raspbian-wheezy/console

03:32:54 not ok 282 sequential/test-inspector-stop-profile-after-done
03:32:54   ---
03:32:54   duration_ms: 360.87
03:32:54   severity: fail
03:32:54   stack: |-
03:32:54     timeout

@BridgeAR BridgeAR added windows Issues and PRs related to the Windows platform. arm Issues and PRs related to the ARM platform. labels Jan 19, 2018
@Trott
Copy link
Member

Trott commented Jan 19, 2018

And today, back on Windows...

https://ci.nodejs.org/job/node-test-binary-windows/14329/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=1/console

not ok 515 sequential/test-inspector-stop-profile-after-done
  ---
  duration_ms: 122.356
  severity: fail
  stack: |-
    timeout

MylesBorins added a commit that referenced this issue Feb 7, 2018
@MylesBorins
test: mark test-inspector-stop-profile-after-done flaky
076ca9f 
This test is consistently failing and making CI red.

PR-URL: #18491
Refs: #16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
MylesBorins added a commit that referenced this issue Feb 11, 2018
@MylesBorins
test: mark test-inspector-stop-profile-after-done flaky
d97376f 
This test is consistently failing and making CI red.

PR-URL: #18491
Refs: #16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
MylesBorins added a commit that referenced this issue Feb 12, 2018
@MylesBorins
test: mark test-inspector-stop-profile-after-done flaky
69271a6 
This test is consistently failing and making CI red.

PR-URL: #18491
Refs: #16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
MylesBorins added a commit that referenced this issue Feb 13, 2018
@MylesBorins
test: mark test-inspector-stop-profile-after-done flaky
694828d 
This test is consistently failing and making CI red.

PR-URL: #18491
Refs: #16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
MylesBorins added a commit that referenced this issue Feb 20, 2018
@MylesBorins
test: mark test-inspector-stop-profile-after-done flaky
3d8db8b 
This test is consistently failing and making CI red.

PR-URL: #18491
Refs: #16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
MylesBorins added a commit that referenced this issue Feb 21, 2018
@MylesBorins
test: mark test-inspector-stop-profile-after-done flaky
ee2075f 
This test is consistently failing and making CI red.

PR-URL: #18491
Refs: #16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
MylesBorins added a commit that referenced this issue Feb 21, 2018
@MylesBorins
test: mark test-inspector-stop-profile-after-done flaky
da0d776 
This test is consistently failing and making CI red.

PR-URL: #18491
Refs: #16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Trott added a commit to Trott/io.js that referenced this issue Mar 2, 2018
@Trott
wip: fix flaky inspector-stop-profile-after-done
7e6ce28 
In test/sequential/test-inspector-stop-profile-after-done.js, if
`Profile.start` is received after the debugger is waiting to disconnect,
then the process will not terminate.

Fixes: nodejs#16772
@Trott Trott closed this as completed in 81d73fe Mar 2, 2018
addaleax pushed a commit to addaleax/node that referenced this issue Mar 5, 2018
@Trott @addaleax
test: fix flaky inspector-stop-profile-after-done
cc90bbd 
Use common.platformTimeout() to give longer durations to Raspberry Pi
devices to make test more reliable.

PR-URL: nodejs#18126
Fixes: nodejs#16772
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
MayaLekova pushed a commit to MayaLekova/node that referenced this issue May 8, 2018
@MylesBorins @MayaLekova
test: mark test-inspector-stop-profile-after-done flaky
ce23fb1 
This test is consistently failing and making CI red.

PR-URL: nodejs#18491
Refs: nodejs#16772
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
MayaLekova pushed a commit to MayaLekova/node that referenced this issue May 8, 2018
@Trott @MayaLekova
test: fix flaky inspector-stop-profile-after-done
96559c9 
Use common.platformTimeout() to give longer durations to Raspberry Pi
devices to make test more reliable.

PR-URL: nodejs#18126
Fixes: nodejs#16772
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
MylesBorins pushed a commit that referenced this issue Aug 17, 2018
@Trott @MylesBorins
test: fix flaky inspector-stop-profile-after-done
7d0f02e 
Use common.platformTimeout() to give longer durations to Raspberry Pi
devices to make test more reliable.

PR-URL: #18126
Fixes: #16772
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arm Issues and PRs related to the ARM platform. flaky-test Issues and PRs related to the tests with unstable failures on the CI. inspector Issues and PRs related to the V8 inspector protocol test Issues and PRs related to the tests. windows Issues and PRs related to the Windows platform.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: fix flaky inspector-stop-profile-after-done Trott/io.js
5 participants
@refack @MylesBorins @Trott @mmarchini @BridgeAR