test: fix flaky test-runner-cli-concurrency.js #50108

cjihrig · 2023-10-10T01:24:01Z

This test was flaky on Windows when trying to clean up the tmp directory between tests. This commit updates the test to not delete anything between tests.

Fixes: #50101

nodejs-github-bot · 2023-10-10T01:24:06Z

Review requested:

@nodejs/test_runner

nodejs-github-bot · 2023-10-10T12:57:07Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54655/

nodejs-github-bot · 2023-10-10T14:44:09Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54666/

aduh95 · 2023-10-10T18:39:58Z

FWIW it's also flaky on non-Windows, e.g.: https://ci.nodejs.org/job/node-test-commit-linux-containered/39780/

nodejs-github-bot · 2023-10-10T19:04:02Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54674/

nodejs-github-bot · 2023-10-10T20:53:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54682/

StefanStojanovic · 2023-10-10T20:57:52Z

I've taken these changes and tested them locally. Although the drop in the number of failures is drastic I still saw a few errors like this:

not ok 31 parallel/test-runner-cli-concurrency
  ---
  duration_ms: 2415.70200
  severity: fail
  exitcode: 1
  stack: |-
    TAP version 13
    # Subtest: concurrency of one
    ok 1 - concurrency of one
      ---
      duration_ms: 1028.2008
      ...
    # Subtest: concurrency of two
    ok 2 - concurrency of two
      ---
      duration_ms: 1027.8476
      ...
    1..2
    # tests 2
    # suites 0
    # pass 2
    # fail 0
    # cancelled 0
    # skipped 0
    # todo 0
    # duration_ms 2071.5305
    Can't clean tmpdir: E:\work\node\test\.tmp.31
    Files blocking: [ '1', 'test-1.js', 'test-2.js' ]
    
    E:\work\node\test\common\tmpdir.js:69
        throw e;
        ^
    
    Error: EBUSY: resource busy or locked, rmdir '\\?\E:\work\node\test\.tmp.31\1'
        at rmdirSync (node:fs:1232:10)
        at _rmdirSync (node:internal/fs/rimraf:235:5)
        at rimrafSync (node:internal/fs/rimraf:193:7)
        at node:internal/fs/rimraf:253:9
        at Array.forEach (<anonymous>)
        at _rmdirSync (node:internal/fs/rimraf:250:7)
        at rimrafSync (node:internal/fs/rimraf:193:7)
        at Object.rmSync (node:fs:1281:10)
        at rmSync (E:\work\node\test\common\tmpdir.js:20:8)
        at onexit (E:\work\node\test\common\tmpdir.js:54:5) {
      errno: -4082,
      syscall: 'rmdir',
      code: 'EBUSY',
      path: '\\\\?\\E:\\work\\node\\test\\.tmp.31\\1'
    }
    
    Node.js v21.0.0-pre
  ...

To be honest, this is not a new failure as I saw it happening in the old implementation as well (the call stack is a bit different because of the code changes, but it's the same error). The other error (from #50101) where the second test would fail after the first one passed I was not able to reproduce in 10k+ runs.

This test was flaky on Windows when trying to clean up the tmp directory, probably because it relied on child processes timing out and being killed. This commit updates the test to check for debug output from the test runner. This should be adequate because the original change was effectively: let concurrency = getOptionValue('--test-concurrency') || true; The test runner now logs the value of the concurrency variable. Fixes: nodejs#50101

cjihrig · 2023-10-10T22:26:42Z

I have completely rewritten the test. The original change that this test was written for is effectively only this:

let concurrency = getOptionValue('--test-concurrency') || true;

So all we really care about is the value of concurrency. I have updated the test runner to use the NODE_DEBUG variable to log its settings, including that variable.

nodejs-github-bot · 2023-10-10T23:15:51Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54686/

nodejs-github-bot · 2023-10-11T11:45:46Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54698/

nodejs-github-bot · 2023-10-11T13:47:40Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54699/

nodejs-github-bot · 2023-10-11T18:04:15Z

CI: https://ci.nodejs.org/job/node-test-pull-request/54709/

nodejs-github-bot · 2023-10-12T01:37:02Z

Landed in 0d8faf2

This test was flaky on Windows when trying to clean up the tmp directory, probably because it relied on child processes timing out and being killed. This commit updates the test to check for debug output from the test runner. This should be adequate because the original change was effectively: let concurrency = getOptionValue('--test-concurrency') || true; The test runner now logs the value of the concurrency variable. Fixes: nodejs#50101 PR-URL: nodejs#50108 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>

This test was flaky on Windows when trying to clean up the tmp directory, probably because it relied on child processes timing out and being killed. This commit updates the test to check for debug output from the test runner. This should be adequate because the original change was effectively: let concurrency = getOptionValue('--test-concurrency') || true; The test runner now logs the value of the concurrency variable. Fixes: #50101 PR-URL: #50108 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>

This test was flaky on Windows when trying to clean up the tmp directory, probably because it relied on child processes timing out and being killed. This commit updates the test to check for debug output from the test runner. This should be adequate because the original change was effectively: let concurrency = getOptionValue('--test-concurrency') || true; The test runner now logs the value of the concurrency variable. Fixes: nodejs#50101 PR-URL: nodejs#50108 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>

This test was flaky on Windows when trying to clean up the tmp directory, probably because it relied on child processes timing out and being killed. This commit updates the test to check for debug output from the test runner. This should be adequate because the original change was effectively: let concurrency = getOptionValue('--test-concurrency') || true; The test runner now logs the value of the concurrency variable. Fixes: nodejs/node#50101 PR-URL: nodejs/node#50108 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>

nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to the tests. labels Oct 10, 2023

richardlau added the request-ci Add this label to start a Jenkins CI on a PR. label Oct 10, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Oct 10, 2023

cjihrig force-pushed the test-flake branch from e4a1b3d to cb1688a Compare October 10, 2023 14:43

aduh95 approved these changes Oct 10, 2023

View reviewed changes

GeoffreyBooth approved these changes Oct 10, 2023

View reviewed changes

lpinca approved these changes Oct 10, 2023

View reviewed changes

cjihrig force-pushed the test-flake branch from cb1688a to e8f7146 Compare October 10, 2023 22:23

github-actions bot mentioned this pull request Oct 11, 2023

CI Reliability 2023-10-11 nodejs/reliability#690

Open

30 tasks

anonrig approved these changes Oct 11, 2023

View reviewed changes

anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Oct 11, 2023

richardlau mentioned this pull request Oct 11, 2023

test: add EOVERFLOW as an allowed error #50128

Merged

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Oct 12, 2023

nodejs-github-bot merged commit 0d8faf2 into nodejs:main Oct 12, 2023
50 checks passed

cjihrig deleted the test-flake branch October 12, 2023 13:15

RafaelGSS mentioned this pull request Oct 15, 2023

2023-10-17, Version 21.0.0 (Current) #49870

Merged

richardlau mentioned this pull request Oct 17, 2023

fs: add flush option to appendFile() functions #50095

Merged

targos mentioned this pull request Nov 12, 2023

v20.10.0 release proposal #50682

Merged

targos mentioned this pull request Nov 28, 2023

v18.19.0 release proposal #50953

Merged

juanarbol mentioned this pull request Dec 6, 2023

2023-12-06, N|Solid v5.0.1 Iron Release nodesource/nsolid#41

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: fix flaky test-runner-cli-concurrency.js #50108

test: fix flaky test-runner-cli-concurrency.js #50108

cjihrig commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

aduh95 commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

StefanStojanovic commented Oct 10, 2023

cjihrig commented Oct 10, 2023 •

edited

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 11, 2023

nodejs-github-bot commented Oct 11, 2023

nodejs-github-bot commented Oct 11, 2023

nodejs-github-bot commented Oct 12, 2023

test: fix flaky test-runner-cli-concurrency.js #50108

test: fix flaky test-runner-cli-concurrency.js #50108

Conversation

cjihrig commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

aduh95 commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 10, 2023

StefanStojanovic commented Oct 10, 2023

cjihrig commented Oct 10, 2023 • edited

nodejs-github-bot commented Oct 10, 2023

nodejs-github-bot commented Oct 11, 2023

nodejs-github-bot commented Oct 11, 2023

nodejs-github-bot commented Oct 11, 2023

nodejs-github-bot commented Oct 12, 2023

cjihrig commented Oct 10, 2023 •

edited