benchmark: test refactoring #31396

BridgeAR · 2020-01-17T08:03:41Z

I went ahead and refactored all our benchmark tests. The benchmark suite itself now has a test functionality built-in (using test). It is also possible to run all benchmark suites by using all. The tests are now run independent from benchmark changes and should always keep on working.

The fixtures directory is now skipped and in case there's a typo for the benchmark, the error message is now more informative.

I removed warnings from benchmark files in case they occurred.

Some tests require special options as a minimum. Those may now be passed through using the test option. These values will be used in case the benchmark test is run.

It is now possible to pass through individual configurations instead of having to use an array.

Fixes: #31083

I did not yet find time to clean up the commits.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

benchmark/_test-double-benchmarker.js

BridgeAR · 2020-01-17T10:09:21Z

benchmark/_test-double-benchmarker.js

      run();
    } else {
      console.log(JSON.stringify({ throughput }));
      if (client) {
        client.destroy();
+        process.exit(0);


@nodejs/http2 @ronag I had to end the process hard in case this runs the http2 benchmarks. Maybe someone could have a look what part stays alive.

BridgeAR · 2020-01-20T10:25:59Z

This is ready for review. I just cleaned up the commits.

sam-github

Looks reasonable to me.

BridgeAR · 2020-01-20T17:56:15Z

CI: https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/11672/ ✔️

BridgeAR · 2020-01-20T18:23:06Z

This does reduce the test runtime for almost all benchmarks. Mostly by a few hundred milliseconds. The runtime for test-benchmark-async-hooks is reduced more significantly from ~6 to ~1 seconds and test-benchmark-module from ~3 to ~1 seconds.

BridgeAR · 2020-01-22T14:20:00Z

This could use another review @nodejs/benchmarking

Trott · 2020-01-27T06:36:26Z

Needs a rebase.

Trott

Rubber-stamp LGTM. The concept is sound.

BridgeAR · 2020-02-06T15:22:11Z

CI https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/12025/

BridgeAR · 2020-02-08T22:41:52Z

CI https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/12097/

This is the basis to refactor the helper to use modern class language features such as private fields. It also refactors the exports to use module.exports. That way it's immediately clear what parts are exported.

This reverts commit b70741e. Refs: #31396 PR-URL: #31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

mscdex · 2020-02-11T12:21:47Z

Also something that I just now noticed is that this PR changed the formatting of the configuration string (when writing the CSV output) such that each parameter is preceded by a space whether it's needed or not, which breaks tooling (something I'm currently working on) that monitors changes in benchmark results by comparing results for a particular configuration.

Just something to keep in mind if these changes get merged again later on.

EDIT: FWIW I'm now incorporating extra safeguards to prevent this sort of thing in my tooling in the future...

This is the basis to refactor the helper to use modern class language features such as private fields. It also refactors the exports to use module.exports. That way it's immediately clear what parts are exported. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This adds a new `test` option. Using it automatically uses a single minimal option matrix to verify the benchmark works as expected. Using the new `all` option makes sure all test suites are run. On top of that the benchmarks will from now on report properly what category might have a typo, if any. The http duration was also refactored to use a option instead of relying on a configuration setting. The fixture folder is ignored as test suite from now on. PR-URL: #31396 Fixes: #31083 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

It was necessary to have fallbacks to run the original tests. This is obsolete with the new test mode. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This reverts commit 357230f. Refs: #31396 PR-URL: #31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This reverts commit 78aa348. Refs: #31396 PR-URL: #31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This reverts commit dac5795. Refs: #31396 PR-URL: #31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This reverts commit b70741e. Refs: #31396 PR-URL: #31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

codebytere · 2020-03-15T06:15:13Z

@BridgeAR could you please manually backport this to v12.x?

This is the basis to refactor the helper to use modern class language features such as private fields. It also refactors the exports to use module.exports. That way it's immediately clear what parts are exported. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

It was necessary to have fallbacks to run the original tests. This is obsolete with the new test mode. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This is the basis to refactor the helper to use modern class language features such as private fields. It also refactors the exports to use module.exports. That way it's immediately clear what parts are exported. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

It was necessary to have fallbacks to run the original tests. This is obsolete with the new test mode. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This is the basis to refactor the helper to use modern class language features such as private fields. It also refactors the exports to use module.exports. That way it's immediately clear what parts are exported. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

It was necessary to have fallbacks to run the original tests. This is obsolete with the new test mode. PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

This reverts commit 357230f. Refs: nodejs#31396 PR-URL: nodejs#31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This reverts commit b70741e. Refs: nodejs#31396 PR-URL: nodejs#31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This reverts commit 357230f. Refs: #31396 PR-URL: #31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This reverts commit b70741e. Refs: #31396 PR-URL: #31722 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Vladimir de Turckheim <vlad2t@hotmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

BridgeAR requested a review from Trott January 17, 2020 08:03

mscdex reviewed Jan 17, 2020

View reviewed changes

benchmark/_test-double-benchmarker.js Show resolved Hide resolved

BridgeAR commented Jan 17, 2020

View reviewed changes

BridgeAR force-pushed the 2020-01-17-benchmark-test-refactoring branch 2 times, most recently from 1cb7b50 to 27eee27 Compare January 20, 2020 10:22

BridgeAR marked this pull request as ready for review January 20, 2020 10:25

sam-github approved these changes Jan 20, 2020

View reviewed changes

BridgeAR added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jan 20, 2020

BridgeAR force-pushed the 2020-01-17-benchmark-test-refactoring branch from 27eee27 to 3fd40c6 Compare January 25, 2020 09:47

Trott approved these changes Jan 27, 2020

View reviewed changes

BridgeAR force-pushed the 2020-01-17-benchmark-test-refactoring branch from 3fd40c6 to 6ea89f5 Compare February 5, 2020 13:20

BridgeAR force-pushed the 2020-01-17-benchmark-test-refactoring branch 2 times, most recently from 9bdb710 to f7926f3 Compare February 8, 2020 22:40

benchmark: refactor helper into a class

1e17257

This is the basis to refactor the helper to use modern class language features such as private fields. It also refactors the exports to use module.exports. That way it's immediately clear what parts are exported.

BridgeAR mentioned this pull request Feb 12, 2020

benchmark: reland test refactoring #31755

Closed

4 tasks

codebytere pushed a commit that referenced this pull request Feb 17, 2020

test: refactor all benchmark tests to use the new test option

4878c7a

PR-URL: #31396 Reviewed-By: Sam Roberts <vieuxtech@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

codebytere mentioned this pull request Feb 17, 2020

v13.9.0 proposal #31837

Merged

codebytere added backport-requested-v12.x and removed backport-requested-v12.x labels Mar 15, 2020

codebytere mentioned this pull request Mar 17, 2020

v12.16.2 proposal #32313

Merged

targos removed author ready PRs that have at least one approval, no pending requests for changes, and a CI started. backport-blocked-v12.x labels Apr 25, 2020

targos added the dont-land-on-v12.x label Apr 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

benchmark: test refactoring #31396

benchmark: test refactoring #31396

BridgeAR commented Jan 17, 2020 •

edited

BridgeAR Jan 17, 2020 •

edited

BridgeAR commented Jan 20, 2020

sam-github left a comment

BridgeAR commented Jan 20, 2020 •

edited

BridgeAR commented Jan 20, 2020

BridgeAR commented Jan 22, 2020

Trott commented Jan 27, 2020

Trott left a comment •

edited

BridgeAR commented Feb 6, 2020

BridgeAR commented Feb 8, 2020

mscdex commented Feb 11, 2020 •

edited

codebytere commented Mar 15, 2020

benchmark: test refactoring #31396

benchmark: test refactoring #31396

Conversation

BridgeAR commented Jan 17, 2020 • edited

Checklist

BridgeAR Jan 17, 2020 • edited

Choose a reason for hiding this comment

BridgeAR commented Jan 20, 2020

sam-github left a comment

Choose a reason for hiding this comment

BridgeAR commented Jan 20, 2020 • edited

BridgeAR commented Jan 20, 2020

BridgeAR commented Jan 22, 2020

Trott commented Jan 27, 2020

Trott left a comment • edited

Choose a reason for hiding this comment

BridgeAR commented Feb 6, 2020

BridgeAR commented Feb 8, 2020

mscdex commented Feb 11, 2020 • edited

codebytere commented Mar 15, 2020

BridgeAR commented Jan 17, 2020 •

edited

BridgeAR Jan 17, 2020 •

edited

BridgeAR commented Jan 20, 2020 •

edited

Trott left a comment •

edited

mscdex commented Feb 11, 2020 •

edited