Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: add benchmark for coverage #36972

Closed
wants to merge 1 commit into from
Closed

Conversation

@bcoe
Copy link
Member

@bcoe bcoe commented Jan 17, 2021

Working on #36956, I became a bit worried that as we extend code coverage to support more types of branching, performance might start to become unacceptable.

I created this benchmark to experiment:

Node 14.15.4, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -18.94 % ±0.77% ±1.02% ±1.30%

Node 15.6.0, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -17.95 % ±1.08% ±1.43% ±1.83%

Coverage with #36956, vs., no coverage:

process/coverage.js n=100000 *** -19.37 % ±1.36% ±1.79% ±2.30%

To Summarize

In the current release of 14/15, we see that running Node with coverage decreases performance by about 18%.

With the introduction of optional chaining in this benchmark, we see a 1 - 2% additional hit to performance.

Other interesting facts

  • I benchmarked coverage.js using Istanbul, vs., NODE_V8_COVERAGE, it was 37.46% slower than no coverage.
  • I benchmarked #36956 with no coverage, vs., 15.6.0 with no coverage; there did not appear to be a significant difference in performance (0.37 % ±1.14% ±1.50% ±1.92%).

Conclusions

  • My gut is that it's worth taking the 1 - 2% performance hit to support a new type of branching in our coverage reporting.
  • As new constructs like this are added to JavaScript, we need to be mindful that the cost of branch coverage will continue to increase.
  • I was excited to see that v8's coverage offers a significant performance benefit over pre-instrumenting the JavaScript code.

CC: @schuay

@bcoe
Copy link
Member Author

@bcoe bcoe commented Jan 17, 2021

@joyeecheung I was thinking this benchmark would be a really good use of v8.startCoverage, because it would let us turn on coverage collection programmatically, at which point it would be easier to use this benchmark to detect regressions in coverage performance in the future.

benchmark/process/coverage.js Outdated Show resolved Hide resolved
Trott
Trott approved these changes Jan 17, 2021
@Trott
Copy link
Member

@Trott Trott commented Jan 17, 2021

Subsystem in commit message should be benchmark rather than test.

@bcoe bcoe force-pushed the benchmark-coverage branch from d206807 to 1d8a88b Jan 18, 2021
@bcoe bcoe changed the title test: add benchmark for coverage benchmark: add benchmark for coverage Jan 18, 2021
@bcoe bcoe force-pushed the benchmark-coverage branch 2 times, most recently from c5507d4 to b5eadb0 Jan 18, 2021
@bcoe bcoe force-pushed the benchmark-coverage branch from b5eadb0 to efea2d6 Jan 18, 2021
@bcoe
Copy link
Member Author

@bcoe bcoe commented Jan 18, 2021

@Trott refactored so that the benchmark is easier to use, it now spawns a subprocess with coverage enabled (_so no need to create a separate bin) 👍

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 18, 2021

benchmark/process/coverage.js Outdated Show resolved Hide resolved
@bcoe bcoe force-pushed the benchmark-coverage branch from efea2d6 to ed4e294 Jan 19, 2021
@nodejs-github-bot

This comment has been hidden.

@bcoe bcoe force-pushed the benchmark-coverage branch from ed4e294 to 682f4e9 Jan 22, 2021
@nodejs-github-bot

This comment has been hidden.

@bcoe bcoe force-pushed the benchmark-coverage branch from 682f4e9 to ae141a5 Jan 22, 2021
@nodejs-github-bot

This comment has been hidden.

@bcoe bcoe force-pushed the benchmark-coverage branch from ae141a5 to 2823b83 Jan 22, 2021
@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 22, 2021

benchmark/process/coverage.js Outdated Show resolved Hide resolved
@bcoe bcoe force-pushed the benchmark-coverage branch from 2823b83 to 56cf8b5 Jan 22, 2021
@bcoe bcoe requested a review from richardlau Jan 22, 2021
@nodejs-github-bot

This comment has been hidden.

@nodejs-github-bot

This comment has been hidden.

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 25, 2021

bcoe added a commit that referenced this issue Jan 27, 2021
PR-URL: #36972
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
@bcoe
Copy link
Member Author

@bcoe bcoe commented Jan 27, 2021

Landed in 8c9dc4e

@bcoe bcoe closed this Jan 27, 2021
targos added a commit that referenced this issue Feb 2, 2021
PR-URL: #36972
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
@targos targos mentioned this pull request Feb 2, 2021
targos added a commit that referenced this issue May 1, 2021
PR-URL: #36972
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
@danielleadams danielleadams mentioned this pull request May 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants