Introducing Atom benchmarks #12984

Merged
merged 13 commits into from Oct 14, 2016

Conversation

Projects
None yet
6 participants
@as-cii
Member

as-cii commented Oct 14, 2016

For a long time in Atom we have been wanting to measure how changes in the codebase can affect performance. Until now we have been relying solely on Chrome's wonderful profiler to explain performance optimizations or regressions when opening pull requests or issues; however, even though DevTools are a great companion when it comes to understanding what's slow and why, we felt the need for a sharper tool that could allow everyone (not just the person opening the pull request) to:

  • Run benchmarks on their machine.
  • Easily compare and contrast two runs of the same exact benchmark on two different versions of Atom.
  • Store benchmarks and keep track of them.

As part of our efforts to make performance in Atom better and more transparent, we are introducing Window: Run Benchmarks and atom --benchmark.

Disclaimer: this feature is meant to be used exclusively for atom/atom and, although it could work, we recommend against using it for running benchmarks on packages.

A quick tour

When opening the command palette, you can now invoke the Window: Run Benchmarks command. This will open a new window and run all the benchmarks found in the benchmarks/ directory. As soon as a benchmark completes, a plot of its data will be displayed on screen:

screen shot 2016-10-14 at 12 08 03

Alternatively, you can invoke atom --benchmark with one or more benchmark paths and get textual output instead:

Opening and rendering a TextEditor in large file mode:

0   199.98000000000002
1   18.894999999999982
2   11.099999999999909
3   9.565000000000055
4   8.554999999999836
5   9.670000000000073

You might have noticed that the above output is formatted as CSV (i.e. there is a tab between the columns). This allows to import data into an external tool like Numbers.app pretty easily to e.g. perform comparisons between two or more versions of Atom:

screen shot 2016-10-14 at 12 30 00

Or to switch to a different representation of the same data:

screen shot 2016-10-14 at 12 47 08

"Single-value" benchmarks are also supported, and they will not show any graph when run through Window: Run benchmarks:

screen shot 2016-10-14 at 12 52 27

Implementation details

Data is plotted via Chart.js and each benchmark is a javascript file (named *.bench.js) that exports a function (synchronous or asynchronous) with the following signature:

// `test` is a boolean indicating whether the user wants to run benchmarks in a
// lightweight mode.
export default function ({test}) {
  const t0 = window.performance.now()
  fun1()
  const t1 = window.performance.now()
  fun1()
  const t2 = window.performance.now()
  fun2()
  const t3 = window.performance.now()

  // This will plot one graph for `fun-1` and one "single-value" benchmark for
  // `fun-2`. Note how the latter doesn't need to supply an `x` value in the
  // object.
  return [
    {name: 'fun-1', x: 1, duration: t1 - t0},
    {name: 'fun-1', x: 2, duration: t2 - t1},
    {name: 'fun-2', duration: t3 - t2}
  ]
}

It's also interesting to note that, for the time being, we will not run benchmarks on dedicated hardware. However, we will make sure that they won't become stale or out-of-date by running a more lightweight version of them (see the test parameter above) on CI via atom --benchmark-test (which, with this pull request, will be executed as part of script/test).

/cc: @atom/core

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Oct 14, 2016

Contributor

Amazing.

Contributor

nathansobo commented Oct 14, 2016

Amazing.

@as-cii as-cii merged commit c6db4df into master Oct 14, 2016

5 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/appveyor/branch AppVeyor build succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@as-cii as-cii deleted the mb-benchmark-mode branch Oct 14, 2016

@modulexcite

This comment has been minimized.

Show comment
Hide comment
@modulexcite

modulexcite Jan 12, 2017

Sounds great. Thanks for an excellent addition to Atom.
Where exactly is the benchmarks folder stored? I can't seem to locate it on a Windows 7 machine with Atom version 1.13.0. When I open it from the command palette, nothing is displayed but when I try to start Atom with atom --benchmark it asks me to specify at least one benchmark path which I have no clue what that should be.

Sounds great. Thanks for an excellent addition to Atom.
Where exactly is the benchmarks folder stored? I can't seem to locate it on a Windows 7 machine with Atom version 1.13.0. When I open it from the command palette, nothing is displayed but when I try to start Atom with atom --benchmark it asks me to specify at least one benchmark path which I have no clue what that should be.

@TJProgrammer

This comment has been minimized.

Show comment
Hide comment
@TJProgrammer

TJProgrammer Jan 12, 2017

I caught an error.
When I click in "Window: Run benchemarks", appears a new window and a red error like this
module.js:455 Uncaught (in promise) Error: Cannot find module '/home/.../.../.../benchmarks'.
Seems that the benchmark goes to find a benchmark folder inside the actual project folder instead of atom's folder.

Fedora 25
Atom 1.13

I caught an error.
When I click in "Window: Run benchemarks", appears a new window and a red error like this
module.js:455 Uncaught (in promise) Error: Cannot find module '/home/.../.../.../benchmarks'.
Seems that the benchmark goes to find a benchmark folder inside the actual project folder instead of atom's folder.

Fedora 25
Atom 1.13

@saranshbansal

This comment has been minimized.

Show comment
Hide comment
@saranshbansal

saranshbansal Jan 18, 2017

It would be great if you could add a sample benchmark file. Where do I get these files?
image

It would be great if you could add a sample benchmark file. Where do I get these files?
image

@as-cii

This comment has been minimized.

Show comment
Hide comment
@as-cii

as-cii Jan 19, 2017

Member

You can look at an example benchmark here: if you clone this repository and run atom --benchmark benchmarks/text-editor-large-file-construction.bench.js it should work as expected.

Member

as-cii commented Jan 19, 2017

You can look at an example benchmark here: if you clone this repository and run atom --benchmark benchmarks/text-editor-large-file-construction.bench.js it should work as expected.

@liuderchi

This comment has been minimized.

Show comment
Hide comment
@liuderchi

liuderchi Mar 9, 2017

Contributor

Found that Run Benchmarks menu item is added only to menus/darwin.cson

Will it be added to menu for both Win32 and Linux platform?
If it's yes, I can help create issue/PR by the way to enhance it.

Contributor

liuderchi commented Mar 9, 2017

Found that Run Benchmarks menu item is added only to menus/darwin.cson

Will it be added to menu for both Win32 and Linux platform?
If it's yes, I can help create issue/PR by the way to enhance it.

@nathansobo

This comment has been minimized.

Show comment
Hide comment
@nathansobo

nathansobo Mar 10, 2017

Contributor

I'm open to it being added elsewhere. Any objections @as-cii?

Contributor

nathansobo commented Mar 10, 2017

I'm open to it being added elsewhere. Any objections @as-cii?

@as-cii

This comment has been minimized.

Show comment
Hide comment
@as-cii

as-cii Mar 10, 2017

Member

Yeah, that sounds good to me too. 👍

Member

as-cii commented Mar 10, 2017

Yeah, that sounds good to me too. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment