New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web Tooling Benchmark #138

Closed
bmeurer opened this Issue Aug 29, 2017 · 10 comments

Comments

Projects
None yet
5 participants
@bmeurer
Member

bmeurer commented Aug 29, 2017

As discussed in the last benchmarking WG meeting yesterday, I'd like to propose a new benchmark suite that covers common tools that are used daily by developers to build web pages, a so-called Web Tooling Benchmark. This will include at least tools like:

As far as I can tell this kind of use case is not yet covered and well-represented in the Main Use Case Categories.

The idea for this benchmark suite is to have it running both in Node, but also in the browser, to be able to use this as a general measure to advance JavaScript engine performance. It should ideally mock out all I/O and focus on the core execution logic inside those tools.

I expect work on this to start end of Q3 / beginning of Q4. Help from the community in both design and coding would be highly appreciated.

@donaldpipowitch

This comment has been minimized.

Show comment
Hide comment
@donaldpipowitch

donaldpipowitch Aug 29, 2017

This sounds awesome. I hope that the tools are not only tested in isolation, but also in tandem. Many performance problems occur, if you combine several tools with each other. E.g. at my current company we use TypeScript together with Babel (and maybe one or two other loaders) with WebPack. Not just TypeScript or just Babel. Our linting step currently also consists of three tools: prettier, TSLint and ESLint. (Mainly because TS support in ESLint is still new, so we use both linters.)

All in all that is an awesome project. 👍

I wonder how this benchmark will look like? Do you will use dedicated machines for test runs? Can tooling authors add their own tools via Pull Requests? Sorry, if this is already to detailed 😆

donaldpipowitch commented Aug 29, 2017

This sounds awesome. I hope that the tools are not only tested in isolation, but also in tandem. Many performance problems occur, if you combine several tools with each other. E.g. at my current company we use TypeScript together with Babel (and maybe one or two other loaders) with WebPack. Not just TypeScript or just Babel. Our linting step currently also consists of three tools: prettier, TSLint and ESLint. (Mainly because TS support in ESLint is still new, so we use both linters.)

All in all that is an awesome project. 👍

I wonder how this benchmark will look like? Do you will use dedicated machines for test runs? Can tooling authors add their own tools via Pull Requests? Sorry, if this is already to detailed 😆

@bmeurer

This comment has been minimized.

Show comment
Hide comment
@bmeurer

bmeurer Aug 29, 2017

Member

The idea is to start small and eventually include more tools. Pull requests are of course welcome.

I'd prefer to have the benchmark suite runnable standalone (i.e. independent of a concrete machine setup). Similar to let's say the Octane benchmark, which you could easily run on any machine via node or d8, but also in any browser.

Member

bmeurer commented Aug 29, 2017

The idea is to start small and eventually include more tools. Pull requests are of course welcome.

I'd prefer to have the benchmark suite runnable standalone (i.e. independent of a concrete machine setup). Similar to let's say the Octane benchmark, which you could easily run on any machine via node or d8, but also in any browser.

@hashseed

This comment has been minimized.

Show comment
Hide comment
@hashseed

hashseed Aug 29, 2017

Member

I agree. Being able to run it anywhere, including the browser, would be a huge plus.

Member

hashseed commented Aug 29, 2017

I agree. Being able to run it anywhere, including the browser, would be a huge plus.

@robpalme

This comment has been minimized.

Show comment
Hide comment
@robpalme

robpalme Aug 29, 2017

Please ensure this includes stress testing of the source-map package. (Transitive) sourcemap generation is hugely expensive today and is heavy on memory allocations and usage meaning it stresses GC.

robpalme commented Aug 29, 2017

Please ensure this includes stress testing of the source-map package. (Transitive) sourcemap generation is hugely expensive today and is heavy on memory allocations and usage meaning it stresses GC.

@mhdawson

This comment has been minimized.

Show comment
Hide comment
@mhdawson

mhdawson Aug 29, 2017

Member

As discussed in the bench-marking meeting yesterday it would be great to add coverage for that use case category. As you mention we don't have good coverage there.

One quick question, how long do you think the benchmark will run for ? I'm assuming it will be relatively short (<1 hour) so that we can easily run it nightly as part of the bench runs.

Member

mhdawson commented Aug 29, 2017

As discussed in the bench-marking meeting yesterday it would be great to add coverage for that use case category. As you mention we don't have good coverage there.

One quick question, how long do you think the benchmark will run for ? I'm assuming it will be relatively short (<1 hour) so that we can easily run it nightly as part of the bench runs.

@bmeurer

This comment has been minimized.

Show comment
Hide comment
@bmeurer

bmeurer Aug 29, 2017

Member

Yes, the idea is that it runs only for a couple of minutes.

Member

bmeurer commented Aug 29, 2017

Yes, the idea is that it runs only for a couple of minutes.

@bmeurer

This comment has been minimized.

Show comment
Hide comment
@bmeurer

bmeurer Sep 11, 2017

Member

Ok quick status update: I compiled an initial version of the WebToolingBenchmark, consisting of a test case for Chai and one for Espree. I'm currently working on generating a useful test case for TypeScript and I'm in contact with the webpack folks about a proper benchmark.

Member

bmeurer commented Sep 11, 2017

Ok quick status update: I compiled an initial version of the WebToolingBenchmark, consisting of a test case for Chai and one for Espree. I'm currently working on generating a useful test case for TypeScript and I'm in contact with the webpack folks about a proper benchmark.

@bmeurer

This comment has been minimized.

Show comment
Hide comment
@bmeurer

bmeurer Oct 13, 2017

Member

Ok, I have a demoable version at github.com/v8/web-tooling-benchmark. Please give it a go and let me know what you think.

Member

bmeurer commented Oct 13, 2017

Ok, I have a demoable version at github.com/v8/web-tooling-benchmark. Please give it a go and let me know what you think.

bmeurer added a commit to v8/web-tooling-benchmark that referenced this issue Oct 18, 2017

Initial import of the Web Tooling Benchmark.
This is the initial version of the Web Tooling Benchmark as discussed in
nodejs/benchmarking#138, and contains tests
for Chai and Espree only. It's meant to be a starting point for further
extension.

bmeurer added a commit to v8/web-tooling-benchmark that referenced this issue Oct 18, 2017

[source-map] Import a benchmark for the mozilla/source-map package.
As requested earlier in the discussion on

  nodejs/benchmarking#138 (comment)

this adds a benchmark for the popular source-map package, which covers
both the serialization and the parsing of source maps. The payload is
the source map for the minified source-map.js itself.

We use `backbone-min.map`, `jquery.min.map`, `preact.min.js.map` and
`source-map.min.js.map` (all from the lastest versions) as payloads
for the benchmark.
@mhdawson

This comment has been minimized.

Show comment
Hide comment
@mhdawson

mhdawson Nov 22, 2017

Member

Benchmark is now running in nightly builds. I think this issue can now be closed. Closing, please let me know if it should still be open.

Member

mhdawson commented Nov 22, 2017

Benchmark is now running in nightly builds. I think this issue can now be closed. Closing, please let me know if it should still be open.

@mhdawson mhdawson closed this Nov 22, 2017

@bmeurer

This comment has been minimized.

Show comment
Hide comment
@bmeurer

bmeurer Nov 23, 2017

Member

Yes, thanks @tebbi and @mhdawson for getting it up and running!

Member

bmeurer commented Nov 23, 2017

Yes, thanks @tebbi and @mhdawson for getting it up and running!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment