Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add performance testing and benchmarking #1524

Closed
lmccart opened this Issue Jul 13, 2016 · 41 comments

Comments

Projects
None yet
6 participants
@lmccart
Copy link
Member

lmccart commented Jul 13, 2016

We need some way to test performance of various p5 functions, cross-browser and cross-platform to determine when optimizations are actually that, and when they add unnecessary complexity.
This is a follow up from #1512.

@kadamwhite

This comment has been minimized.

Copy link
Contributor

kadamwhite commented Jul 14, 2016

In #1520 @rwaldron proposed using grunt-benchmark for this. It seems we'd need to do a little extra wiring to execute them in multiple browsers but that's probably a good place to start.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 27, 2017

@lmccart I would like to take this one. Performance benchmarking get's me exited! 😆

It sounds like the goal here is to have an npm task to benchmark performance optimizations when implementing them to know that they are helping. I think it sounds like more of a dev task than part of the automated build.

If it doesn't need to run on every build, but a task that developers kick off to test their performance optimizations, then I recommend using karma, and karma-benchmark.

Karma lets you run code in multiple real browsers on your local machine, and karma-benchmark uses benchmark.js (used by jsPerf) to execute and output benchmarks in multiple browsers.

Here is example output of a test I ran on my laptop:

Chrome 62.0.3202 (Linux)  Array iteration: util.each at 73838720 ops/sec
Chrome 62.0.3202 (Linux)  Array iteration: Array.forEach at 248844823 ops/sec
Chrome 62.0.3202 (Linux)  Array search: util.contains at 25346940 ops/sec
Chrome 62.0.3202 (Linux)  Array search: Array.indexOf at 12303891 ops/sec
Chrome 62.0.3202 (Linux)
  Array iteration: Array.forEach at 248844823 ops/sec (3.37x faster than util.each)
  Array search: util.contains at 25346940 ops/sec (2.06x faster than Array.indexOf)

Running "karma:firefox" (karma) task
WARN [karma]: Port 9876 in use
INFO [karma]: Karma v0.10.10 server started at http://localhost:9877/
INFO [launcher]: Starting browser Firefox
INFO [Firefox 56.0.0 (Fedora)]: Connected on socket ZgSjYh_osO5rZacRazqw
Firefox 56.0.0 (Fedora)  Array iteration: util.each at 116097316 ops/sec
Firefox 56.0.0 (Fedora)  Array iteration: Array.forEach at 120121466 ops/sec
Firefox 56.0.0 (Fedora)  Array search: util.contains at 29879971 ops/sec
Firefox 56.0.0 (Fedora)  Array search: Array.indexOf at 28150961 ops/sec
Firefox 56.0.0 (Fedora)
  Array iteration: Array.forEach at 120121466 ops/sec (1.03x faster than util.each)
  Array search: util.contains at 29879971 ops/sec (1.06x faster than Array.indexOf)

Done, without errors.
[jsprague@localhost karma-benchmark-example]$ 

You can see it ran the same test in both Firefox and Chrome and output the results. Anyone have any objections to using karma-benchmark for this?

@meiamsome

This comment has been minimized.

Copy link
Member

meiamsome commented Oct 27, 2017

It may be beneficial to use Karma for the automated testing that is performed too - #2174
I was considering looking into using Karma instead of the current system but I don't know how well it runs on Travis

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 29, 2017

Ok here are an initial set of requirements based on disussion in #1512.:

  • Able to benchmark code across multiple browsers

  • Able to only run specific benchmarks or all benchmarks

  • Easy to add new benchmarks

  • Able to benchmark local build vs what is in production for comparison

  • Able to benchmark both un-minified and minified targets

I think that's good to start with.

Regarding benchmarking across multiple platforms i.e. Linux, Windows, Mac. I don't think this is possible using just grunt. It would require some kind of docker or virtualization setup, or a cloud based build system, which is much harder and I think is out of scope for this issue. But could be raised in a separate issue. I think we can rely on each other if we want to benchmark across platform, and just ask for someone who is on windows, mac or Linux to run the benchmark if you want to see how it compares across platforms. Once we have a consistent way of benchmarking this will be easy.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 29, 2017

I have a working proof of concept for each of the above requirements. I'll try and get it documented and a pull request made by the end of today.

Jared-Sprague added a commit to Jared-Sprague/p5.js that referenced this issue Oct 30, 2017

Jared-Sprague added a commit to Jared-Sprague/p5.js that referenced this issue Oct 30, 2017

Jared-Sprague added a commit to Jared-Sprague/p5.js that referenced this issue Oct 30, 2017

Jared-Sprague added a commit to Jared-Sprague/p5.js that referenced this issue Oct 30, 2017

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 30, 2017

Ok I completed my initial branch that meets all of the above requirements. I would love to have someone else test it out!

If you want to test it out checkout my benchmarking branch

I have added a grunt task that runs the performance benchmarks in multiple real browsers on the developers local machine. It will automatically detect which browsers are installed from the following list (Chrome, Firefox, Safari, Edge, IE) and run the benchmarks in all installed browsers and report the results.

Basic instructions:

Install the new dependancies

npm install

Do a build, the benchmarks expect a local built p5.js library

grunt

Run all the benchmarks

grunt karma

Run a specific benchmark

grunt karma:<target_benchmark>

Example

grunt karma:random-dev

Outputs:

Chrome 62.0.3202 (Linux 0.0.0)
  p5 random() vs Math.random(): Math.random() at 95811115 ops/sec (1.26x faster than p5 random())
Firefox 56.0.0 (Fedora 0.0.0)
  p5 random() vs Math.random(): Math.random() at 2367566507 ops/sec (1.14x faster than p5 random())

Done, without errors.

Adding new benchmarks

  1. Create a new benchmark file at this path and name format: bench/*.bench.js
  2. Write the benchmark here is documentation karma-benchmark
  3. Add the benchmark target to grunt-karma.js

Comparing Prod to Dev

karma-benchmark can load remote files. So it's easy to include the p5.js prod build and compare it to the dev build. This is important when you're working on improving performance to compare your changes to what is in production. To do this simply make two targets in grunt-karma.js one for prod and one for dev.

  'random-prod': {
    options: {
      files: [
        'https://cdnjs.cloudflare.com/ajax/libs/p5.js/<%= pkg.version %>/p5.js',
        'bench/random.bench.js',
      ],
    },
  },
  'random-dev': {
    options: {
      files: [
        'lib/p5.js',
        'bench/random.bench.js',
      ],
    },
  },

You can see that random-prod actually loads the latest build from CDN. Then to compare you can run both targets using:

grunt karma:random-prod karma:random-dev

Notes

I chose to put the grunt-karma tasks in it's own file grunt-karma.js instead of the main Gruntfile.js, because as we add more benchmarks overtime the file could grow quite long, and I wanted to keep the main Gruntfile clean.

Let me know if you have any questions!

@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 30, 2017

Is it possible to have something like grunt karma:random that automatically does grunt karma:random-prod karma:random-dev? Since we want to compare with production it would be nice to have a quick shortcut to run that.

Also, I haven't run it myself yet but what is the typical runtime for this benchmarking test? Would it be feasible to have it as part of the default task?

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 30, 2017

@limzykenneth Thanks for the reply! Yes since the commands are just regular grunt tasks, You could make a short-cut the same way we do for other things using:

grunt.registerTask('random', ['karma:random-prod', 'karma:random-dev']);

These are grouped together at the bottom of the Gruntfile.js under the comment // Create the multitasks.

As far as making it part of the default task; I would not recommend this. For a few reasons:

  1. The main reason is that it will launch the installed browsers on your computer to run the benchmarks in them. Which could get quite annoying if every time you run a build browsers launch in your face.
  2. They do take time to complete so it will add to the build time. In this future when we have a lot of benchmarks this extra time could be significant. Which is why I designed it so it's easy to run specific benchmarks.
  3. Without a baseline service level objective (SLO) for performance that will fail if it's not met, then running performance benchmarks every build isn't telling you anything useful.

I think for now running karma is most useful as a standalone development task. It IS very useful for the following:

  1. Measuring gain when working on performance optimizations.
  2. Troubleshooting reports of performance problems.
  3. Finding areas of the library where performance could be improved.

I hope you get a chance to try it out, just clone this branch and follow the above instructions.

So far I've tested it on Linux and Mac and it works. I still need to test it on windows.

@meiamsome

This comment has been minimized.

Copy link
Member

meiamsome commented Oct 30, 2017

I'm assuming it isn't very easy to run the production vs development directly? Perhaps after loading one of them you could have a script that does devp5 = p5; and then load the second p5? Sadly, I think some functions are still referring to p5.prototype when they shouldn't be, so this doesn't work but it would be nice to have if possible

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 30, 2017

@meiamsome yeah I'm not sure if that's possible with karma. They way karma works is you tell it what files you want want included in the browser as <script> tags then it runs the benchmarks. So to benchmark against prod it includes the prod build directly from CDN runs the benchmarks, then includes the dev build from the local file system and runs the benchmarks. I don't think you could include them both at the same time unless you could namespace them somehow. But I don't think karma supports that.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 30, 2017

I'd like to re-open this issue until at least one other person can test it, and verify it works for them.

I tested it on windows and it detected IE and Edge and ran in all my browsers:

IE 11.0.0 (Windows 10 0.0.0)
  p5 random() vs Math.random(): Math.random() at 46588302 ops/sec (1.31x faster than p5 random())
Edge 15.15063.0 (Windows 10 0.0.0)
  p5 random() vs Math.random(): Math.random() at 103991873 ops/sec (1.96x faster than p5 random())
Chrome 61.0.3163 (Windows 10 0.0.0)
  p5 random() vs Math.random(): Math.random() at 64355820 ops/sec (1.14x faster than p5 random())
Firefox 56.0.0 (Windows 10 0.0.0)
  p5 random() vs Math.random(): Math.random() at 2481743170 ops/sec (1.32x faster than p5 random())

Done, without errors.

C:\Users\Jared\projects\p5.js>

@lmccart lmccart reopened this Oct 31, 2017

@lmccart

This comment has been minimized.

Copy link
Member Author

lmccart commented Oct 31, 2017

I agree, I wanted to review that PR before it got merged, but I think it's ok to leave the merge and test and close this after we've confirmed it covers all our bases. thank you for this by the way!!

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@lmccart sounds good, thanks! A side note Firefox Math.random() is insanely fast compared to the other browsers. Firefox runs 2.5 BILLION ops / sec vs. Chrome's 64 million.

@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 31, 2017

@Jared-Sprague I just got to test this and unfortunately I don't think it is running correctly on my machine. It keeps giving me the following warning then aborted:

Running "karma:random-prod" (karma) task
31 10 2017 11:03:36.948:INFO [framework.detect-browsers]: The following browsers were detected on your system: [ 'Chrome', 'Firefox', 'Safari' ]
31 10 2017 11:03:36.998:INFO [karma]: Karma v1.7.1 server started at http://0.0.0.0:9876/
31 10 2017 11:03:36.998:INFO [launcher]: Launching browsers Chrome, Firefox, Safari with unlimited concurrency
31 10 2017 11:03:37.004:INFO [launcher]: Starting browser Chrome
31 10 2017 11:03:37.011:INFO [launcher]: Starting browser Firefox
31 10 2017 11:03:37.015:INFO [launcher]: Starting browser Safari
31 10 2017 11:03:40.060:INFO [Chrome 62.0.3202 (Mac OS X 10.13.0)]: Connected on socket 5t7e9X9olscbuyVEAAAA with id 34856655
31 10 2017 11:03:42.227:INFO [Firefox 56.0.0 (Mac OS X 10.13.0)]: Connected on socket Yvljif8m8nMAUUYIAAAB with id 65778005
31 10 2017 11:03:43.382:INFO [Safari 11.0.0 (Mac OS X 10.13.0)]: Connected on socket eTMTztn53Gwip6ssAAAC with id 90053629
31 10 2017 11:03:43.729:INFO [Safari 11.0.0 (Mac OS X 10.13.0)]: Connected on socket SrCKWdUDbObBw46cAAAD with id 56938425
31 10 2017 11:03:48.120:WARN [Chrome 62.0.3202 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Chrome 62.0.3202 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 35533823 ops/sec
31 10 2017 11:03:49.780:WARN [Firefox 56.0.0 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Firefox 56.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 1153423286 ops/sec
31 10 2017 11:03:53.384:WARN [Safari 11.0.0 (Mac OS X 10.13.0)]: Disconnected (1 times), because no message in 10000 ms.
Safari 11.0.0 (Mac OS X 10.13.0) ERROR
  Disconnected, because no message in 10000 ms.
31 10 2017 11:03:53.731:WARN [Safari 11.0.0 (Mac OS X 10.13.0)]: Disconnected (1 times), because no message in 10000 ms.
Safari 11.0.0 (Mac OS X 10.13.0) ERROR
  Disconnected, because no message in 10000 ms.
Chrome 62.0.3202 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 71790547 ops/sec
Firefox 56.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 1378679235 ops/sec
Chrome 62.0.3202 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 71790547 ops/sec (2.02x faster than p5 random())
Firefox 56.0.0 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 1378679235 ops/sec (1.20x faster than p5 random())
Warning: Task "karma:random-prod" failed. Use --force to continue.

Aborted due to warnings.

It happens with all commands grunt karma grunt karma:random-prod karma:random-dev grunt karma:random-prod grunt karma:random-dev. At first look it seems the timeout need to be increased?

Running on a late 2016 MacBook Pro, versions are included in the logs.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth Hey! Thanks for testing! Looking at your log, it ran successfully in in Chrome and Firefox.

The last thing it outputs is the calculation of which benchmark is faster, this part:

Chrome 62.0.3202 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 71790547 ops/sec (2.02x faster than p5 random())
Firefox 56.0.0 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 1378679235 ops/sec (1.20x faster than p5 random())

But it wasn't able to run in Safari for some reason, which caused a warning. Grunt picked up that warning and output the last message Aborted due to warnings. Which is kind of misleading because it didn't actually abort. It finished the whole run but since there was a warning grunt says that.

I was able to run it successfully on my friends macbook in safari so I'm wondering what is different about your safari? Do you have any plugins installed on safari? I'm going to do some testing on my macbook today. Also could you try commenting out Chrome and Firefox, and only running it in safari, and comment out safari and only run in chrome and firefox. you can do that in karma.conf.js:

        // Filter out nightly and dev builds
        // if (availableBrowser.indexOf('Chrome') >-1 )  browsers.push('Chrome');
        // if (availableBrowser.indexOf('Firefox') >-1 ) browsers.push('Firefox');
        if (availableBrowser.indexOf('Safari') >-1 )  browsers.push('Safari');
        // if (availableBrowser.indexOf('Edge') >-1 )    browsers.push('Edge');
        // if (availableBrowser.indexOf('IE') >-1 )      browsers.push('IE');
@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 31, 2017

Yeah, any idea why it failed on Safari? I have several tabs opened when I ran it because Safari is my default browser, could that be the reason?

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth Ah knowing that safari is your default browser helps. My first guess is addons blocking websockets. Do you have any ad blockers installed? Could you try running with addons disabled?

You can try running the tests just against safari by commenting out the other browsers in karam.conf.js:

        // Filter out nightly and dev builds
        // if (availableBrowser.indexOf('Chrome') >-1 )  browsers.push('Chrome');
        // if (availableBrowser.indexOf('Firefox') >-1 ) browsers.push('Firefox');
        if (availableBrowser.indexOf('Safari') >-1 )  browsers.push('Safari');
        // if (availableBrowser.indexOf('Edge') >-1 )    browsers.push('Edge');
        // if (availableBrowser.indexOf('IE') >-1 )      browsers.push('IE');
@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 31, 2017

For addons, I only have an adblocker installed but running it with that disabled it still failed the same way.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth If you comment out safari, and just run in chrome and firefox does it finish without errors?

@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 31, 2017

Yup, it finish completely, running both prod and dev.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth Does it launch safari at all?

@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 31, 2017

Yes it does, and it appears to be running the benchmark.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth Ok I'm going to get my wife's macbook setup so I can try reproducing this.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth Well the good news is I got p5.js build setup on my macbook running the same version of safari as you 11, and I'm able to reproduce the exact same error. It runs fine in Chrome but times out in safari. I'm really glad I can reproduce it because now I can fix it without having to troubleshoot over github issue 😄

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth Ok adding these configs to karma.conf.js fixes the problem for me, can you try adding these to karma.conf.js locally?

captureTimeout: 60000,
browserDisconnectTimeout: 10000,
browserDisconnectTolerance: 3,
browserNoActivityTimeout: 60000,
@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 31, 2017

yup, works perfectly for me as well!

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

@limzykenneth Sweet! 🎉 Can you post your results for random-prod and random-dev? Here are mine:

Running "karma:random-dev" (karma) task
31 10 2017 12:36:41.641:INFO [framework.detect-browsers]: The following browsers were detected on your system: [ 'Chrome', 'Safari' ]
31 10 2017 12:36:41.719:INFO [karma]: Karma v1.7.1 server started at http://0.0.0.0:9876/
31 10 2017 12:36:41.719:INFO [launcher]: Launching browsers Chrome, Safari with unlimited concurrency
31 10 2017 12:36:41.726:INFO [launcher]: Starting browser Chrome
31 10 2017 12:36:41.733:INFO [launcher]: Starting browser Safari
31 10 2017 12:36:42.489:INFO [Chrome 61.0.3163 (Mac OS X 10.12.6)]: Connected on socket 87wsWxa6-z9SvDwXAAAA with id 14259841
31 10 2017 12:36:43.248:INFO [Safari 11.0.0 (Mac OS X 10.12.6)]: Connected on socket LbSYWodF_byuuiBOAAAB with id 32057535
31 10 2017 12:36:49.282:WARN [Chrome 61.0.3163 (Mac OS X 10.12.6)]: Adapter did not report total number of specs.
Chrome 61.0.3163 (Mac OS X 10.12.6)  p5 random() vs Math.random(): p5 random() at 52287819 ops/sec
31 10 2017 12:36:50.052:WARN [Safari 11.0.0 (Mac OS X 10.12.6)]: Adapter did not report total number of specs.
Safari 11.0.0 (Mac OS X 10.12.6)  p5 random() vs Math.random(): p5 random() at 24620666 ops/sec
Chrome 61.0.3163 (Mac OS X 10.12.6)  p5 random() vs Math.random(): Math.random() at 70721897 ops/sec
Safari 11.0.0 (Mac OS X 10.12.6)  p5 random() vs Math.random(): Math.random() at 45985632 ops/sec
Chrome 61.0.3163 (Mac OS X 10.12.6)
  p5 random() vs Math.random(): Math.random() at 70721897 ops/sec (1.35x faster than p5 random())
Safari 11.0.0 (Mac OS X 10.12.6)
  p5 random() vs Math.random(): Math.random() at 45985632 ops/sec (1.87x faster than p5 random())

Running "karma:random-prod" (karma) task
31 10 2017 12:36:56.896:INFO [framework.detect-browsers]: The following browsers were detected on your system: [ 'Chrome', 'Safari' ]
31 10 2017 12:36:56.913:INFO [karma]: Karma v1.7.1 server started at http://0.0.0.0:9876/
31 10 2017 12:36:56.913:INFO [launcher]: Launching browsers Chrome, Safari with unlimited concurrency
31 10 2017 12:36:56.916:INFO [launcher]: Starting browser Chrome
31 10 2017 12:36:56.919:INFO [launcher]: Starting browser Safari
31 10 2017 12:36:57.797:INFO [Chrome 61.0.3163 (Mac OS X 10.12.6)]: Connected on socket sN-bNwi2k6UqwCT-AAAC with id 73478204
31 10 2017 12:36:58.375:INFO [Safari 11.0.0 (Mac OS X 10.12.6)]: Connected on socket xHZDv3TWmYxgWmV4AAAD with id 38149157
31 10 2017 12:37:04.468:WARN [Chrome 61.0.3163 (Mac OS X 10.12.6)]: Adapter did not report total number of specs.
Chrome 61.0.3163 (Mac OS X 10.12.6)  p5 random() vs Math.random(): p5 random() at 52189893 ops/sec
31 10 2017 12:37:04.574:WARN [Safari 11.0.0 (Mac OS X 10.12.6)]: Adapter did not report total number of specs.
Safari 11.0.0 (Mac OS X 10.12.6)  p5 random() vs Math.random(): p5 random() at 24447015 ops/sec
Safari 11.0.0 (Mac OS X 10.12.6)  p5 random() vs Math.random(): Math.random() at 248728691 ops/sec
Chrome 61.0.3163 (Mac OS X 10.12.6)  p5 random() vs Math.random(): Math.random() at 70143628 ops/sec
Chrome 61.0.3163 (Mac OS X 10.12.6)
  p5 random() vs Math.random(): Math.random() at 70143628 ops/sec (1.34x faster than p5 random())
Safari 11.0.0 (Mac OS X 10.12.6)
  p5 random() vs Math.random(): Math.random() at 248728691 ops/sec (10.17x faster than p5 random())

Done, without errors.

One thing I noticed interesting, Safari's Math.random() is sometimes 10x faster, and other times not, it seems non-deterministic compared to Chrome and Firefox which are both consistently around 1.0-1.5x faster every run. So there may be something unique about Safari.

Jared-Sprague added a commit to Jared-Sprague/p5.js that referenced this issue Oct 31, 2017

@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Oct 31, 2017

For me it's rather consistent for all browsers (to a certain extend, depends on how much error you are prepared to take), only Chrome is fluctuating around 2x to 9x. I think it may be worth to run this in more controlled environment to make sure the fluctuations are not due to our machines using resources elsewhere, also maybe not run all the browsers at once?

Another thing I noticed is that it opens a lot of tabs in Safari for some reason.

Running "karma:random-prod" (karma) task
31 10 2017 17:30:59.932:INFO [framework.detect-browsers]: The following browsers were detected on your system: [ 'Chrome', 'Firefox', 'Safari' ]
31 10 2017 17:30:59.975:INFO [karma]: Karma v1.7.1 server started at http://0.0.0.0:9876/
31 10 2017 17:30:59.975:INFO [launcher]: Launching browsers Chrome, Firefox, Safari with unlimited concurrency
31 10 2017 17:30:59.985:INFO [launcher]: Starting browser Chrome
31 10 2017 17:31:00.005:INFO [launcher]: Starting browser Firefox
31 10 2017 17:31:00.011:INFO [launcher]: Starting browser Safari
31 10 2017 17:31:05.376:INFO [Firefox 56.0.0 (Mac OS X 10.13.0)]: Connected on socket CdPq2wJ7BL-eNYPlAAAA with id 22964058
31 10 2017 17:31:05.580:INFO [Chrome 62.0.3202 (Mac OS X 10.13.0)]: Connected on socket hmgGb0_BibY6KiHoAAAB with id 81083656
31 10 2017 17:31:06.390:INFO [Safari 11.0.0 (Mac OS X 10.13.0)]: Connected on socket BhWyDVn2SvhOmiwZAAAC with id 44887796
31 10 2017 17:31:06.865:INFO [Safari 11.0.0 (Mac OS X 10.13.0)]: Connected on socket KAipOYNiaFqkKnitAAAD with id 43515002
31 10 2017 17:31:13.296:WARN [Chrome 62.0.3202 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Chrome 62.0.3202 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 34316625 ops/sec
31 10 2017 17:31:13.609:WARN [Safari 11.0.0 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Safari 11.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 21048380 ops/sec
31 10 2017 17:31:14.615:WARN [Firefox 56.0.0 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Firefox 56.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 991388970 ops/sec
Chrome 62.0.3202 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 52153282 ops/sec
Firefox 56.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 1194981558 ops/sec
Safari 11.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 32470376 ops/sec
Chrome 62.0.3202 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 52153282 ops/sec (1.52x faster than p5 random())
Safari 11.0.0 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 32470376 ops/sec (1.54x faster than p5 random())
Firefox 56.0.0 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 1194981558 ops/sec (1.21x faster than p5 random())

Running "karma:random-dev" (karma) task
31 10 2017 17:31:28.203:INFO [framework.detect-browsers]: The following browsers were detected on your system: [ 'Chrome', 'Firefox', 'Safari' ]
31 10 2017 17:31:28.321:INFO [karma]: Karma v1.7.1 server started at http://0.0.0.0:9876/
31 10 2017 17:31:28.322:INFO [launcher]: Launching browsers Chrome, Firefox, Safari with unlimited concurrency
31 10 2017 17:31:28.324:INFO [launcher]: Starting browser Chrome
31 10 2017 17:31:28.328:INFO [launcher]: Starting browser Firefox
31 10 2017 17:31:28.334:INFO [launcher]: Starting browser Safari
31 10 2017 17:31:31.215:INFO [Chrome 62.0.3202 (Mac OS X 10.13.0)]: Connected on socket ODPWlfEbAehGkEPzAAAE with id 64827918
31 10 2017 17:31:33.697:INFO [Firefox 56.0.0 (Mac OS X 10.13.0)]: Connected on socket hQbX9gODgMpXDOGZAAAF with id 88352701
31 10 2017 17:31:35.353:INFO [Safari 11.0.0 (Mac OS X 10.13.0)]: Connected on socket lxN3bu4-ZYdNokVjAAAG with id 43515002
31 10 2017 17:31:35.531:INFO [Safari 11.0.0 (Mac OS X 10.13.0)]: Connected on socket -LHS4XdG64JdVgVTAAAH with id 85521896
31 10 2017 17:31:39.432:WARN [Chrome 62.0.3202 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Chrome 62.0.3202 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 22620100 ops/sec
31 10 2017 17:31:41.840:WARN [Firefox 56.0.0 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Firefox 56.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 876451178 ops/sec
31 10 2017 17:31:42.185:WARN [Safari 11.0.0 (Mac OS X 10.13.0)]: Adapter did not report total number of specs.
Safari 11.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): p5 random() at 19183798 ops/sec
Chrome 62.0.3202 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 51736147 ops/sec
Firefox 56.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 1174074760 ops/sec
Safari 11.0.0 (Mac OS X 10.13.0)  p5 random() vs Math.random(): Math.random() at 18192796 ops/sec
Chrome 62.0.3202 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 51736147 ops/sec (2.29x faster than p5 random())
Firefox 56.0.0 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): Math.random() at 1174074760 ops/sec (1.34x faster than p5 random())
Safari 11.0.0 (Mac OS X 10.13.0)
  p5 random() vs Math.random(): p5 random() at 19183798 ops/sec (1.05x faster than Math.random())

Done, without errors.
@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Oct 31, 2017

Ok I created a PR #2325. Yeah I agree that developer workstations are never going to be a controlled environment, but this still gives us a lot of useful data. And It can definitely be used for developers to test performance optimizations relative to their own environment. Maybe in the future we can setup a Mac VM in the cloud that can run the benchmarks in a more controlled environment, but I think this is a good first start.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Nov 1, 2017

@limzykenneth I forgot to mention you can change how many browsers the benchmarks run in concurrently by changing the concurency setting in karma.conf.js.

For example if you only want to run in 1 browser at a time sequentialy you would set it 1 like so:

    // Concurrency level
    // how many browser should be started simultaneous
    concurrency: 1,
@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Nov 1, 2017

Ah, ok. Come to think of it, do you think you can write up a wiki just so we have something to refer to for running benchmark tests?

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Nov 1, 2017

Absolutely! I was planning on doing that as soon as this issue was accepted.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Nov 3, 2017

@lmccart @limzykenneth I did some more POC work tonight. I wen't through every comment and the main benchmarks performed done in #1512, and reproduced them in a karma-benchmark. Then I charted the results.

Two things I learned form this excersise:

  1. This is a really powerful system, I tested multiple permutations of a benchmark against all my installed browsers with a single grunt command.
  2. In doing so I discovered some suprising stuff.

Take a look at this chart, FE stands for Friendly Errors:
chart

There are a few main things that jump out in this chart:

  1. Firefox is both the fastest and slowest browser depending on how you are running it.
  2. Running the minified version of p5 is a big performance boost for both Chrome and Firefox
  3. Firefox non-minified with Friendly Errors ON is suuuper slow, it didn't even register on the chart.
  4. Firefox in Instance mode blows away chrome. Even non-minified.

I didn't include Math.random() on the chart because I wanted to compare the various ways of running random(). But here are the results for random() vs Math.random() with Friendly Errors OFF and using p5.js.min:

Chrome 62.0.3202 (Linux 0.0.0)
  Friendly Errors: OFF, Instance random() vs Math.random(): Math.random() at 92063010 ops/sec (1.06x faster than Instance random())
  Friendly Errors: OFF, Window random() vs Math.random(): Math.random() at 93712206 ops/sec (1.15x faster than window random())
Firefox 56.0.0 (Fedora 0.0.0)
  Friendly Errors: OFF, Instance random() vs Math.random(): Math.random() at 2290677743 ops/sec (1.10x faster than Instance random())
  Friendly Errors: OFF, Window random() vs Math.random(): Math.random() at 2278364215 ops/sec (19.00x faster than window random())

Based on the above in Chrome there is no significant difference between Math.random() and random() either in instance or window mode. But with Firefox Instance mode is the same as Math.random() but in window mode Math.random() is 20x faster than window random() But this is still orders of magnitude faster than Chrome.

I'd like to push my benchmarks can so others can test them, but I'd prefer to do it in a separate PR once #2325 has been merged.

Let me know you're thoughts! My thoughts are that I think this meets the requirements you were looking for for a cross-browser benchmarking system.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Nov 3, 2017

You can run the same benchmarks I used to generate the above chart by using my fe-benchmarks branch and using grunt karma

@lmccart what do you think about closing this issue and making issues based on research and discoveries from benchmarks that could be acted on? I think this issue has grown so long, maybe people aren't reading it. I also feel a lot better now that @limzykenneth has verified it works for him, after the changes in PR #2325.

@limzykenneth

This comment has been minimized.

Copy link
Member

limzykenneth commented Nov 3, 2017

It's always expected that the FES will have an impact on performance, as long as it is clear enough that that is the case, we just need to keep in mind to compare p5.js with p5.js and not with p5.min.js etc.

As for individual browsers having different performance boost or slow down, I guess it's down to implementation details of the browsers, I'm hoping that if we follow official specs as close as we can we won't have too big of a problem.

Also +1 to closing this and maybe start thinking about branching out from this work.

@lmccart

This comment has been minimized.

Copy link
Member Author

lmccart commented Nov 3, 2017

sorry I am so late to the game here! I just got a chance to test and it's working great on my machine. I did see the warnings:

03 11 2017 09:12:28.606:WARN [Safari 11.0.1 (Mac OS X 10.12.6)]: Adapter did not report total number of specs.
Safari 11.0.1 (Mac OS X 10.12.6)  p5 random() vs Math.random(): p5 random() at 27458177 ops/sec
03 11 2017 09:12:28.667:WARN [Chrome 61.0.3163 (Mac OS X 10.12.6)]: Adapter did not report total number of specs.
Chrome 61.0.3163 (Mac OS X 10.12.6)  p5 random() vs Math.random(): p5 random() at 60647206 ops/sec
03 11 2017 09:12:30.432:WARN [Firefox 55.0.0 (Mac OS X 10.12.0)]: Adapter did not report total number of specs.

but it seems like these can be ignored.

@Jared-Sprague this is really great work and a super helpful tool! I am in favor of closing this issue and opening other issues to address specific questions related to benchmarking. and thanks @limzykenneth for all the testing and feedback.

one side note, and maybe I am misunderstanding the test intention, but random() doesn't directly make use of the FES system. I think the real slowdown with FES happens when validateParameters() is called, which is in most of the 2D primitives right now. so it might be worth doing some tests to understand what the real impact is.

@lmccart

This comment has been minimized.

Copy link
Member Author

lmccart commented Nov 3, 2017

closing with #2325

@lmccart lmccart closed this Nov 3, 2017

@lmccart

This comment has been minimized.

Copy link
Member Author

lmccart commented Nov 3, 2017

oh and @Jared-Sprague yes, adding this info to the wiki would be very helpful. I would suggest making a new page, and linking to it from the development page.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Nov 3, 2017

@limzykenneth Yeah I definitely discovered that about FES while benchmarking. I wanted to see out of all the permutations of FE ON|OFF, p5.js, p5.min.js, Firefox, Chrome. Which combination was fastest. And it turns out the fastest is Firefox using p5.min.js in Instance mode, by FAR 20x faster than the fastest chrome score.

@lmccart Yeah it's not the FES itself it's the binding that happens when FES is enabled.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Nov 3, 2017

@lmccart Sounds good! I'll make a bench-marking wiki page and add some more notes to the Performance wiki page as well based on my testing. I think I'll write a short blog post too.

@Jared-Sprague

This comment has been minimized.

Copy link
Contributor

Jared-Sprague commented Nov 3, 2017

@lmccart Yep those warnings about adapter count can be ignored. Not sure what they are but they don't affect the benchmarks at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.