prototype benchmark command #849

Closed
wants to merge 3 commits into
from

Conversation

Projects
None yet
3 participants
Collaborator

caridy commented Dec 10, 2012

THIS IS AN EXPERIMENTAL COMMAND

This command enable us to write benchmark tests at the app level. It uses benchmark and benchtable in conjunction with YUI to facilitate such as tests. Also, any mojito or app level yui module is available for benchmarking without any extra configuration.

Steps:

  • npm i benchmark -g
  • npm i benchtable -g
  • mojito create app demo
  • cd demo
  • mkdir benchmarks
  • touch benchmarks/foo-benchmark.js
  • do your stuff on the new benchmark file
  • mojito benchmark ./benchmarks/foo-benchmark.js

Here is an example of a benchmark file:

YUI.add('strings-concat-benchmark', function (Y) {

    var suite = Y.BenchmarkSuite;

    suite.add('concat', function () {
        var b = 'foo' + 'bar';
    });

    suite.add('array join', function () {
        var b = ['foo', 'bar'].join('');
    });

}, '0.1', {requires: ['mojito-benchmark']});

To execute the test, you can do this:

$ mojito benchmark benchmarks/strings-concat-benchmark.js 
Benchmarking YUI module strings-concat-benchmark [./demo/benchmarks/strings-concat-benchmark.js]
Starting benchmarks.
concat x 168,013,961 ops/sec ±44.33% (28 runs sampled)
array join x 20,195,411 ops/sec ±2.09% (94 runs sampled)
⚠ Fastest is concat

Here is another example of the output after executing YUI Base benchmark (from YUI's source) to compare different base implementations:

$ mojito benchmark benchmarks/base-benchmark.js 
Benchmarking YUI module base-benchmark [./demo/benchmarks/base-benchmark.js]
Starting benchmarks.
Base x 6,573 ops/sec ±4.01% (86 runs sampled)
MyBase x 5,893 ops/sec ±3.35% (89 runs sampled)
MyBase with 10 simple value attributes x 4,793 ops/sec ±2.09% (93 runs sampled)
MyBase with 20 varied attributes x 2,792 ops/sec ±1.84% (93 runs sampled)
BaseCore x 31,725 ops/sec ±3.09% (89 runs sampled)
MyBaseCore x 24,552 ops/sec ±6.80% (76 runs sampled)
MyBaseCore with 10 simple value attributes x 8,977 ops/sec ±3.97% (84 runs sampled)
MyBaseCore with 20 varied attributes x 4,606 ops/sec ±3.06% (90 runs sampled)
⚠ Fastest is BaseCore

When it comes to benchtable, you can use different datasets to benchmark a functionality with different data. Here is the output of a benchmark table testing strings concatenation with short and long strings:

$ mojito benchmark benchmarks/strings-concat-benchtable.js 
Benchmarking YUI module strings-concat-benchtable [./demo/benchmarks/strings-concat-benchtable.js]
Starting benchmarks.
concat for params small strings (3 characters) x 12,273,020 ops/sec ±0.88% (88 runs sampled)
concat for params big strings (1025 characters) x 11,763,540 ops/sec ±1.10% (99 runs sampled)
array join for params small strings (3 characters) x 7,447,580 ops/sec ±2.63% (94 runs sampled)
array join for params big strings (1025 characters) x 958,773 ops/sec ±0.77% (100 runs sampled)
⚠ Fastest is concat for params small strings (3 characters)
+------------+------------------------------+-------------------------------+
|            | small strings (3 characters) | big strings (1025 characters) |
+------------+------------------------------+-------------------------------+
| concat     | 12,273,020 ops/sec           | 11,763,540 ops/sec            |
+------------+------------------------------+-------------------------------+
| array join | 7,447,580 ops/sec            | 958,773 ops/sec               |
+------------+------------------------------+-------------------------------+

As you can see you can test multiple dataset with each implementation.

GIST with the examples:

TODO:

  • define dependencies benchmark@1.0.x and benchtable@0.0.x
  • evaluate if the store can identify benchmark files
  • evaluate moving this infrastructure into its own package called mojito-benchmark.
  • verify benchmark (bz #5988048) and benchtable (bz #5988047) as ynpm pkgs.
Contributor

mojit0 commented Dec 10, 2012

+1

Contributor

drewfish commented Dec 10, 2012

A few notes:

  • It looks like the benchmark filename has to be the same name as the YUI module name. This should be explicitly noted in the documentation.
  • You should probably add an on('error') handler to show errors, otherwise they get swallowed and the user has no idea what went wrong with the benchmark.
  • I want to compare different versions of an operation. However, I don't want to do the setup in each iteration. There's no preCycle event (or the equivalent). (For example, I want to compare different versions of store.expandInstanceForEnv(), but the only way to do that is to put them all in different benchmark files, since there's currently no preCyle to do things like store.expandInstanceForEnv = function() {...}.)
Collaborator

caridy commented Dec 11, 2012

  • error handling was addressed:
$ mojito benchmark benchmarks/strings-concat-benchmark.js 
Benchmarking YUI module strings-concat-benchmark [./demo/benchmarks/strings-concat-benchmark.js]
Starting benchmarks.
concat x 142,256,065 ops/sec ±44.61% (24 runs sampled)
✖ array join: ReferenceError: g is not defined
⚠ Fastest is concat
  • Usage doc was expanded to notice 3 important things:
NOTES:
  * The name of the yui module that defines the benchmark test should
    match the filename. In the first example, the test should be defined
    as `YUI.add("foo-benchmark", function (Y) {/*...*/});`, otherwise
    the test will fail.
  * Any yui module from yui core, mojito core or application level module
    that runs on the server runtime could be required as part of the
    "requires" array in the benchmark test without any extra configuration.
  * If you want to require a yui module that is meant to run in the client
    runtime, make sure you specify the proper --context option.

About comparing different implementations, the recommended way will be to have store.expandInstanceForEnv and store.expandInstanceForEnvAlt in your code, so, you can do the same you do in the original, plus some enhancements, and see how it goes. I will recommend not to mock things or write actual code in the tests directly if the intent if to test a feature that already exists.

/cc @drewfish

Contributor

drewfish commented Dec 11, 2012

+1

Collaborator

caridy commented Jan 7, 2013

We will put this on hold in favor of "mojito-benchmak" npm package.

Collaborator

caridy commented Apr 5, 2013

@isao whenever you get a chance, we should revisit this now that we have mojito-cli

Collaborator

caridy commented Feb 14, 2014

very old PR, closing.

@caridy caridy closed this Feb 14, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment