feat(benchmark): add benchmark package and travis job #4133

filipesilva · 2017-01-20T16:32:57Z

Adds @ngtools/benchmark allowing the benchmark of commands via the ngbench binary.

Total command running time is tracked, and specific matches within the command output can be added i.e. the Time: 1443ms report for individual builds.

This is mostly a dev tool. Discussion is still needed to determine if it should be added.

Example usage:

No arguments defaults to benchmarking ng serve rebuild times (the two commands are equivalent).

kamik@T460p MINGW64 /d/sandbox/master-project (master)
$ ngbench
Base command: ng serve --no-progress
Iterations: 5

Full command: ng serve --no-progress
Time average: 16741.4
Times: 16657,16871,16897,16486,16796
Match average: 1433.2
Matches: 1443,1444,1446,1415,1418

Benchmark execution time: 83722ms

kamik@T460p MINGW64 /d/sandbox/master-project (master)
$ ngbench --command="ng serve --no-progress" --match="Time: (.*)ms" --match-count=4 --match-edit-file="src/main.ts" --match-edit-string="'benchmark string';"
Base command: ng serve --no-progress
Iterations: 5

Full command: ng serve --no-progress
Time average: 16998
Times: 17161,16904,17055,17439,16431
Match average: 1498.2
Matches: 1538,1524,1478,1458,1493

Benchmark execution time: 85002ms

You can select a specific command to benchmark total running time:

kamik@T460p MINGW64 /d/sandbox/master-project (master)
$ ngbench --command="ng version"
Base command: ng version
Iterations: 5

Full command: ng version
Time average: 1762.4
Times: 1711,1786,1783,1779,1753

Benchmark execution time: 8826ms

You can also determine a string to match in the command output

kamik@T460p MINGW64 /d/sandbox/master-project (master)
$ ngbench --command="ng build --no-progress" --match="Time: (.*)ms"
Base command: ng build --no-progress
Iterations: 5

Full command: ng build --no-progress
Time average: 10382
Times: 10118,10350,10105,10747,10590
Match average: 6723
Matches: 6490,6776,6398,7263,6688

Benchmark execution time: 51925ms

Fix #3606

filipesilva · 2017-01-27T20:26:46Z

This command should instead be a separate package.

hansl · 2017-02-05T05:23:14Z

packages/@ngtools/benchmark/src/utils.ts

+}
+
+// Returns an array of `length` length, filled with `undefined`
+export function makeArray(length: number) {


Array.from()

hansl · 2017-02-05T05:30:52Z

packages/@ngtools/benchmark/src/utils.ts

+}
+
+// Returns average of all numbers in `arr`
+export function average(arr: any[]) {


We have lodash, _.mean() would work.

hansl · 2017-02-05T05:31:16Z

packages/@ngtools/benchmark/src/utils.ts

+import { BenchmarkOptions } from './benchmark-options';
+
+// Log to console and logFile if it exists
+export function benchmarkLog(benchmarkOptions: BenchmarkOptions, str: string) {


Optional for this; could you use the @ngtools/logger library?

hansl · 2017-02-05T05:32:16Z

packages/@ngtools/benchmark/package.json

@@ -0,0 +1,15 @@
+{
+  "name": "@ngtools/benchmark",
+  "version": "0.1.0",


Could you set the version to 0.0.0?

hansl · 2017-02-05T05:39:28Z

packages/@ngtools/benchmark/src/utils.ts

+// Returns a promise with the result of calling `times` times the `fn` function with `args`
+// `fn` will be called in parallel, and is expected to return a promise
+// Not used currently
+export function parallelMultiPromise(times: number, fn: any, ...args: string[]) {


I had it initially but there was no command that could benefit from being ran in parallel. I'll remove it.

hansl · 2017-02-05T05:42:03Z

packages/@ngtools/benchmark/src/benchmark.ts

+} from './utils';
+
+export const benchmark = (benchmarkOptions: BenchmarkOptions) => {
+  const log = benchmarkLog.bind(null, benchmarkOptions);


Yeah let's use @ngtools/logger.

hansl · 2017-02-05T05:43:31Z

packages/@ngtools/benchmark/src/benchmark.ts

+  serialMultiPromise
+} from './utils';
+
+export const benchmark = (benchmarkOptions: BenchmarkOptions) => {


export function

hansl · 2017-02-05T05:53:08Z

packages/@ngtools/benchmark/src/benchmark.ts

+  flagCombinations.unshift([]);
+
+  const startTime = Date.now();
+  log(`Base command: ${benchmarkOptions.command}`);


I think we could use a JSON.stringify, or maybe a for...in. What do you think?

Order wouldn't be guaranteed with those, nor would they allow to show a more verbose text as easily.

hansl · 2017-02-05T06:14:35Z

packages/@ngtools/benchmark/src/utils.ts

+    childProcess.on('exit', (err: number) => {
+      result.time = Date.now() - startTime;
+      return err && !result.overrideErr
+        ? resolve({ err, stdout, stderr })


No, still I since want to gather the remaining results. If you're doing a lot of iterations you don't want to throw everything away just because one failed.

hansl · 2017-02-05T06:29:33Z

packages/@ngtools/benchmark/src/index.ts

+      default: [],
+      alias: 'ea'
+    },
+    'match': {


It seems like these would always be the same for commands, right? Like for the serve command. I'm not sure, just wondering.

It varies a bit... on some tests I wanted to benchmark the difference in rebuilds between the main chunk and lazy loaded chunks. Or to check the rebuild time for css/html. In those cases the written string also changes.

Added css/html/script tests

hansl · 2017-02-05T06:30:03Z

In case you're missing it or never saw it: https://lodash.com/docs/

filipesilva · 2017-02-06T11:41:49Z

@hansl can you elaborate on which parts could do with being replaced with lodash? Seems excessive to add it just to replace average() and isNumber().

hansl · 2017-05-22T16:36:41Z

Since we're never going to publish this, it should live in tools/bench instead of packages. thanks

Adds `tools/bench` allowing the benchmark of commands via the `ngbench` binary in development environments.

filipesilva · 2017-05-23T14:16:58Z

@hansl done.

angular-automatic-lock-bot · 2019-09-12T08:41:25Z

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

googlebot added the cla: yes label Jan 20, 2017

filipesilva force-pushed the benchmark-command branch 3 times, most recently from 8daba22 to a7d4e5f Compare January 24, 2017 21:52

filipesilva added the state: WIP label Jan 27, 2017

filipesilva changed the title ~~feat(benchmark): add benchmark command~~ feat(benchmark): add benchmark package and travis job Jan 27, 2017

filipesilva force-pushed the benchmark-command branch 20 times, most recently from c1bfc51 to cf8d7a7 Compare February 4, 2017 00:31

filipesilva removed the state: WIP label Feb 4, 2017

filipesilva requested a review from hansl February 4, 2017 00:31

filipesilva force-pushed the benchmark-command branch from cf8d7a7 to 6716db0 Compare February 4, 2017 00:42

hansl suggested changes Feb 5, 2017

View reviewed changes

filipesilva force-pushed the benchmark-command branch 3 times, most recently from 3d85107 to b0194a4 Compare February 8, 2017 19:45

filipesilva mentioned this pull request Feb 8, 2017

[WIP] add benchmark scripts #3313

Closed

filipesilva force-pushed the benchmark-command branch 2 times, most recently from 08df4c2 to 2e534e5 Compare February 9, 2017 09:53

filipesilva force-pushed the benchmark-command branch 3 times, most recently from b593948 to 2691389 Compare February 17, 2017 19:26

filipesilva closed this Feb 25, 2017

filipesilva deleted the benchmark-command branch February 25, 2017 11:13

filipesilva restored the benchmark-command branch February 25, 2017 11:15

filipesilva reopened this Feb 25, 2017

filipesilva assigned hansl May 8, 2017

filipesilva force-pushed the benchmark-command branch from 2691389 to 8ae89b6 Compare May 22, 2017 09:30

filipesilva force-pushed the benchmark-command branch 4 times, most recently from 9ce843f to 31286cb Compare May 23, 2017 11:21

test: add benchmark tests

49ca9dd

Adds `tools/bench` allowing the benchmark of commands via the `ngbench` binary in development environments.

filipesilva force-pushed the benchmark-command branch from 31286cb to 49ca9dd Compare May 23, 2017 13:22

hansl removed their assignment Feb 8, 2018

hansl closed this Feb 22, 2018

angular-automatic-lock-bot bot locked and limited conversation to collaborators Sep 12, 2019

feat(benchmark): add benchmark package and travis job #4133

feat(benchmark): add benchmark package and travis job #4133

Uh oh!

Conversation

filipesilva commented Jan 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filipesilva commented Jan 27, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hansl commented Feb 5, 2017

Uh oh!

filipesilva commented Feb 6, 2017

Uh oh!

hansl commented May 22, 2017

Uh oh!

filipesilva commented May 23, 2017

Uh oh!

angular-automatic-lock-bot bot commented Sep 12, 2019

Uh oh!

Uh oh!

filipesilva commented Jan 20, 2017 •

edited

Loading