Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test/bench/go1: benchmark is noisy on node/wasm #33749

Open
beenshi opened this issue Aug 21, 2019 · 5 comments

Comments

@beenshi
Copy link

commented Aug 21, 2019

Run the go1 benchmark twice, with the same go version.

(with node 8.16.0 and the recent master branch of go)

name                   old time/op    new time/op    delta
BinaryTree17              25.7s ± 0%     25.7s ± 0%  +0.13%  (p=0.000 n=25+29)
Fannkuch11                13.6s ± 0%     13.9s ± 0%  +1.71%  (p=0.000 n=29+30)
FmtFprintfEmpty           273ns ± 0%     272ns ± 0%  -0.37%  (p=0.000 n=25+30)
FmtFprintfString          565ns ± 1%     554ns ± 0%  -1.91%  (p=0.000 n=30+20)
FmtFprintfInt             716ns ± 1%     674ns ± 0%  -5.88%  (p=0.000 n=30+30)
FmtFprintfIntInt         1.24µs ± 0%    1.16µs ± 0%  -6.35%  (p=0.000 n=30+29)
FmtFprintfPrefixedInt    1.19µs ± 1%    1.11µs ± 0%  -6.89%  (p=0.000 n=30+30)
FmtFprintfFloat          2.38µs ± 0%    2.56µs ± 0%  +7.70%  (p=0.000 n=30+30)
FmtManyArgs              5.77µs ± 0%    5.69µs ± 0%  -1.39%  (p=0.000 n=30+29)
GobDecode                65.8ms ± 0%    67.5ms ± 0%  +2.57%  (p=0.000 n=30+29)
GobEncode                33.5ms ± 0%    34.5ms ± 0%  +2.91%  (p=0.000 n=30+30)
Gzip                      1.09s ± 0%     1.08s ± 0%  -0.81%  (p=0.000 n=28+28)
Gunzip                    239ms ± 0%     229ms ± 0%  -4.21%  (p=0.000 n=27+29)
HTTPClientServer          323µs ± 0%     327µs ± 0%  +1.22%  (p=0.000 n=29+28)
JSONEncode               70.0ms ± 0%    71.9ms ± 0%  +2.63%  (p=0.000 n=30+29)
JSONDecode                591ms ± 0%     572ms ± 0%  -3.30%  (p=0.000 n=30+29)
Mandelbrot200            5.79ms ± 0%    5.79ms ± 0%  +0.01%  (p=0.000 n=25+25)
GoParse                  38.8ms ± 0%    39.5ms ± 0%  +1.69%  (p=0.000 n=29+29)
RegexpMatchEasy0_32       654ns ± 0%     670ns ± 0%  +2.52%  (p=0.000 n=29+25)
RegexpMatchEasy0_1K      5.72µs ± 1%    5.41µs ± 0%  -5.55%  (p=0.000 n=28+29)
RegexpMatchEasy1_32       578ns ± 0%     546ns ± 0%  -5.46%  (p=0.000 n=24+28)
RegexpMatchEasy1_1K      5.68µs ± 0%    5.74µs ± 0%  +1.11%  (p=0.000 n=29+29)
RegexpMatchMedium_32     26.8ns ± 0%    26.8ns ± 0%    ~     (all equal)
RegexpMatchMedium_1K      208µs ± 0%     225µs ± 0%  +8.36%  (p=0.000 n=28+25)
RegexpMatchHard_32       11.7µs ± 0%    12.1µs ± 0%  +3.32%  (p=0.000 n=30+30)
RegexpMatchHard_1K        341µs ± 0%     360µs ± 0%  +5.57%  (p=0.000 n=30+29)
Revcomp                   2.32s ± 0%     2.31s ± 0%  -0.65%  (p=0.000 n=23+26)
Template                  723ms ± 0%     724ms ± 0%  +0.16%  (p=0.000 n=30+30)
TimeParse                3.59µs ± 0%    3.66µs ± 0%  +1.96%  (p=0.000 n=29+29)
TimeFormat               3.93µs ± 1%    3.74µs ± 0%  -4.74%  (p=0.000 n=30+30)
[Geo mean]                340µs          339µs       -0.21%
	
name                   old speed      new speed      delta
GobDecode              11.7MB/s ± 0%  11.4MB/s ± 0%  -2.51%  (p=0.000 n=30+29)
GobEncode              22.9MB/s ± 0%  22.2MB/s ± 0%  -2.82%  (p=0.000 n=30+29)
Gzip                   17.8MB/s ± 0%  17.9MB/s ± 0%  +0.83%  (p=0.000 n=24+30)
Gunzip                 81.0MB/s ± 0%  84.6MB/s ± 0%  +4.40%  (p=0.000 n=28+28)
JSONEncode             27.7MB/s ± 0%  27.0MB/s ± 0%  -2.56%  (p=0.000 n=30+29)
JSONDecode             3.28MB/s ± 0%  3.40MB/s ± 0%  +3.45%  (p=0.000 n=30+30)
GoParse                1.49MB/s ± 0%  1.47MB/s ± 0%  -1.34%  (p=0.000 n=30+30)
RegexpMatchEasy0_32    49.0MB/s ± 0%  47.7MB/s ± 0%  -2.51%  (p=0.000 n=28+28)
RegexpMatchEasy0_1K     179MB/s ± 1%   189MB/s ± 0%  +5.87%  (p=0.000 n=28+29)
RegexpMatchEasy1_32    55.4MB/s ± 0%  58.6MB/s ± 0%  +5.78%  (p=0.000 n=29+28)
RegexpMatchEasy1_1K     180MB/s ± 0%   178MB/s ± 0%  -1.09%  (p=0.000 n=29+29)
RegexpMatchMedium_32   37.3MB/s ± 0%  37.3MB/s ± 0%  -0.15%  (p=0.000 n=27+30)
RegexpMatchMedium_1K   4.92MB/s ± 0%  4.54MB/s ± 0%  -7.72%  (p=0.000 n=30+30)
RegexpMatchHard_32     2.73MB/s ± 0%  2.64MB/s ± 0%  -3.41%  (p=0.000 n=24+30)
RegexpMatchHard_1K     3.01MB/s ± 0%  2.85MB/s ± 0%  -5.22%  (p=0.000 n=30+30)
Revcomp                 109MB/s ± 0%   110MB/s ± 0%  +0.65%  (p=0.000 n=23+27)
Template               2.69MB/s ± 0%  2.68MB/s ± 0%  -0.19%  (p=0.000 n=30+30)
[Geo mean]             17.9MB/s       17.8MB/s       -0.57%
@beenshi

This comment has been minimized.

Copy link
Author

commented Aug 21, 2019

any better way to check if my optimzation on WASM works as expected ?

@bcmills

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

Run the go1 benchmark twice, with the same go version.

On what kind of machine, under what conditions? (Did you use perflock first? What else was running on the machine at the time?)

CC @mvdan (for his related recent talks on benchmarking), @dr2chase (general interest in benchmarking), @neelance (WASM)

@bcmills bcmills changed the title The go1 benchmark is noisy on node/wasm test/bench/go1: benchmark is noisy on node/wasm Aug 21, 2019

@bcmills bcmills added the Arch-Wasm label Aug 21, 2019

@bcmills bcmills added this to the Unplanned milestone Aug 21, 2019

@mvdan

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

On top of what @bcmills said: Is the machine idle? Have you tried randomizing the order of the benchmark executions? Is the machine throttling during any of the benchmarks?

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

I'm not sure what we can do on the Go side. The WASM bytecode is run by V8 (or any other execution engine), which has a complex JIT system.

@beenshi

This comment has been minimized.

Copy link
Author

commented Aug 22, 2019

my go1 test condition:

  1. a single desktop machine, with quad-core i5 @ 2.6GHz, ubuntu 18.04 (and its default GUI).
  2. the benchmark test ran from 8:00 PM to 6:00 AM, for 8 rounds, without any other active program running simultaneously, and the GUI was shutdown by the screen saver.
  3. idle or throttling should not happen, during the benchmark's running.
  4. no other special setting.

Actually I use this environment for go1's AMD64/i386 benchmark tests, whose results are OK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.