Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test/bench/go1: benchmark is noisy on node/wasm #33749

Closed
beenshi opened this issue Aug 21, 2019 · 9 comments
Closed

test/bench/go1: benchmark is noisy on node/wasm #33749

beenshi opened this issue Aug 21, 2019 · 9 comments

Comments

@beenshi
Copy link

@beenshi beenshi commented Aug 21, 2019

Run the go1 benchmark twice, with the same go version.

(with node 8.16.0 and the recent master branch of go)

name                   old time/op    new time/op    delta
BinaryTree17              25.7s ± 0%     25.7s ± 0%  +0.13%  (p=0.000 n=25+29)
Fannkuch11                13.6s ± 0%     13.9s ± 0%  +1.71%  (p=0.000 n=29+30)
FmtFprintfEmpty           273ns ± 0%     272ns ± 0%  -0.37%  (p=0.000 n=25+30)
FmtFprintfString          565ns ± 1%     554ns ± 0%  -1.91%  (p=0.000 n=30+20)
FmtFprintfInt             716ns ± 1%     674ns ± 0%  -5.88%  (p=0.000 n=30+30)
FmtFprintfIntInt         1.24µs ± 0%    1.16µs ± 0%  -6.35%  (p=0.000 n=30+29)
FmtFprintfPrefixedInt    1.19µs ± 1%    1.11µs ± 0%  -6.89%  (p=0.000 n=30+30)
FmtFprintfFloat          2.38µs ± 0%    2.56µs ± 0%  +7.70%  (p=0.000 n=30+30)
FmtManyArgs              5.77µs ± 0%    5.69µs ± 0%  -1.39%  (p=0.000 n=30+29)
GobDecode                65.8ms ± 0%    67.5ms ± 0%  +2.57%  (p=0.000 n=30+29)
GobEncode                33.5ms ± 0%    34.5ms ± 0%  +2.91%  (p=0.000 n=30+30)
Gzip                      1.09s ± 0%     1.08s ± 0%  -0.81%  (p=0.000 n=28+28)
Gunzip                    239ms ± 0%     229ms ± 0%  -4.21%  (p=0.000 n=27+29)
HTTPClientServer          323µs ± 0%     327µs ± 0%  +1.22%  (p=0.000 n=29+28)
JSONEncode               70.0ms ± 0%    71.9ms ± 0%  +2.63%  (p=0.000 n=30+29)
JSONDecode                591ms ± 0%     572ms ± 0%  -3.30%  (p=0.000 n=30+29)
Mandelbrot200            5.79ms ± 0%    5.79ms ± 0%  +0.01%  (p=0.000 n=25+25)
GoParse                  38.8ms ± 0%    39.5ms ± 0%  +1.69%  (p=0.000 n=29+29)
RegexpMatchEasy0_32       654ns ± 0%     670ns ± 0%  +2.52%  (p=0.000 n=29+25)
RegexpMatchEasy0_1K      5.72µs ± 1%    5.41µs ± 0%  -5.55%  (p=0.000 n=28+29)
RegexpMatchEasy1_32       578ns ± 0%     546ns ± 0%  -5.46%  (p=0.000 n=24+28)
RegexpMatchEasy1_1K      5.68µs ± 0%    5.74µs ± 0%  +1.11%  (p=0.000 n=29+29)
RegexpMatchMedium_32     26.8ns ± 0%    26.8ns ± 0%    ~     (all equal)
RegexpMatchMedium_1K      208µs ± 0%     225µs ± 0%  +8.36%  (p=0.000 n=28+25)
RegexpMatchHard_32       11.7µs ± 0%    12.1µs ± 0%  +3.32%  (p=0.000 n=30+30)
RegexpMatchHard_1K        341µs ± 0%     360µs ± 0%  +5.57%  (p=0.000 n=30+29)
Revcomp                   2.32s ± 0%     2.31s ± 0%  -0.65%  (p=0.000 n=23+26)
Template                  723ms ± 0%     724ms ± 0%  +0.16%  (p=0.000 n=30+30)
TimeParse                3.59µs ± 0%    3.66µs ± 0%  +1.96%  (p=0.000 n=29+29)
TimeFormat               3.93µs ± 1%    3.74µs ± 0%  -4.74%  (p=0.000 n=30+30)
[Geo mean]                340µs          339µs       -0.21%
	
name                   old speed      new speed      delta
GobDecode              11.7MB/s ± 0%  11.4MB/s ± 0%  -2.51%  (p=0.000 n=30+29)
GobEncode              22.9MB/s ± 0%  22.2MB/s ± 0%  -2.82%  (p=0.000 n=30+29)
Gzip                   17.8MB/s ± 0%  17.9MB/s ± 0%  +0.83%  (p=0.000 n=24+30)
Gunzip                 81.0MB/s ± 0%  84.6MB/s ± 0%  +4.40%  (p=0.000 n=28+28)
JSONEncode             27.7MB/s ± 0%  27.0MB/s ± 0%  -2.56%  (p=0.000 n=30+29)
JSONDecode             3.28MB/s ± 0%  3.40MB/s ± 0%  +3.45%  (p=0.000 n=30+30)
GoParse                1.49MB/s ± 0%  1.47MB/s ± 0%  -1.34%  (p=0.000 n=30+30)
RegexpMatchEasy0_32    49.0MB/s ± 0%  47.7MB/s ± 0%  -2.51%  (p=0.000 n=28+28)
RegexpMatchEasy0_1K     179MB/s ± 1%   189MB/s ± 0%  +5.87%  (p=0.000 n=28+29)
RegexpMatchEasy1_32    55.4MB/s ± 0%  58.6MB/s ± 0%  +5.78%  (p=0.000 n=29+28)
RegexpMatchEasy1_1K     180MB/s ± 0%   178MB/s ± 0%  -1.09%  (p=0.000 n=29+29)
RegexpMatchMedium_32   37.3MB/s ± 0%  37.3MB/s ± 0%  -0.15%  (p=0.000 n=27+30)
RegexpMatchMedium_1K   4.92MB/s ± 0%  4.54MB/s ± 0%  -7.72%  (p=0.000 n=30+30)
RegexpMatchHard_32     2.73MB/s ± 0%  2.64MB/s ± 0%  -3.41%  (p=0.000 n=24+30)
RegexpMatchHard_1K     3.01MB/s ± 0%  2.85MB/s ± 0%  -5.22%  (p=0.000 n=30+30)
Revcomp                 109MB/s ± 0%   110MB/s ± 0%  +0.65%  (p=0.000 n=23+27)
Template               2.69MB/s ± 0%  2.68MB/s ± 0%  -0.19%  (p=0.000 n=30+30)
[Geo mean]             17.9MB/s       17.8MB/s       -0.57%
@beenshi
Copy link
Author

@beenshi beenshi commented Aug 21, 2019

any better way to check if my optimzation on WASM works as expected ?

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 21, 2019

Run the go1 benchmark twice, with the same go version.

On what kind of machine, under what conditions? (Did you use perflock first? What else was running on the machine at the time?)

CC @mvdan (for his related recent talks on benchmarking), @dr2chase (general interest in benchmarking), @neelance (WASM)

@bcmills bcmills changed the title The go1 benchmark is noisy on node/wasm test/bench/go1: benchmark is noisy on node/wasm Aug 21, 2019
@bcmills bcmills added the Arch-Wasm label Aug 21, 2019
@bcmills bcmills added this to the Unplanned milestone Aug 21, 2019
@mvdan
Copy link
Member

@mvdan mvdan commented Aug 21, 2019

On top of what @bcmills said: Is the machine idle? Have you tried randomizing the order of the benchmark executions? Is the machine throttling during any of the benchmarks?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 21, 2019

I'm not sure what we can do on the Go side. The WASM bytecode is run by V8 (or any other execution engine), which has a complex JIT system.

@beenshi
Copy link
Author

@beenshi beenshi commented Aug 22, 2019

my go1 test condition:

  1. a single desktop machine, with quad-core i5 @ 2.6GHz, ubuntu 18.04 (and its default GUI).
  2. the benchmark test ran from 8:00 PM to 6:00 AM, for 8 rounds, without any other active program running simultaneously, and the GUI was shutdown by the screen saver.
  3. idle or throttling should not happen, during the benchmark's running.
  4. no other special setting.

Actually I use this environment for go1's AMD64/i386 benchmark tests, whose results are OK.

@agnivade
Copy link
Contributor

@agnivade agnivade commented Sep 21, 2019

I agree that there isn't much that can be done on the Go side. The entire execution is run by V8 which can optimize or de-optimize code at will. We can run using perflock on a quiet machine and hope for the best. Perhaps also run with Node 10 to check if things are any better.

@beenshi - Did you want anything else to be done on this issue ?

@beenshi
Copy link
Author

@beenshi beenshi commented Sep 25, 2019

Have you tried the way you suggested? Or even the result can be shown? Though we can do less on the go side, I expect to see a solid way to verify WASM backend's performance.

@agnivade
Copy link
Contributor

@agnivade agnivade commented May 8, 2020

I haven't tried with perflock, but on some optimization CLs, I have indeed noticed performance improvements with the Go1 benchmarks. Although I confess I haven't tried running the test multiple times with the same Go version. But what I was trying to say is that, personally, I don't see an actionable item that can be done in the code to address this.

@neelance
Copy link
Member

@neelance neelance commented Oct 28, 2020

This does not seem to be actionable. I'm closing it for now. Maybe in the future some non-V8 runtime will give more stable results.

@neelance neelance closed this Oct 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.