net: TCP benchmarks are wildly variable #9774
Comments
What machine and OS do you use? |
OS X 10.10.2, Core i7. |
To measure benchmark stability, you need to disable TuroBoost and SpeedStep in BIOS. Also bind the program to specific set of cores (for GOMAXPROCS=1 to a single core). Also make the system as idle as possible, in particular disable background processes like updaters, antiviruses, etc. On my linux machine the results are reasonably stable: $ nice -20 taskset 16 go test -run=none -bench=BenchmarkTCP4Persistent$ -cpu=1,1,1,1,1,1,1,1 net OneShot is more problematic, because it creates tons of TCP connections and kernel can't keep up. |
In your table you say old ns/op and new ns/op. What is old and what is new. On Thu, Feb 5, 2015 at 3:33 AM, Dmitry Vyukov notifications@github.com
|
@RLH I think it's the same version. Josh just used benchcmp as a way of showing the variance between two runs (that one would hope would be nearly the same). |
@dvyukov thanks for the tips. Most benchmarks are quite stable on my system; the net ones are the only ones that show significant variability. It seems plausible that this is OS X-specific and kernel-related. Any objection to me adding b.Skip to the relevant benchmarks on darwin, since they're so noisy? |
I object to adding t.Skip. It may be appropriate if the benchmark is failing episodically. You can filter these benchmarks with -bench flag or just grep -v afterwards. |
Fair enough. |
The net TCP benchmarks are all over the map. Here are two consecutive runs:
This is in my experience a typical amount of variation. Is there a way to stabilize them? If not, what purpose do they serve?
The text was updated successfully, but these errors were encountered: