It seems that some versions of Go's otherwise excellent testing infrastructure are susceptible to certain "contamination effects" in which the presence (or absence) of test cases influences the performance measured by benchmarks. Check it out:
$ benchstat without.txt with.txt
name old time/op new time/op delta
PushFrontQueue-2 85.5µs ± 1% 97.7µs ± 1% +14.31% (p=0.000 n=10+10)
PushFrontList-2 159µs ± 0% 167µs ± 1% +4.93% (p=0.000 n=9+10)
PushBackQueue-2 85.9µs ± 1% 98.3µs ± 1% +14.38% (p=0.000 n=10+10)
PushBackList-2 159µs ± 0% 166µs ± 1% +4.91% (p=0.000 n=9+9)
PushBackChannel-2 117µs ± 1% 146µs ± 2% +25.01% (p=0.000 n=10+10)
RandomQueue-2 153µs ± 0% 174µs ± 3% +13.69% (p=0.000 n=9+9)
RandomList-2 284µs ± 1% 297µs ± 1% +4.68% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
PushFrontQueue-2 40.9kB ± 0% 40.9kB ± 0% ~ (all equal)
PushFrontList-2 57.4kB ± 0% 57.4kB ± 0% ~ (all equal)
PushBackQueue-2 40.9kB ± 0% 40.9kB ± 0% ~ (all equal)
PushBackList-2 57.4kB ± 0% 57.4kB ± 0% ~ (all equal)
PushBackChannel-2 24.7kB ± 0% 24.7kB ± 0% ~ (all equal)
RandomQueue-2 45.7kB ± 0% 45.7kB ± 0% ~ (all equal)
RandomList-2 90.8kB ± 0% 90.8kB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
PushFrontQueue-2 1.03k ± 0% 1.03k ± 0% ~ (all equal)
PushFrontList-2 2.05k ± 0% 2.05k ± 0% ~ (all equal)
PushBackQueue-2 1.03k ± 0% 1.03k ± 0% ~ (all equal)
PushBackList-2 2.05k ± 0% 2.05k ± 0% ~ (all equal)
PushBackChannel-2 1.03k ± 0% 1.03k ± 0% ~ (all equal)
RandomQueue-2 1.63k ± 0% 1.63k ± 0% ~ (all equal)
RandomList-2 3.24k ± 0% 3.24k ± 0% ~ (all equal)
$ go version
go version go1.7.5 linux/amd64
The only difference between old time and new time is that the
latter had two additional test cases in queue_test.go.
The presence of those test cases makes all benchmarks appear worse
for some reason.
If I comment them out and run the exact same benchmarks again,
everything is "back to normal" as it were.
For now I am reporting the worse results in the official README just
to be conservative, but I believe something needs to be fixed on the Go
side of things.
Update: Something was apparently fixed in a more recent version of Go. Check it out:
$ benchstat without.txt with.txt
name old time/op new time/op delta
PushFrontQueue-2 82.7µs ± 1% 82.8µs ± 1% ~ (p=0.832 n=19+16)
PushFrontList-2 161µs ± 1% 162µs ± 1% +0.48% (p=0.003 n=19+18)
PushBackQueue-2 83.3µs ± 1% 83.4µs ± 1% ~ (p=0.756 n=19+16)
PushBackList-2 156µs ± 1% 158µs ± 3% +1.03% (p=0.024 n=19+20)
PushBackChannel-2 110µs ± 2% 110µs ± 2% ~ (p=0.341 n=20+20)
RandomQueue-2 158µs ± 4% 161µs ± 2% +2.34% (p=0.000 n=20+20)
RandomList-2 279µs ± 1% 281µs ± 4% ~ (p=0.223 n=19+19)
GrowShrinkQueue-2 111µs ± 2% 110µs ± 1% -1.39% (p=0.000 n=18+19)
GrowShrinkList-2 168µs ± 1% 170µs ± 5% +1.21% (p=0.017 n=19+20)
name old alloc/op new alloc/op delta
PushFrontQueue-2 40.9kB ± 0% 40.9kB ± 0% ~ (all equal)
PushFrontList-2 57.4kB ± 0% 57.4kB ± 0% ~ (all equal)
PushBackQueue-2 40.9kB ± 0% 40.9kB ± 0% ~ (all equal)
PushBackList-2 57.4kB ± 0% 57.4kB ± 0% ~ (all equal)
PushBackChannel-2 24.7kB ± 0% 24.7kB ± 0% ~ (all equal)
RandomQueue-2 45.7kB ± 0% 45.7kB ± 0% ~ (all equal)
RandomList-2 90.8kB ± 0% 90.8kB ± 0% ~ (all equal)
GrowShrinkQueue-2 57.2kB ± 0% 57.2kB ± 0% ~ (all equal)
GrowShrinkList-2 57.4kB ± 0% 57.4kB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
PushFrontQueue-2 1.03k ± 0% 1.03k ± 0% ~ (all equal)
PushFrontList-2 2.05k ± 0% 2.05k ± 0% ~ (all equal)
PushBackQueue-2 1.03k ± 0% 1.03k ± 0% ~ (all equal)
PushBackList-2 2.05k ± 0% 2.05k ± 0% ~ (all equal)
PushBackChannel-2 1.03k ± 0% 1.03k ± 0% ~ (all equal)
RandomQueue-2 1.63k ± 0% 1.63k ± 0% ~ (all equal)
RandomList-2 3.24k ± 0% 3.24k ± 0% ~ (all equal)
GrowShrinkQueue-2 1.04k ± 0% 1.04k ± 0% ~ (all equal)
GrowShrinkList-2 2.05k ± 0% 2.05k ± 0% ~ (all equal)
$ go version
go version go1.8.1 linux/amd64
The additional test cases still have some effect on the benchmarks, but it's nowhere near as extreme as it was before. I am not perfectly happy with this, but I am happy enough.