Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: benchmark verbose output changed in 1.14 #37643

Closed
robaho opened this issue Mar 4, 2020 · 10 comments
Closed

testing: benchmark verbose output changed in 1.14 #37643

robaho opened this issue Mar 4, 2020 · 10 comments

Comments

@robaho
Copy link

@robaho robaho commented Mar 4, 2020

Using github.com/robaho/fixed and go bench:

In 1.9.7 results are similar to:

goos: darwin
goarch: amd64
pkg: github.com/robaho/fixed
BenchmarkAddFixed-8         	2000000000	         1.26 ns/op
BenchmarkAddDecimal-8       	 5000000	       322 ns/op
BenchmarkAddBigInt-8        	100000000	        14.5 ns/op
BenchmarkAddBigFloat-8      	20000000	        76.5 ns/op
BenchmarkMulFixed-8         	300000000	         4.24 ns/op
BenchmarkMulDecimal-8       	20000000	        69.5 ns/op
BenchmarkMulBigInt-8        	100000000	        17.0 ns/op
BenchmarkMulBigFloat-8      	50000000	        34.4 ns/op
BenchmarkDivFixed-8         	300000000	         4.59 ns/op
BenchmarkDivDecimal-8       	 2000000	       854 ns/op
BenchmarkDivBigInt-8        	30000000	        54.7 ns/op
BenchmarkDivBigFloat-8      	20000000	       118 ns/op

and in 1.14

goos: darwin
goarch: amd64
pkg: github.com/robaho/fixed
BenchmarkAddFixed
BenchmarkAddFixed-8         	1000000000	         0.519 ns/op
BenchmarkAddDecimal
BenchmarkAddDecimal-8       	 5005356	       236 ns/op
BenchmarkAddBigInt
BenchmarkAddBigInt-8        	82658592	        14.9 ns/op
BenchmarkAddBigFloat
BenchmarkAddBigFloat-8      	15501452	        77.2 ns/op
BenchmarkMulFixed
BenchmarkMulFixed-8         	293514025	         4.02 ns/op
BenchmarkMulDecimal
BenchmarkMulDecimal-8       	16031581	        72.6 ns/op
BenchmarkMulBigInt
BenchmarkMulBigInt-8        	70126652	        17.7 ns/op
BenchmarkMulBigFloat
BenchmarkMulBigFloat-8      	33527037	        35.1 ns/op
BenchmarkDivFixed
BenchmarkDivFixed-8         	265002393	         4.53 ns/op

Notice the extra rows without the PROCs count.

@robaho robaho changed the title go bench output format changed in 14.1 cmd/go: go bench output format changed in 14.1 Mar 4, 2020
@robaho robaho changed the title cmd/go: go bench output format changed in 14.1 cmd/go: go bench output format changed in 1.14 Mar 4, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Mar 4, 2020

Please fill out the complete issue template. What was the exact command? What does go env report?

@bcmills bcmills added wayit WaitingForInfo and removed wayit labels Mar 4, 2020
@robaho
Copy link
Author

@robaho robaho commented Mar 4, 2020

Sorry, I thought it was a trivial report. I run the tests via Intellij so I didn't change any parameters - just the GOROOT for the project.

In 1.9.7 the command is

GOROOT=/usr/local/Cellar/go@1.9/1.9.7/libexec #gosetup
GOPATH=/Users/robertengels/go #gosetup
/usr/local/Cellar/go@1.9/1.9.7/libexec/bin/go test -c -o /private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go github.com/robaho/fixed #gosetup
/private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go -test.v -test.bench "^BenchmarkAddFixed|BenchmarkAddDecimal|BenchmarkAddBigInt|BenchmarkAddBigFloat|BenchmarkMulFixed|BenchmarkMulDecimal|BenchmarkMulBigInt|BenchmarkMulBigFloat|BenchmarkDivFixed|BenchmarkDivDecimal|BenchmarkDivBigInt|BenchmarkDivBigFloat|BenchmarkCmpFixed|BenchmarkCmpDecimal|BenchmarkCmpBigInt|BenchmarkCmpBigFloat|BenchmarkStringFixed|BenchmarkStringNFixed|BenchmarkStringDecimal|BenchmarkStringBigInt|BenchmarkStringBigFloat|BenchmarkWriteTo$" -test.run ^$ #gosetup

in 1.14 the command is

GOROOT=/usr/local/Cellar/go/1.14/libexec #gosetup
GOPATH=/Users/robertengels/go #gosetup
/usr/local/Cellar/go/1.14/libexec/bin/go test -c -o /private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go github.com/robaho/fixed #gosetup
/private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go -test.v -test.bench "^BenchmarkAddFixed|BenchmarkAddDecimal|BenchmarkAddBigInt|BenchmarkAddBigFloat|BenchmarkMulFixed|BenchmarkMulDecimal|BenchmarkMulBigInt|BenchmarkMulBigFloat|BenchmarkDivFixed|BenchmarkDivDecimal|BenchmarkDivBigInt|BenchmarkDivBigFloat|BenchmarkCmpFixed|BenchmarkCmpDecimal|BenchmarkCmpBigInt|BenchmarkCmpBigFloat|BenchmarkStringFixed|BenchmarkStringNFixed|BenchmarkStringDecimal|BenchmarkStringBigInt|BenchmarkStringBigFloat|BenchmarkWriteTo$" -test.run ^$ #gosetup
@robaho
Copy link
Author

@robaho robaho commented Mar 4, 2020

The output of 'go env' is:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/robertengels/Library/Caches/go-build"
GOENV="/Users/robertengels/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/robertengels/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/go-build436672849=/tmp/go-build -gno-record-gcc-switches -fno-common"
@robaho
Copy link
Author

@robaho robaho commented Mar 4, 2020

Interesting too that the 'operations' counts are not whole numbers in 1.14

@mengzhuo
Copy link
Contributor

@mengzhuo mengzhuo commented Mar 4, 2020

You have to remove -v

#24929
a813d3c

@bcmills bcmills changed the title cmd/go: go bench output format changed in 1.14 testing: benchmark output format changed in 1.14 Mar 4, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Mar 4, 2020

@robaho, it seems like the extra lines here are intended to aid in debugging if one of the benchmarks unexpectedly stalls. Other than being different, are they causing any particular harm?

@bcmills bcmills changed the title testing: benchmark output format changed in 1.14 testing: benchmark verbose output changed in 1.14 Mar 4, 2020
@robaho
Copy link
Author

@robaho robaho commented Mar 4, 2020

@mengzhuo I can't - the parameters passed are part of the IntelliJ GoLand/Plugin test runner.

@bcmills not really, but I noticed strange behavior by the test runner in IntelliJ - the progress lines are blank. I have to assume that other CI tools may have problems due to the format change, additionally the non-whole numbers might cause 'regression alerts' in some CI tools.

Just giving a heads-up - not blocking me in any way.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 4, 2020

Thanks for letting us know. Doesn't seem like something we should fix in the Go toolchain at this time.

(FYI @dlsniper; this may be something for you to look into on the IntelliJ side.)

@bcmills bcmills closed this Mar 4, 2020
@bcmills bcmills removed the WaitingForInfo label Mar 4, 2020
@bcmills bcmills added this to the Go1.14.1 milestone Mar 4, 2020
@dlsniper
Copy link
Contributor

@dlsniper dlsniper commented Mar 5, 2020

but I noticed strange behavior by the test runner in IntelliJ - the progress lines are blank.

@robaho I'd love to have more information on this, but I don't think this is the right place. Please file an issue on our tracker at https://youtrack.jetbrains.com/issues/Go or reach out to us via Gophers Slack in the #goland channel (or me personally using the same handle on Gophers Slack or Twitter).

At the moment I'd rather not change anything because I do like the idea of being able to debug stalling benchmarks, which is the reason this was added in the first place.

As for running without -test.v, we can't really do that as it helps us build the UI around the output from go test.

@robaho
Copy link
Author

@robaho robaho commented Mar 7, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.