Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: benchmark ignores time spent while timer is stopped when choosing N #23998

Closed
ITR13 opened this issue Feb 21, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@ITR13
Copy link

commented Feb 21, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.1 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\Go\GoPath
set GORACE=
set GOROOT=D:\Go
set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Mikael\AppData\Local\Temp\go-build643448347=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

Benchmark a function that is quick but has a lot of delay while the timer is stopped.
The benchmark's N will grow very large due to the recorded time being short, but due to the delay during the time the timer is stopped, it will last a long amount of time

Here are the methods I tested with:
https://play.golang.org/p/uXHHPKoapbg

What did you expect to see?

Any of the following:

  • Time spent while the Timer is stopped to be taken into account when choosing N
    • In this scenario, time spent before calling ResetTimer should not be taken into account
  • Time spent while the Timer is stopped to be ignored for the timeout time
    • This would be a bad solution due to not catching infinite loops while the timer is stopped
  • Time spent while the Timer is stopped has a separate longer timeout time than the benchmark itself
    • This would still cause problems, just have bigger room for failure

What did you see instead?

Benchmark got killed due to taking too long

@ITR13

This comment has been minimized.

Copy link
Author

commented Feb 21, 2018

Accidentally used the wrong play.golang.org link, updated now

@ALTree ALTree changed the title Benchmark ignores time spent while timer is stopped when choosing N testing: benchmark ignores time spent while timer is stopped when choosing N Feb 21, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2018

I suppose you could just increase the timeout. None of the suggestions you make seem right to me. The timeout is there to catch problems with the test, and adjusting it as you suggest will cause it to catch fewer problems.

@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Feb 21, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Jun 29, 2018

I think the answer when this happens is to increase the timeout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.