testing: benchmark ignores time spent while timer is stopped when choosing N

[ITR13]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.1 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\Go\GoPath
set GORACE=
set GOROOT=D:\Go
set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Mikael\AppData\Local\Temp\go-build643448347=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

Benchmark a function that is quick but has a lot of delay while the timer is stopped.
The benchmark's N will grow very large due to the recorded time being short, but due to the delay during the time the timer is stopped, it will last a long amount of time

Here are the methods I tested with:
https://play.golang.org/p/uXHHPKoapbg

What did you expect to see?

Any of the following:

• Time spent while the Timer is stopped to be taken into account when choosing N
  • In this scenario, time spent before calling ResetTimer should not be taken into account
• Time spent while the Timer is stopped to be ignored for the timeout time
  • This would be a bad solution due to not catching infinite loops while the timer is stopped
• Time spent while the Timer is stopped has a separate longer timeout time than the benchmark itself
  • This would still cause problems, just have bigger room for failure

What did you see instead?

Benchmark got killed due to taking too long

[ITR13]

Accidentally used the wrong play.golang.org link, updated now

[ianlancetaylor]

I suppose you could just increase the timeout. None of the suggestions you make seem right to me. The timeout is there to catch problems with the test, and adjusting it as you suggest will cause it to catch fewer problems.

[ianlancetaylor]

I think the answer when this happens is to increase the timeout.