Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go/benchmark only takes timer measurement as parameter to increase b.N #40226

Closed
joesonw opened this issue Jul 15, 2020 · 3 comments
Closed

go/benchmark only takes timer measurement as parameter to increase b.N #40226

joesonw opened this issue Jul 15, 2020 · 3 comments

Comments

@joesonw
Copy link

@joesonw joesonw commented Jul 15, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.2 darwin/amd64

Does this issue reproduce with the latest release?

I don't know. AFAIK, this is not a bug, so in theory, YES.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPRIVATE=""
GOTMPDIR=""
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/b4/t4v6zpln4rb16cvg747dz3j80000gn/T/go-build515811079=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm trying to benchmark a buffer library. I shall test write and read separately. But I need to write data to the buffer before read.

func BenchmarkXXX(b *testing.B) {
    buf := NewSomeBuffer()

    b.Run("Write", func (b *testing.B) {
        for i := 0; i < b.N; i++ {
            buf.Write(make([]byte, SOME_SIZE))
        }
    })

    b.Run("Write", func (b *testing.B) {
        for i := 0; i < b.N; i++ {
            buf.Write(make([]byte, SOME_SIZE))
        }

        b.ResetTimer() // Here is what went wrong

        for i := 0; i < b.N; i++ {
            buf.Read(make([]byte, SOME_SIZE))
        }
    })
}

What did you expect to see?

Hence in this scenario, read is much faster that write. It should ran similar amount of samples.

What did you see instead?

But, because b.ResetTimer undid all measurements. When go/benchmark trying to workout b.N values, It only measured the amount of time to read, It ended up timed out because of the enormous amount of data to write. Say 1,000,000 samples when test write alone, and 5,000,000 samples when test read

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 15, 2020

Use b.StopTimer/b.StartTimer.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For asking questions, see:

@davecheney davecheney closed this Jul 15, 2020
@joesonw
Copy link
Author

@joesonw joesonw commented Jul 15, 2020

@davecheney

It produces the same issue. When it predicates the next b.N, it only takes account of the timer run duration, not the actual function run duration.

b.start = time.Now()

b.duration += time.Since(b.start)

go/src/testing/benchmark.go

Lines 296 to 324 in c5d7f2f

d := b.benchTime.d
for n := int64(1); !b.failed && b.duration < d && n < 1e9; {
last := n
// Predict required iterations.
goalns := d.Nanoseconds()
prevIters := int64(b.N)
prevns := b.duration.Nanoseconds()
if prevns <= 0 {
// Round up, to avoid div by zero.
prevns = 1
}
// Order of operations matters.
// For very fast benchmarks, prevIters ~= prevns.
// If you divide first, you get 0 or 1,
// which can hide an order of magnitude in execution time.
// So multiply first, then divide.
n = goalns * prevIters / prevns
// Run more iterations than we think we'll need (1.2x).
n += n / 5
// Don't grow too fast in case we had timing errors previously.
n = min(n, 100*last)
// Be sure to run at least one more than last time.
n = max(n, last+1)
// Don't run more than 1e9 times. (This also keeps n in int range on 32 bit platforms.)
n = min(n, 1e9)
b.runN(int(n))
}
}
b.result = BenchmarkResult{b.N, b.duration, b.bytes, b.netAllocs, b.netBytes, b.extra}

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 15, 2020

Thank you for raising this issue. This doesn't appear to be a bug in Go. We only use the issue tracker for tracking bugs and proposals. Please see the resources I mentioned above for appropriate places to ask questions. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.