Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: single benchmark starts multiple time #23398

Closed
jeffreystoke opened this issue Jan 10, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@jeffreystoke
Copy link

commented Jan 10, 2018

What version of Go are you using (go version)?

go version go1.10beta1 darwin/amd64

Does this issue reproduce with the latest release?

yes with go1.9.2

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jeffrey/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jeffrey/GoSpace:/Users/jeffrey/GolangProjects"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/r9/zplfsltx4mx8yf05hrx0dnwm0000gp/T/go-build516212567=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I was trying to make a benchmark with my mqtt client lib (libmqtt), using go test as

go test -v -bench . -benchmem -count=1

And here is the code

func BenchmarkLibmqttClient(b *testing.B) {
	b.N = 1
	b.ReportAllocs()
	var client lib.Client
	var err error

	b.Log("init client")
	if client, err = lib.NewClient(
		lib.WithServer("localhost:1883"),
		lib.WithKeepalive(3600, 1.2),
		lib.WithCleanSession(true)); err != nil {
		b.Error(err)
	}

	client.HandleUnSub(func(topic []string, err error) {
		if err != nil {
			b.Error(err)
		}
                // destroy client after  unsub success
		client.Destroy(true)
	})

	client.Connect(func(server string, code lib.ConnAckCode, err error) {
		if err != nil {
			b.Error(err)
		} else if code != lib.ConnAccepted {
			b.Error(code)
		}

		b.ResetTimer()
		for i := 0; i < b.N; i++ {
			client.Publish(&lib.PublishPacket{
				TopicName: testTopic,
				Qos:       testQos,
				Payload:   testTopicMsg,
			})
		}
		client.UnSubscribe(testTopic)
	})
	client.Wait()
}

Once added the client.Destroy(true) call (which just cancel the context of the client, so all workers can return and client.Wait() will return) will result in multiple restart

What did you expect to see?

benchmark starts and finishes both only one time

What did you see instead?

benchmark starts multiple times and finishes only one time

Here is the output for the benchmark

go test -v -bench . -benchmem -count=1
--- BENCH: BenchmarkLibmqttClient-4
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        client_test.go:53: init client
        ... [output truncated]
PASS
ok      github.com/goiiot/libmqtt/benchmark     0.173s
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 10, 2018

This is expected, the testing package runs your benchmark multiple times with increasing values of b.N.

http://talks.godoc.org/github.com/davecheney/high-performance-go-workshop/high-performance-go-workshop.slide#21

Closing as there is no bug to be found here.

@davecheney davecheney closed this Jan 10, 2018

@jeffreystoke

This comment has been minimized.

Copy link
Author

commented Jan 10, 2018

@davecheney thanks for your nice slides.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 10, 2018

@mikioh mikioh changed the title single benchmark starts multiple time testing: single benchmark starts multiple time Jan 14, 2018

@golang golang locked and limited conversation to collaborators Jan 14, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.