Error parsing stack

[marshall-mcmullen]

I'm seeing a mysterious issue when I use leaktest in combination with a high -count flag, as in:

go test -run . -failfast -count 100 --tags=unit ./...
    leaktest.go:83: leaktest: error parsing stack: "goro"
FAIL

The reported line number in leaktest.go is always the same. But the exact string it's failing to parse changes from run to run. e.g.

leaktest.go:83: leaktest: error parsing stack: "gorouti"
leaktest.go:83: leaktest: error parsing stack: "goroutine 99673 [select]:"

It seems to happen more predictably when I run with high -count values.

Any ideas what could be causing this?

Here is my go build env:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/marshall/.cache/go-build"
GOENV="/home/marshall/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/marshall/code/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="x86_64-pc-linux-gnu-gcc"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build261227009=/tmp/go-build -gno-record-gcc-switches"

[fortytw2]

Thanks for the report!

double checking what -count does here:

$ go help testflag | grep 'count' -A4
	-count n
	   Run each test and benchmark n times (default 1).
	   If -cpu is set, run n times for each GOMAXPROCS value.
	   Examples are always run once.

I feel like this may be caused by the goroutine count going above 100k, but I can't fathom why that'd be. Can you reproduce this in a very minimal test case, or is it only in your codebase?

[marshall-mcmullen]

@fortytw2 thanks for the reply! I can't imagine my test having more than 100k goroutines... that's crazy. I will keep debugging this and see if I can recreate it more reliably and update this case with any info I find. So far I've only seen this in one of my projects even though I use leaktest in many projects.

[fortytw2]

Awesome! Let me know if you can find a reproduction. Is it closed source code? (If it is, and I can take a look anyways, ping me on gophers slack under this same username - https://invite.slack.golangbridge.org/)

[marshall-mcmullen]

Well, I have been unable to recreate this outside our proprietary code base. What I did was lower our integration test suite from -count 100 to -count 50 and this problem has stopped happening for us. Not the best solution, but it's good enough for now so that I'm not blocked by this.

Feel free to close this since I am unable to recreate this as a standalone test outside our code base... your call.

Thanks!