cmd/go: go test -cover fails when loading plugin built without coverage

[pfi79]

Go version

go version go1.22.3 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/runner/.cache/go-build'
GOENV='/home/runner/.config/go/env'
GOEXE=''
GOEXPERIMENT='nocoverageredesign'
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/opt/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/opt/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/hostedtoolcache/go/1.22.3/x64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/hostedtoolcache/go/1.22.3/x64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/runner/work/fabric/fabric/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build684709877=/tmp/go-build -gno-record-gcc-switches'

What did you do?

https://github.com/pfi79/fabric/tree/up-go-1.22-error
I haven't been able to do a separate test yet.

go test -v -cover github.com/hyperledger/fabric/core/handlers/endorsement/api github.com/hyperledger/fabric/core/handlers/library

?       github.com/hyperledger/fabric/core/handlers/endorsement/api     [no test files]
2024-05-16 14:41:01.117 MSK 0001 PANI [core.handlers] loadPlugin -> Error opening plugin at path /var/folders/6v/kmxqr0yn61588pbz646cfdyc0000gq/T/TestEndorsementPlugin4080854438/001/endorsementplugin.so: plugin.Open("/var/folders/6v/kmxqr0yn61588pbz646cfdyc0000gq/T/TestEndorsementPlugin4080854438/001/endorsementplugin"): plugin was built with a different version of package github.com/hyperledger/fabric/core/handlers/endorsement/api
--- FAIL: TestEndorsementPlugin (3.03s)
panic: Error opening plugin at path /var/folders/6v/kmxqr0yn61588pbz646cfdyc0000gq/T/TestEndorsementPlugin4080854438/001/endorsementplugin.so: plugin.Open("/var/folders/6v/kmxqr0yn61588pbz646cfdyc0000gq/T/TestEndorsementPlugin4080854438/001/endorsementplugin"): plugin was built with a different version of package github.com/hyperledger/fabric/core/handlers/endorsement/api [recovered]
        panic: Error opening plugin at path /var/folders/6v/kmxqr0yn61588pbz646cfdyc0000gq/T/TestEndorsementPlugin4080854438/001/endorsementplugin.so: plugin.Open("/var/folders/6v/kmxqr0yn61588pbz646cfdyc0000gq/T/TestEndorsementPlugin4080854438/001/endorsementplugin"): plugin was built with a different version of package github.com/hyperledger/fabric/core/handlers/endorsement/api

goroutine 14 [running]:
testing.tRunner.func1.2({0x10bdafc80, 0xc0003d48b0})
        /usr/local/go/src/testing/testing.go:1631 +0x24a
testing.tRunner.func1()
        /usr/local/go/src/testing/testing.go:1634 +0x377
panic({0x10bdafc80?, 0xc0003d48b0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x0?, 0x0?, {0x0?, 0x0?, 0xc00050a260?})
        /Users/fedor/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:196 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000396750, {0x0, 0x0, 0x0})
        /Users/fedor/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:262 +0x24e
go.uber.org/zap.(*SugaredLogger).log(0xc00048c488, 0x4, {0xc0002a8300?, 0x0?}, {0x0?, 0xc000506750?, 0xc00008be10?}, {0x0, 0x0, 0x0})
        /Users/fedor/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:316 +0xec
go.uber.org/zap.(*SugaredLogger).Panicf(...)
        /Users/fedor/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:202
github.com/hyperledger/fabric-lib-go/common/flogging.(*FabricLogger).Panicf(...)
        /Users/fedor/go/src/github.com/hyperledger/fabric/vendor/github.com/hyperledger/fabric-lib-go/common/flogging/zap.go:74
github.com/hyperledger/fabric/core/handlers/library.(*registry).loadPlugin(0xc00004dec0, {0xc0003981c0, 0x69}, 0x2, {0xc00004df40, 0x1, 0x1})
        /Users/fedor/go/src/github.com/hyperledger/fabric/core/handlers/library/plugin.go:30 +0x245
github.com/hyperledger/fabric/core/handlers/library.TestEndorsementPlugin(0xc0000cc4e0)
        /Users/fedor/go/src/github.com/hyperledger/fabric/core/handlers/library/registry_plugin_test.go:100 +0x172
testing.tRunner(0xc0000cc4e0, 0x10bf6d950)
        /usr/local/go/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
        /usr/local/go/src/testing/testing.go:1742 +0x390
FAIL    github.com/hyperledger/fabric/core/handlers/library     9.970s
FAIL

What did you see happen?

If you remove the cover flag or remove the github.com/hyperledger/fabric/core/handlers/endorsement/api package or put another package without tests or go 1.21 or GOEXPERIMENT=nocoverageredesign everything works fine.

I'll add an observation:

• There is a package A, it has 1 file and it has one interface. When this package gets into testing, it seems that go adds some data, for example covervars.go.
• There is a folder testdata, in it an example plugin that depends on package A
• There is package B, it has a test file in it. Package B also depends on package A. The command to build a plugin from testdata is executed in the test (go build -o t.so -buildmode=plugin). After that it tries to open it.

This is where the error occurs. Perhaps, I think it is because package A has a covervars.go file in the test. But when compiling the plugin, it does not.

What did you expect to see?

I want the test to pass and not fail.

[dmitshur]

CC @thanm.

[pfi79]

I'll add an observation:

• There is a package A, it has 1 file and it has one interface. When this package gets into testing, it seems that go adds some data, for example covervars.go.
• There is a folder testdata, in it an example plugin that depends on package A
• There is package B, it has a test file in it. Package B also depends on package A. The command to build a plugin from testdata is executed in the test (go build -o t.so -buildmode=plugin). After that it tries to open it.

This is where the error occurs. Perhaps, I think it is because package A has a covervars.go file in the test. But when compiling the plugin, it does not.

[thanm]

Thanks for the report.

This problem is no longer reproducible on tip (in Go 1.23); I am not precisely sure which patch is responsible, but it does appear to no longer happening. There are a number of changes to the Go command cover machinery that have gone in since 1.22.

In general plugins and coverage are not going to mix well, however, simply because of the way we implement the package consistency check: package build ID is going to be different for a packaeg whose source code has been run through the cover tool as compared with the original sources.

You can see this in action for this Go module, which contains 3 packages "a", "b", and "c". Package a test builds plugin "b", which imports "c". Thus when "a" tries to load "b.so"

https://go.dev/play/p/_yQzBJyY0r_I

$ go test -coverpkg=./... ./...
?   	re.pro/b	[no test files]
?   	re.pro/c	[no test files]
--- FAIL: TestIt (0.46s)
    a_test.go:17: plugin build output: 
    a_test.go:21: could not load plugin b.so: plugin.Open("b"): plugin was built with a different version of package re.pro/c
FAIL
coverage: 0.0% of statements in ./...
FAIL	re.pro/a	0.470s
FAIL

This is effectively equivalent to what you're seeing, and this also happens on tip. Note that "-buildmode=plugin" is not compatible with "-cover", so it is hard to think of ways to work around this.

As you can see from the plugin package docs there are a lot of limitations, and I think this is another one that we can add to the list.

[pfi79]

@thanm If it helps, but it seems to me that it happened because the covervars.go file appeared in the source cache of package “c” and the test was built with it, but the plugin was built without it.

[thanm]

@thanm If it helps, but it seems to me that it happened because the covervars.go file appeared in the source cache of package “c” and the test was built with it, but the plugin was built without it.

I think I agree, but the cover tool does more than just add "covervars.go", it also modifies all of the functions. So not sure what your point is.

[pfi79]

@thanm alas, we have not fixed the problem with version go 1.23.
I tweaked your example a bit to make it similar to our case.
https://go.dev/play/p/x8RrtA-p9l2

go test -coverpkg=./... ./...
        re.pro/b                coverage: 0.0% of statements
        re.pro/c                coverage: 0.0% of statements
--- FAIL: TestIt (1.01s)
    a_test.go:18: plugin build output: 
    a_test.go:22: could not load plugin b.so: plugin.Open("b"): plugin was built with a different version of package re.pro/c
FAIL
coverage: 0.0% of statements in ./...
FAIL    re.pro/a        1.408s
FAIL

[pfi79]

@dmitshur @thanm @seankhliao the error has not been corrected.
Please tell me, will you fix it or do something about it?