New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: -coverpkg=all gives different coverage value when run on a package list vs ./... #23883

Open
AlexRouSg opened this Issue Feb 17, 2018 · 10 comments

Comments

Projects
None yet
7 participants
@AlexRouSg
Contributor

AlexRouSg commented Feb 17, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/chotepud/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="
/.local/go"
GOTMPDIR=""
GOTOOLDIR="~/.local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build941063065=/tmp/go-build -gno-record-gcc-switches"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

package test

import "fmt"

func Foo() {
	fmt.Println("Hello World")
}
package test_test

import (
	"testing"

	"github.com/AlexRouSg/test"
)

func TestFoo(t *testing.T) {
	test.Foo()
}

Run both go test -coverpkg=all and go test -coverpkg=all ./...

What did you expect to see?

Same coverage value for go test -coverpkg=all vs go test -coverpkg=all ./...

What did you see instead?

go test -coverpkg=all ./...

ok github.com/AlexRouSg/test 0.002s coverage: 16.1% of statements in all

go test -coverpkg=all

Hello World
PASS
coverage: 14.8% of statements in all
ok github.com/AlexRouSg/test 0.002s

In actual packages the difference could be as big as go test -coverpkg=all ./... 47.3% vs go test -coverpkg=all 94.4% when comparing the same package.

@ianlancetaylor ianlancetaylor changed the title from cmd/test: -coverpkg=all gives different coverage value when run on a package list vs ./... to cmd/go: -coverpkg=all gives different coverage value when run on a package list vs ./... Feb 17, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 17, 2018

It's not clear to me that it has anything to do with whether you specify ./... or not. If I use go test -c -coverpkg=all to generate a test binary, I see different coverage numbers each time I run it.

One issue is that -coverpkg=all really means all. All packages, including the standard library, are recompiled for coverage. It's not clear that this is a useful or meaningful thing to do. It may not even be possible to get consistent coverage results in this case.

@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Feb 17, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 17, 2018

I'm marking this for 1.11 since as far as I can tell -coverpkg=all in 1.9 tried to recompile everything in GOPATH. At least we're doing better than that.

@AlexRouSg

This comment has been minimized.

Contributor

AlexRouSg commented Feb 17, 2018

@ianlancetaylor
I think I finally figured out what's the deal with the different numbers when I ran test with ./... and without.

To make this easier to explain, I have a test only package containing some non test files to run cgo functions.

When running test without ./... -coverpkg only matches my test package for some reason, which is why it gave 94.4%. However running with ./... includes my other packages that I'm actually interested in, which then gave a lower value of 47.3%.

Modifiying the test case such that func Foo is in a separate folder than the test function should show 0% coverage vs 100% coverage with running test -coverpkg=foopackage/... with ./... and without.

@jdef

This comment has been minimized.

jdef commented Apr 16, 2018

I'm having a related problem where I'm using go test -c ... to generate a test-binary. The test-binary is generated from package ./e2e and this package ONLY contains test code. I've tried generating the binary with -coverpkg set to ./... or all and I consistently get 0% coverage, even though the executed tests leverage tons of other packages in the project. If I specify a fully-qualified package name for coverpkg (for a package other than e2e) then go test -c ... complains that I've specified a package that's not exercised by the test -- which is crazy talk

warning: no packages being tested depend on matches for pattern github.com/myorg/myproject/pkg/...

@AlexRouSg

This comment has been minimized.

Contributor

AlexRouSg commented Apr 16, 2018

@jdef Try running go test -c ./... on the root package or passing a list of all the package you want coverage for. I have a hunch that coverage only checks against packages you're testing and maybe the std.

@jdef

This comment has been minimized.

jdef commented Apr 17, 2018

data point: I created a coverage_imports.go file in my project's e2e/ (test-only) package. it anonymously imports every other package in the project, besides itself. when I pre-compile the e2e/ test binary (via go test -c -coverpkg ./... ./e2e) then the subsequent execution of said binary FINALLY emits a coverage report for the other packages in the project. this feels pretty hacky, but I'm not aware of a better workaround.

@rsc

This comment has been minimized.

Contributor

rsc commented Apr 18, 2018

/cc @bcmills

@founderio

This comment has been minimized.

founderio commented May 29, 2018

Description of my experiences, which led me here: https://groups.google.com/forum/#!topic/golang-nuts/63zlrBhnNw8

My use case: One package with tests I want to run isolated from the other tests but report coverage on all dependent sibling packages.

@maxxgl

This comment has been minimized.

maxxgl commented Jun 13, 2018

run isolated from the other tests but report coverage on all dependent sibling packages.

I have a similar use case. I have it working under windows with:

go test my_test_package -coverpkg "sibling_pkg"

even though it throws the "no packages being tested depend on match for pattern" warning.

I absolutely cannot make it work under Ubuntu 18.04.

@bcmills

This comment has been minimized.

Member

bcmills commented Oct 24, 2018

Likely the same underlying cause as #23910.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment