TestFailedStartupExitCode fails during make test-short

[knweiss]

Bug Report

What did you do?
I've tried to compile the release-2.6.0branch using go1.11.4 and noticed a make test-short failure:

What did you expect to see?
All test should pass.

What did you see instead? Under which circumstances?

TestFailedStartupExitCode fails during make test-short. All other tests pass.

$ make test-short
>> running short tests
GO111MODULE=on go test -short  -mod=vendor ./...
--- FAIL: TestFailedStartupExitCode (0.00s)
    main_test.go:173: unable to retrieve the exit status for prometheus: fork/exec : no such file or directory
FAIL
FAIL    github.com/prometheus/prometheus/cmd/prometheus 0.017s
...

AFAICS the root cause is that promPath is empty as it is not set in TestMain() for short tests:

prometheus/cmd/prometheus/main_test.go

Lines 34 to 43 in dbd1d58

	func TestMain(m *testing.M) {
	flag.Parse()
	if testing.Short() {
	os.Exit(m.Run())
	}
	// On linux with a global proxy the tests will fail as the go client(http,grpc) tries to connect through the proxy.
	os.Setenv("no_proxy", "localhost,127.0.0.1,0.0.0.0,:")
	var err error
	promPath, err = os.Getwd()

prometheus/cmd/prometheus/main_test.go

Lines 160 to 175 in dbd1d58

	// Let's provide an invalid configuration file and verify the exit status indicates the error.
	func TestFailedStartupExitCode(t *testing.T) {
	fakeInputFile := "fake-input-file"
	expectedExitStatus := 1
	prom := exec.Command(promPath, "--config.file="+fakeInputFile)
	err := prom.Run()
	testutil.NotOk(t, err, "")
	if exitError, ok := err.(*exec.ExitError); ok {
	status := exitError.Sys().(syscall.WaitStatus)
	testutil.Equals(t, expectedExitStatus, status.ExitStatus())
	} else {
	t.Errorf("unable to retrieve the exit status for prometheus: %v", err)
	}
	}

Environment

• System information:

$ cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
$ uname -srm
Linux 3.10.0-862.14.4.el7.x86_64 x86_64

• Prometheus version:

$ ./prometheus --version
prometheus, version 2.6.0 (branch: release-2.6, revision: dbd1d58c894775c0788470944b818cc724f550fb)
  build user:       knweiss@mangan
  build date:       20190104-11:38:56
  go version:       go1.11.4
$ git status
On branch release-2.6
Your branch is up to date with 'origin/release-2.6'.

nothing to commit, working tree clean

[rleungx]

It seems that it always fails since #4296 was merged. May I take this issue?

[codesome]

@rleungx go ahead!

[knweiss]

@rleungx I had this simple fix (short test skip) in mind as well but thought it wouldn't be accepted. ;-) Thanks.