Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: move tests for runtime-gdb.py to misc/ and do not run them during all.bash #39204

Open
bcmills opened this issue May 21, 2020 · 3 comments
Open

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented May 21, 2020

The tests in runtime-gdb_test.go, for the GDB bindings in runtime-gdb.py, have historically been very flaky. They are currently skipped on a large fraction of platforms due to:

If we had an incoming change to add a test with this many skips today, I suspect that we would turn it down: it tests what is supposed to be cross-platform code, but only checks the actual behavior of that code for a narrow subset of users — because that behavior does not work reliably across platforms.

But it doesn't work reliably on the remaining tested platforms either: despite all of those skips, the tests are still flaky today:

#39021 in particular describes a regression during the 1.15 cycle that is interfering with the SlowBots and TryBots on pending CLs, will likely interfere with release testing as well, and so far has resisted attempts at bisection because the test (and its failure mode) is highly nondeterministic.

If we consider the runtime “incorrect” if it does not work with gdb, then we should take the time to fix and maintain the Go bindings properly and portably — not just Skip their tests! — in the same way that we fix and maintain other features of the runtime across platforms. Otherwise, it is not appropriate to run the tests for those bindings as part of the tests for the runtime package, which is a dependency of most Go binaries (and is therefore likely to have its tests run in users' CI systems).

Most of the other Go project tests for integration with third-party tools are in the misc subdirectory. I propose that we move the gdb integration tests there.

I further propose that we should not run these tests as part of all.bash, run.bash, or on the Go project builders until such time as they can be made reliable, with only one Skip based on a check for a gdb executable at a sufficiently recent version.

@gopherbot gopherbot added this to the Proposal milestone May 21, 2020
@gopherbot gopherbot added the Proposal label May 21, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 21, 2020

I don't think this has to be a proposal, in that I don't think it has to go through the proposal committee. This does not affect any user visible API. I think the runtime package maintainers can decide.

@bcmills bcmills changed the title proposal: runtime: move tests for runtime-gdb.py to misc/ and do not run them during all.bash runtime: move tests for runtime-gdb.py to misc/ and do not run them during all.bash May 21, 2020
@bcmills bcmills modified the milestones: Proposal, Go1.15 May 21, 2020
@mundaym
Copy link
Member

@mundaym mundaym commented May 25, 2020

The tests in runtime-gdb_test.go, for the GDB bindings in runtime-gdb.py ...

TestGdbBacktrace doesn't actually load the python script. It just gets GDB to execute a Go program, sets a breakpoint and triggers a backtrace. It is one of the few places where we test that the debug information in a Go executable can be consumed by a debugger like GDB.

I'm not sure what the solution to the flakiness is but I think having basic GDB support is valuable and should be tested on platforms where GDB is available. In particular I think we should continue to run the tests on the builders wherever possible - even if it is just a subset of platforms.

@bcmills
Copy link
Member Author

@bcmills bcmills commented May 26, 2020

@mundaym

I'm not sure what the solution to the flakiness is but I think having basic GDB support is valuable and should be tested on platforms where GDB is available.

Note that we already do not test basic GDB support on most platforms for which GDB is available, due to the bugs mentioned in the first section.¹ If we agree that that support is valuable, then we should remove those skips and fix the behavior on those platforms. If we don't fix the tests when they are broken, what's the point in running them?

Either way, I still don't think these tests belong with the runtime package: they are testing integration with a third-party tool (gdb), not the Go-observable behavior of the runtime package.

¹

func checkGdbEnvironment(t *testing.T) {
testenv.MustHaveGoBuild(t)
switch runtime.GOOS {
case "darwin":
t.Skip("gdb does not work on darwin")
case "netbsd":
t.Skip("gdb does not work with threads on NetBSD; see https://golang.org/issue/22893 and https://gnats.netbsd.org/52548")
case "windows":
t.Skip("gdb tests fail on Windows: https://golang.org/issue/22687")
case "linux":
if runtime.GOARCH == "ppc64" {
t.Skip("skipping gdb tests on linux/ppc64; see https://golang.org/issue/17366")
}
if runtime.GOARCH == "mips" {
t.Skip("skipping gdb tests on linux/mips; see https://golang.org/issue/25939")
}
case "freebsd":
t.Skip("skipping gdb tests on FreeBSD; see https://golang.org/issue/29508")
case "aix":
if testing.Short() {
t.Skip("skipping gdb tests on AIX; see https://golang.org/issue/35710")
}
case "plan9":
t.Skip("there is no gdb on Plan 9")
}
if final := os.Getenv("GOROOT_FINAL"); final != "" && runtime.GOROOT() != final {
t.Skip("gdb test can fail with GOROOT_FINAL pending")
}
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.