x/telemetry/internal/counter: persistent test failures with "panic: runtime error: invalid memory address or nil pointer dereference" on GOOS=android GOARCH={386,amd64} via Android emulator builders

[dmitshur]

#!watchflakes
post <- goos == "android" && pkg == "golang.org/x/telemetry/internal/counter" && `panic: runtime error: invalid memory address`

The android-386-emu and android-amd64-emu builders are persistently failing on TestBasic in this package. Example failure:

--- FAIL: TestBasic (0.00s)
    counter_test.go:31: GOOS android GARCH amd64
    counter_test.go:40: counter: disabled by GOTELEMETRY=off
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7a7d97f5c2fb]

goroutine 19 [running]:
testing.tRunner.func1.2({0x7a7d97fac680, 0x7a7d9809bb70})
	/workdir/go/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
	/workdir/go/src/testing/testing.go:1548 +0x397
panic({0x7a7d97fac680?, 0x7a7d9809bb70?})
	/workdir/go/src/runtime/panic.go:914 +0x21f
golang.org/x/telemetry/internal/counter.close(0x7a7d97f1230a?)
	/workdir/gopath/src/golang.org/x/telemetry/internal/counter/counter_test.go:73 +0x1b
runtime.Goexit()
	/workdir/go/src/runtime/panic.go:523 +0x145
testing.(*common).FailNow(0xc000086b60)
	/workdir/go/src/testing/testing.go:999 +0x4a
testing.(*common).Fatal(0xc000086b60, {0xc00004af20?, 0xf?, 0xc000036730?})
	/workdir/go/src/testing/testing.go:1076 +0x54
golang.org/x/telemetry/internal/counter.TestBasic(0xc000086b60)
	/workdir/gopath/src/golang.org/x/telemetry/internal/counter/counter_test.go:40 +0x1bc
testing.tRunner(0xc000086b60, 0x7a7d97fcd6e0)
	/workdir/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/workdir/go/src/testing/testing.go:1648 +0x3ad
FAIL	golang.org/x/telemetry/internal/counter	74.161s

See https://build.golang.org/log/cc69cc12aae4e352100a3ec136d1eb8443ea7b9f for an example log. More are visible at https://build.golang.org/?repo=golang.org%2fx%2ftelemetry.

CC @golang/android as port owners.

If package owners (CC @jamalc, @pjweinb, @hyangah) decide this repository isn't meant to support GOOS=android and doesn't need to be tested on this builder, the right fix is to adjust the test policy in x/build/dashboard. I'm happy to help with that if that's the path to resolve this issue. Thanks.

[gopherbot]

Change https://go.dev/cl/505580 mentions this issue: internal/counter: skip tests on solaris, android, and GOARCH=386

[gopherbot]

Change https://go.dev/cl/505720 mentions this issue: internal/counter: avoid nil check panic

[hyangah]

:-) this revealed a bug in the test code.

But the root cause of this failure is that the telemetry package couldn't create a telemetry collection directory in the android builder environment, so telemetry was disabled. https://github.com/golang/telemetry/blob/b6b6ba5c0fe03deba73c9244d30c283757992cfa/types.go#L82

The failed test assumed telemetry was enabled and eventually hit the bug in the test.

From an old log I see "mkdir /.config: read-only file system" error. The telemetry code attempts to create telemetry data under os.UserConfigDir and on android, it must be a read-only directory.

• Is os.UserConfigDir a right place to place collected telemetry data?

Since some code will end up in the go main project, I think it's better to keep running on android and skip individual tests that don't make sense, as we discover.

[bcmills]

The telemetry code attempts to create telemetry data under os.UserConfigDir and on android, it must be a read-only directory.

Oof. Compare:

• x/tools/gopls/internal/lsp/filecache: failures on android with read-only file system as of CL 443099 #57630
• os: UserCacheDir and UserConfigDir have no tests #57638

But also: it seems like a mistake for the telemetry tests to write to the user's shared telemetry cache — I would assume that we don't want the tests to contaminate the real telemetry-collection backend.

That sort of use case is exactly what the new testing.Testing() function is for — perhaps the telemetry package should use that to disable use of the default location?

[hyangah]

But also: it seems like a mistake for the telemetry tests to write to the user's shared telemetry cache — I would assume that we don't want the tests to contaminate the real telemetry-collection backend.

cc @jamalc @pjweinb

[gopherbot]

Change https://go.dev/cl/506917 mentions this issue: all: delay creation of the telemetry dirs until counter.Open

[gopherbot]

Change https://go.dev/cl/506916 mentions this issue: internal/telemetry: centralize telemetry settings (directories and mode)