x/build: TestGdbBacktrace fails on Linux builders with gdb version >=11.1 due to AppArmor restrictions on /proc/.../task/.../mem

[thanm]

What version of Go are you using (go version)?

This is with tip.

What operating system and processor architecture are you using (go env)?

linux/amd64 on the linux-amd64-alpine builder

What did you do?

go test -test.v -test.run=TestGdbBacktrace runtime

What did you expect to see?

pass

What did you see instead?

--- FAIL: TestGdbBacktrace (0.93s)
    runtime-gdb_test.go:76: gdb version 11.2
    testenv.go:473: [gdb -nx -batch -iex add-auto-load-safe-path /workdir/go/src/runtime -ex set startup-with-shell off -ex break main.eee -ex run -ex backtrace -ex continue /workdir/tmp/TestGdbBacktrace1177699378/001/a.exe] exit status: exit status 1
    runtime-gdb_test.go:428: gdb output:
        Loading Go Runtime support.
        Breakpoint 1 at 0x456f40: file /workdir/tmp/TestGdbBacktrace1177699378/001/main.go, line 17.
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
        #0  _rt0_amd64_linux () at /workdir/go/src/runtime/rt0_linux_amd64.s:8
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
    runtime-gdb_test.go:443: gdb exited with error: exit status 1

This needs to be looked into -- I am not sure what the issue might be. I did try:

echo 0 > /proc/sys/kernel/yama/ptrace_scope

on the builder, but I am still seeing the failure.

[gopherbot]

Change https://go.dev/cl/422294 mentions this issue: runtime: disable gdb testpoints on alpine pending builder fix

[rsc]

Strace on gdb (without -f to avoid interfering entirely) shows

open("/proc/19662/task/19662/mem", O_RDWR|O_LARGEFILE|O_CLOEXEC) = 11
...
pread64(11, "\270", 1, 4554112)         = 1
pwrite64(11, "\314", 1, 4554112)        = -1 EACCES (Permission denied)

That EACCES is not in the /proc implementation itself. It could have been (and I suspect was) introduced by AppArmor ahead of the actual call into the /proc implementation.

This started with the update from Alpine 3.10 to Alpine 3.16 because Alpine 3.16 includes gdb 11.2. Starting in gdb 11.1 (Sept 2021), gdb uses the /proc/.../task/.../mem file to do short writes to the user process memory, such as when setting breakpoints. Prior to gdb 11.1, large writes used the mem file, but writes smaller than 3 words (the breakpoint is only 1 byte) used ptrace directly.

I suspect all our other builders have gdb prior to 11.1. I confirmed that the linux-amd64 builder has gdb 10.1.

Probably there is a way to change the container configuration to override the AppArmor configuration to allow the writes to proc. I found the 'docker-default' configuration here and it seems to allow writing to /proc/.../task/.../mem.

We don't seem to be doing anything explicitly with AppArmor for our containers in the coordinator. But perhaps the Container-Optimized OS image has a different default AppArmor config.

Leaving the test disabled for now seems fine. Someone who understands Container-Optimized OS and AppArmor will need to relax the permissions.

[heschi]

It would be nice to get this prioritized -- based on Dmitri's experience in #57520, it seems like it's going to bite us every time we try to set up a modern Linux builder.

[mknyszek]

Since this has been punted a couple times already, I'm inclined to move this to Backlog. Though, based on Russ' comment, perhaps this should be in Unreleased? It sounds like the fix here is actually on the x/build side?

Updating the issue title optimistically. Please comment if you disagree. Thanks!