x/build: `TestGdbBacktrace` fails on Linux builders with `gdb` version `>=11.1` due to AppArmor restrictions on `/proc/.../task/.../mem`

[thanm]

What version of Go are you using (go version)?

This is with tip.

What operating system and processor architecture are you using (go env)?

linux/amd64 on the linux-amd64-alpine builder

What did you do?

go test -test.v -test.run=TestGdbBacktrace runtime

What did you expect to see?

pass

What did you see instead?

--- FAIL: TestGdbBacktrace (0.93s)
    runtime-gdb_test.go:76: gdb version 11.2
    testenv.go:473: [gdb -nx -batch -iex add-auto-load-safe-path /workdir/go/src/runtime -ex set startup-with-shell off -ex break main.eee -ex run -ex backtrace -ex continue /workdir/tmp/TestGdbBacktrace1177699378/001/a.exe] exit status: exit status 1
    runtime-gdb_test.go:428: gdb output:
        Loading Go Runtime support.
        Breakpoint 1 at 0x456f40: file /workdir/tmp/TestGdbBacktrace1177699378/001/main.go, line 17.
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
        #0  _rt0_amd64_linux () at /workdir/go/src/runtime/rt0_linux_amd64.s:8
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
    runtime-gdb_test.go:443: gdb exited with error: exit status 1

This needs to be looked into -- I am not sure what the issue might be. I did try:

echo 0 > /proc/sys/kernel/yama/ptrace_scope

on the builder, but I am still seeing the failure.

[gopherbot]

Change https://go.dev/cl/422294 mentions this issue: runtime: disable gdb testpoints on alpine pending builder fix

[rsc]

Strace on gdb (without -f to avoid interfering entirely) shows

open("/proc/19662/task/19662/mem", O_RDWR|O_LARGEFILE|O_CLOEXEC) = 11
...
pread64(11, "\270", 1, 4554112)         = 1
pwrite64(11, "\314", 1, 4554112)        = -1 EACCES (Permission denied)

That EACCES is not in the /proc implementation itself. It could have been (and I suspect was) introduced by AppArmor ahead of the actual call into the /proc implementation.

This started with the update from Alpine 3.10 to Alpine 3.16 because Alpine 3.16 includes gdb 11.2. Starting in gdb 11.1 (Sept 2021), gdb uses the /proc/.../task/.../mem file to do short writes to the user process memory, such as when setting breakpoints. Prior to gdb 11.1, large writes used the mem file, but writes smaller than 3 words (the breakpoint is only 1 byte) used ptrace directly.

I suspect all our other builders have gdb prior to 11.1. I confirmed that the linux-amd64 builder has gdb 10.1.

Probably there is a way to change the container configuration to override the AppArmor configuration to allow the writes to proc. I found the 'docker-default' configuration here and it seems to allow writing to /proc/.../task/.../mem.

We don't seem to be doing anything explicitly with AppArmor for our containers in the coordinator. But perhaps the Container-Optimized OS image has a different default AppArmor config.

Leaving the test disabled for now seems fine. Someone who understands Container-Optimized OS and AppArmor will need to relax the permissions.

[heschi]

It would be nice to get this prioritized -- based on Dmitri's experience in #57520, it seems like it's going to bite us every time we try to set up a modern Linux builder.