Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: TestGdbBacktrace failure on linux-amd64-alpine builder #54352

Open
thanm opened this issue Aug 9, 2022 · 2 comments
Open

runtime: TestGdbBacktrace failure on linux-amd64-alpine builder #54352

thanm opened this issue Aug 9, 2022 · 2 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@thanm
Copy link
Contributor

thanm commented Aug 9, 2022

What version of Go are you using (go version)?

This is with tip.

What operating system and processor architecture are you using (go env)?

linux/amd64 on the linux-amd64-alpine builder

What did you do?

go test -test.v -test.run=TestGdbBacktrace runtime

What did you expect to see?

pass

What did you see instead?

--- FAIL: TestGdbBacktrace (0.93s)
    runtime-gdb_test.go:76: gdb version 11.2
    testenv.go:473: [gdb -nx -batch -iex add-auto-load-safe-path /workdir/go/src/runtime -ex set startup-with-shell off -ex break main.eee -ex run -ex backtrace -ex continue /workdir/tmp/TestGdbBacktrace1177699378/001/a.exe] exit status: exit status 1
    runtime-gdb_test.go:428: gdb output:
        Loading Go Runtime support.
        Breakpoint 1 at 0x456f40: file /workdir/tmp/TestGdbBacktrace1177699378/001/main.go, line 17.
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
        #0  _rt0_amd64_linux () at /workdir/go/src/runtime/rt0_linux_amd64.s:8
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
    runtime-gdb_test.go:443: gdb exited with error: exit status 1

This needs to be looked into -- I am not sure what the issue might be. I did try:

echo 0 > /proc/sys/kernel/yama/ptrace_scope

on the builder, but I am still seeing the failure.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Aug 9, 2022
@thanm thanm added this to the Backlog milestone Aug 9, 2022
@thanm thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 9, 2022
@gopherbot
Copy link

gopherbot commented Aug 9, 2022

Change https://go.dev/cl/422294 mentions this issue: runtime: disable gdb testpoints on alpine pending builder fix

gopherbot pushed a commit that referenced this issue Aug 9, 2022
Disable the TestGdb* testpoints until we can figure out why they are
failing and reconfigure the machine properly.

Updates #54352.

Change-Id: Id9c76a0ba6e23b5deff24f521a1c8e0aafb6481a
Reviewed-on: https://go-review.googlesource.com/c/go/+/422294
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
Disable the TestGdb* testpoints until we can figure out why they are
failing and reconfigure the machine properly.

Updates golang#54352.

Change-Id: Id9c76a0ba6e23b5deff24f521a1c8e0aafb6481a
Reviewed-on: https://go-review.googlesource.com/c/go/+/422294
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
@rsc
Copy link
Contributor

rsc commented Aug 10, 2022

Strace on gdb (without -f to avoid interfering entirely) shows

open("/proc/19662/task/19662/mem", O_RDWR|O_LARGEFILE|O_CLOEXEC) = 11
...
pread64(11, "\270", 1, 4554112)         = 1
pwrite64(11, "\314", 1, 4554112)        = -1 EACCES (Permission denied)

That EACCES is not in the /proc implementation itself. It could have been (and I suspect was) introduced by AppArmor ahead of the actual call into the /proc implementation.

This started with the update from Alpine 3.10 to Alpine 3.16 because Alpine 3.16 includes gdb 11.2. Starting in gdb 11.1 (Sept 2021), gdb uses the /proc/.../task/.../mem file to do short writes to the user process memory, such as when setting breakpoints. Prior to gdb 11.1, large writes used the mem file, but writes smaller than 3 words (the breakpoint is only 1 byte) used ptrace directly.

I suspect all our other builders have gdb prior to 11.1. I confirmed that the linux-amd64 builder has gdb 10.1.

Probably there is a way to change the container configuration to override the AppArmor configuration to allow the writes to proc. I found the 'docker-default' configuration here and it seems to allow writing to /proc/.../task/.../mem.

We don't seem to be doing anything explicitly with AppArmor for our containers in the coordinator. But perhaps the Container-Optimized OS image has a different default AppArmor config.

Leaving the test disabled for now seems fine. Someone who understands Container-Optimized OS and AppArmor will need to relax the permissions.

@mknyszek mknyszek modified the milestones: Backlog, Go1.20 Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: Todo
Development

No branches or pull requests

5 participants