Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trace/exec: Fix crash when using --cwd #2422

Merged
merged 1 commit into from
Jan 29, 2024

Conversation

mauriciovasquezbernal
Copy link
Member

@mauriciovasquezbernal mauriciovasquezbernal commented Jan 29, 2024

The ebpf code has some optimization to only send the right amount of bytes used to hold the arguments.
(See around L200 in pkg/gadgets/trace/exec/tracer/bpf/execsnoop.bpf.c). bpf2go defines Args as [7680]uint8 and the tracer was using args := bpfEvent.Args, hence copying the array. It an issue since it's trying to copy memory outside the sample received from the perf ring buffer.

Testing

The issue is easy to reproduce on the local machine (at least on mine):

# Something that generates events 
$ docker run --rm -it --name myfoo2 docker.io/library/busybox:latest sh -c 'while true; do cat /dev/null; done;'

$ go build ./cmd/ig/ && bash -c 'for i in {1..30}; do sudo ./ig trace exec --cwd --timeout 2 || exit 1; done'
...

myfoo2                          1862038    35334      cat               0   /bin/cat /dev/null                        /                                       
myfoo2                          1862039    35334      cat               0   /bin/cat /dev/null                        /                                       
unexpected fault address 0xc005400000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0xc005400000 pc=0x35fa293]

goroutine 87 [running]:
runtime.throw({0x4405a4d?, 0xc000680400?})
	/usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0xc003501c18 sp=0xc003501be8 pc=0x1b8477c
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:875 +0x285 fp=0xc003501c78 sp=0xc003501c18 pc=0x1b9bf25
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/tracer.(*Tracer).run(0xc0002e0a10)
	/home/mvb/kinvolk/ebpf/inspektor-gadget/pkg/gadgets/trace/exec/tracer/tracer.go:179 +0x813 fp=0xc003503fc8 sp=0xc003501c78 pc=0x35fa293
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/tracer.(*Tracer).Run.func2()
	/home/mvb/kinvolk/ebpf/inspektor-gadget/pkg/gadgets/trace/exec/tracer/tracer.go:212 +0x25 fp=0xc003503fe0 sp=0xc003503fc8 pc=0x35fa625
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc003503fe8 sp=0xc003503fe0 pc=0x1bb93c1
created by github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/exec/tracer.(*Tracer).Run in goroutine 1
	/home/mvb/kinvolk/ebpf/inspektor-gadget/pkg/gadgets/trace/exec/tracer/tracer.go:212 +0x27c

Bug Investigation

It's not clear to me why this bug doesn't happen more often. The code with the defect was introduced in 34edb8b. I used git bisect and found out that it started to happen only after 62a333d. I first thought it that it was related to the overwritable perf ring support, I tried to use git bisect on ebpf-go and found out that this only happens after cilium/ebpf@8fa4c90. I have no idea why it doesn't happen before that commit.

ref

The ebpf code has some optimization to only send the right amount of
bytes used to hold the arguments.
(See around L200 in pkg/gadgets/trace/exec/tracer/bpf/execsnoop.bpf.c).
bpf2go defines `Args` as `[7680]uint8` and the tracer was using
`args := bpfEvent.Args`, hence copying the array. It an issue since it's
trying to copy memory outside the sample received from the perf ring buffer.

Fixes: 34edb8b ("trace/exec: Add cwd field")

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>
Copy link
Member

@alban alban left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I wonder if there are other similar problems remaining. I'll dig a bit more, but this can be merged already.

@mauriciovasquezbernal mauriciovasquezbernal merged commit 7502710 into main Jan 29, 2024
56 checks passed
@mauriciovasquezbernal mauriciovasquezbernal deleted the mauricio/fix-trace-exec branch January 29, 2024 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants