Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent crash with /proc path canonicalization #1637

Merged
merged 2 commits into from Nov 24, 2020

Conversation

xdrop
Copy link
Contributor

@xdrop xdrop commented Nov 19, 2020

Issue

Attempting to trace an executable (with either namespace relative, or /proc rooted path) that is in a different mount namespace than the one bpftrace is running in fails.

# bpftrace -e 'usdt:/proc/92493/root/home/usdt_test:tracetest:testprobe { printf("hit\n")}' -p 92493
ERROR: failed to write elf: filesystem error: cannot canonicalize: No such file or directory [/proc/92493/root/home/usdt_test] [/bcc-build/bpftrace/build]

# bpftrace -e 'usdt:/home/usdt_test:tracetest:testprobe { printf("hit\n")}' -p 92493
ERROR: failed to write elf: filesystem error: cannot canonicalize: No such file or directory [/proc/92493/root/home/usdt_test] [/bcc-build/bpftrace/build]

Cause

The reason is attempting to perform filesystem::canonical('/proc/<pid>/root/usr/bin/..') to a proc path of a pid that is in a different mount namespace fails with No such file or directory. (it looks like canonical() does an lstat on the current root instead and doesn't find the file there)

Call stack

The call originates from abs_path, and this is the call stack before it blows up:

* thread #1, name = 'bpftrace', stop reason = breakpoint 1.1
  * frame #0: 0x000055555565dd52 bpftrace`bpftrace::abs_path(rel_path="p~\xb9UUU"...) at utils.cpp:927
    frame #1: 0x00005555555c2ed6 bpftrace`_ZNK8bpftrace8BPFtrace21find_wildcard_matchesB5cxx11ERKNS_3ast11AttachPointE(this=0x00007fffffffe120, attach_point=0x0000555555b80bf0) at bpftrace.cpp:406
    frame #2: 0x000055555569250e bpftrace`bpftrace::ast::CodegenLLVM::visit(this=0x00007fffffffdd60, probe=0x0000555555b11fb0) at codegen_llvm.cpp:2139
    frame #3: 0x0000555555676ff8 bpftrace`bpftrace::ast::Probe::accept(this=0x0000555555b11fb0, v=0x00007fffffffdd60) at ast.cpp:41
    frame #4: 0x00005555556987a5 bpftrace`bpftrace::ast::CodegenLLVM::accept(this=0x00007fffffffdd60, node=0x0000555555b11fb0) at codegen_llvm.cpp:2819
    frame #5: 0x0000555555692e52 bpftrace`bpftrace::ast::CodegenLLVM::visit(this=0x00007fffffffdd60, program=0x00007fffe4015fd0) at codegen_llvm.cpp:2228
    frame #6: 0x000055555567702c bpftrace`bpftrace::ast::Program::accept(this=0x00007fffe4015fd0, v=0x00007fffffffdd60) at ast.cpp:42
    frame #7: 0x00005555556987a5 bpftrace`bpftrace::ast::CodegenLLVM::accept(this=0x00007fffffffdd60, node=0x00007fffe4015fd0) at codegen_llvm.cpp:2819
    frame #8: 0x000055555569808b bpftrace`bpftrace::ast::CodegenLLVM::generate_ir(this=0x00007fffffffdd60) at codegen_llvm.cpp:2723
    frame #9: 0x0000555555668fc8 bpftrace`main(argc=5, argv=0x00007fffffffe688) at main.cpp:787
    frame #10: 0x00007fffed11dbf7 libc.so.6`__libc_start_main(main=(bpftrace`main at main.cpp:272), argc=5, argv=0x00007fffffffe688, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffe678) at libc-start.c:310
    frame #11: 0x0000555555579aaa bpftrace`_start + 42

Fix

The fix is to give up trying to canonicalize /proc based baths. That of course means something like tracing /proc/92493/root/home/../home/usdt_test will not work (it doesn't event work now anyway), but I don't see a strong use case for trying to support canonicalization of /proc based paths.

Checklist
  • User-visible and non-trivial changes updated in CHANGELOG.md
  • The new behaviour is covered by tests

@xdrop xdrop force-pushed the bugfix/usdt-in-mount-ns branch 2 times, most recently from 8c8cee7 to ac052ce Compare November 19, 2020 19:07
@xdrop xdrop marked this pull request as ready for review November 19, 2020 19:12
Copy link
Member

@danobi danobi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also add another test that exercises the previously broken behavior?

src/utils.cpp Outdated
Comment on lines 927 to 929
// filesystem::canonical does not work very well with /proc paths
// failing during canonicalization. See iovisor:bpftrace#1595
if (rel_path.rfind("/proc", 0) != 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, the issue isn't that fs::canonical() doesn't work on procfs [0], it's that we're trying to canonicalize across mount namespaces. And part of the issue is that /proc/pid/root is a symlink to target pid's root as viewed by the target.

Wouldn't a better fix be to check for /root as well as /proc?

[0]: http://ix.io/2EQ0 prints "/usr/bin" for me

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have clarified the comment, limited the check for /proc/<pid>/root, and improved the test to cover the functions behaviour.

I assume by "test that exercises the previously broken behavior" you meant a test that actually tries to trace with a path of a process from another mount ns? That test already exists but is disabled, I have created an issue to re-enable it: #1638

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume by "test that exercises the previously broken behavior" you meant a test that actually tries to trace with a path of a process from another mount ns? That test already exists but is disabled, I have created an issue to re-enable it: #1638

Nice :)

@xdrop xdrop force-pushed the bugfix/usdt-in-mount-ns branch 5 times, most recently from 5e2d975 to 102e8c0 Compare November 21, 2020 20:30
src/utils.cpp Outdated
// filesystem::canonical does not work very well with /proc/<pid>/root paths
// of processes in a different mount namespace (than the one bpftrace is
// running in), failing during canonicalization. See iovisor:bpftrace#1595
if (!std::regex_match(rel_path, std::regex("^/proc/\\d+/root/.*")))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be expensive to keep compiling regex. Let's just do it once.

ie:

static auto re = std::regex("^/proc/\\d+/root/.*");
if (!std::regex_match(rel, path, re))
  ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

The filesystem::canonical does not work very well with /proc paths,
explictly ignoring those paths from filesystem::canonical for now.
@dalehamel
Copy link
Contributor

LGTM

@danobi
Copy link
Member

danobi commented Nov 24, 2020

Thanks for the fix!

@danobi danobi merged commit ccdb0e7 into bpftrace:master Nov 24, 2020
casparant added a commit to casparant/bpftrace that referenced this pull request Nov 24, 2020
* 'master' of github.com:iovisor/bpftrace:
  Prevent crash with /proc path canonicalization (bpftrace#1637)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants