Allow uprobes to work across filesystem namespace boundaries #2662
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
ivankelly
force-pushed
the
uprobe_container
branch
from
6b027e2
to
ba71178
Compare
When attaching to a uprobe, bpftrace will try to cache symbols for all pids using that program. It does this by canonicalizing the executable path and scanning /proc for processes whose /exe matches. For containers, this will not work as the canonicalize command will resolve /proc/123/root/foo/bar to /foo/bar in the host filesystem and then error trying to canonicalize that because it doesn't exist on the host filesystem. The proposed solution is, when getting the pids for a program, if the path is of the form /proc/<pid>/<root>/<path>, don't bother with the search and just extract the pid from the path itself. Testing uses a simulation of docker root mount setup, using unshare and pivot_root.
ivankelly
force-pushed
the
uprobe_container
branch
from
07478a5
to
acbb739
Compare
There was a problem hiding this comment.
Thanks! Looks ok to me, just a couple of nits and suggestions, mainly for the tests.
@lenticularis39 mind having a quick look?
src/utils.cpp
Outdated
| process.path() / "exe", ec); | ||
| if (!ec && program_abs == pid_program) | ||
| pids.emplace_back(std::stoi(filename)); | ||
| pids.emplace_back(std::stoi(match.str(1))); |
There was a problem hiding this comment.
I'm not sure if this is the right idea, I can trace e.g. /proc/123/root/bin/bash, and 123 does not have to be the ID of the bash process, it can be e.g. the root of the namespace or another process in the namespace. Since the cache is loaded by PID, it won't cause incorrect symbol resolution, but perhaps a smarter solution could be implemented (see comment below).
| if (!std::all_of(filename.begin(), filename.end(), ::isdigit)) | ||
| continue; | ||
| std::error_code ec; | ||
| std_filesystem::path pid_program = std_filesystem::read_symlink( |
There was a problem hiding this comment.
Idea: I am not sure what this does for programs which are in a different mount namespace. Maybe also trying std_filesystem::absolute next to std_filesystem::canonical could work.
There was a problem hiding this comment.
I don't think absolute is correct either, as the absolute path would simply be the path passed in. Perhaps weakly_canonical would work though. I'll give it a go.
There was a problem hiding this comment.
nope
[ FAILED ] uprobe.uprobes - attach to probe for executable in a pivot_root'd mount namespace
Command: ../src//bpftrace -e 'uprobe:/proc/20424/root/uprobe_test_static:function1 { printf("func %s\n", func); exit(); }'
Unclean exit code: 1
Output: ERROR: failed to write elf: filesystem error: cannot make canonical path: No such file or directory [/proc/20424/root/uprobe_test_static]\n
Which is strange, because that path does exist, but maybe the problem is that its reading across a filesystem boundary or something.
A solution could be to just catch the exception and return no pids if it gets a file not found. This method is only used to cache symbols for pids at probe attach time. if it doesn't cache anything, that's no big whoop, they can be resolved at output time.
|
@viktormalik @lenticularis39 Addressed comments. Now get_pids_for_program just bails if it cannot get the canonical path. |
|
Closing in favour of #2802 |
When attaching to a uprobe, bpftrace will try to cache symbols for all pids using that program. It does this by canonicalizing the executable path and scanning /proc for processes whose /exe matches.
For containers, this will not work as the canonicalize command will resolve
/proc/123/root/foo/barto/foo/barin the host filesystem and then error trying to canonicalize that because it doesn't exist on the host filesystem.The proposed solution is, when getting the pids for a program, if the path is of the form
/proc/<pid>/<root>/<path>, don't bother with the search and just extract the pid from the path itself.Checklist
man/adoc/bpftrace.adocand if needed indocs/reference_guide.mdCHANGELOG.md