feat(process-monitor): Include information about namespaces #207
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vadorovsky
commented
Sep 25, 2023
|
|
||
| fn get_process_namespaces(pid: Pid) -> Result<Namespaces, Error> { | ||
| Ok(Namespaces { | ||
| uts: get_process_namespace(pid, "uts").unwrap_or_default(), |
There was a problem hiding this comment.
For now I'm doing unwrap_or_default here, because apparently some processes have malformed namespace data, like:
$ sudo -E ./target/release/pulsar-exec pulsard
[2023-09-25T01:06:12Z INFO pulsar::pulsard::daemon] Starting module process-monitor
[2023-09-25T01:06:12Z ERROR pulsar::pulsard::module_manager] Error in module process-monitor. Module stopped. Error initializing process filtering
Caused by:
failed to get the uts namespace for process 11358
pulsar on namespaces [$!] via v1.72.0 took 2s
❯ sudo ls -la /proc/11358/ns/
ls: cannot read symbolic link '/proc/11358/ns/net': No such file or directory
ls: cannot read symbolic link '/proc/11358/ns/uts': No such file or directory
ls: cannot read symbolic link '/proc/11358/ns/ipc': No such file or directory
ls: cannot read symbolic link '/proc/11358/ns/pid_for_children': No such file or directory
ls: cannot read symbolic link '/proc/11358/ns/mnt': No such file or directory
ls: cannot read symbolic link '/proc/11358/ns/cgroup': No such file or directory
ls: cannot read symbolic link '/proc/11358/ns/time': No such file or directory
ls: cannot read symbolic link '/proc/11358/ns/time_for_children': No such file or directory
total 0
dr-x--x--x 2 root root 0 Sep 25 03:04 .
dr-xr-xr-x 9 vadorovsky vadorovsky 0 Sep 25 02:48 ..
lrwxrwxrwx 1 root root 0 Sep 25 03:07 cgroup
lrwxrwxrwx 1 root root 0 Sep 25 03:07 ipc
lrwxrwxrwx 1 root root 0 Sep 25 03:07 mnt
lrwxrwxrwx 1 root root 0 Sep 25 03:07 net
lrwxrwxrwx 1 root root 0 Sep 25 03:07 pid -> 'pid:[4026532723]'
lrwxrwxrwx 1 root root 0 Sep 25 03:07 pid_for_children
lrwxrwxrwx 1 root root 0 Sep 25 03:07 time
lrwxrwxrwx 1 root root 0 Sep 25 03:07 time_for_children
lrwxrwxrwx 1 root root 0 Sep 25 03:07 user -> 'user:[4026532721]'
lrwxrwxrwx 1 root root 0 Sep 25 03:04 uts
pulsar on namespaces [$!] via v1.72.0
❯ sudo cat /proc/11358/comm
zypak-sandbox
pulsar on namespaces [$!] via v1.72.0
❯ pstree -s -p 11358
systemd(1)───systemd(3137)───gnome-shell(3287)───bwrap(11301)───bwrap(11322)───brave(11323)───zypak-sandbox(11358)I'm wondering if we should log a warning in such case. The thing is that we don't seem to have logs at all in the bpf-filtering crate, but I could add it if you are fine with that @banditopazzo
There was a problem hiding this comment.
yes I agree to log the warning as a reminder to understand the reason behind this malformed data.
this is another examples of silent default usages:
https://github.com/Exein-io/pulsar/blob/7616035894404f00cfb5e4ff31f5f0c16a805013/crates/bpf-filtering/src/process_tree.rs#L90-L99
probably should be moved to a warning, what do you think?
There was a problem hiding this comment.
Alright, I will use log::warn then.
|
The output in monitor looks like: [2023-09-25T01:28:33Z EVENT /usr/sbin/containerd (8160)] [process-monitor] Fork { ppid: 1953, namespaces: { uts: 4026531838, ipc: 4026531839, mnt: 4026531841, pid: 4026531836, net: 4026531840, time: 4026531834, cgroup: 4026531835 } }
[2023-09-25T01:28:33Z EVENT /usr/sbin/containerd (8160)] [process-monitor] Exec { filename: /usr/sbin/containerd-shim-runc-v2, argc: 4026531838, argv: [ /usr/sbin/containerd-shim-runc-v2, -namespace, moby, -address, /var/run/docker/containerd/containerd.sock, -publish-binary, /usr/sbin/containerd, -id, 0b6a24ab54de6456ded6eab141e59cfde7f25d2652df928c821fbf304251b743, start ], namespaces: { uts: 4026531841, ipc: 4026531836, mnt: 4026531840, pid: 4026531834, net: 4026531835, time: 0, cgroup: 0 } }
[2023-09-25T01:28:33Z EVENT /usr/sbin/containerd-shim-runc-v2 (8166)] [process-monitor] Fork { ppid: 8160, namespaces: { uts: 4026531838, ipc: 4026531839, mnt: 4026531841, pid: 4026531836, net: 4026531840, time: 4026531834, cgroup: 4026531835 } }
[2023-09-25T01:28:33Z EVENT /usr/sbin/containerd-shim-runc-v2 (8166)] [process-monitor] Exec { filename: /usr/sbin/containerd-shim-runc-v2, argc: 4026531838, argv: [ /usr/sbin/containerd-shim-runc-v2, -namespace, moby, -id, 0b6a24ab54de6456ded6eab141e59cfde7f25d2652df928c821fbf304251b743, -address, /var/run/docker/containerd/containerd.sock ], namespaces: { uts: 4026531841, ipc: 4026531836, mnt: 4026531840, pid: 4026531834, net: 4026531835, time: 0, cgroup: 0 } }
[2023-09-25T01:28:33Z EVENT /usr/sbin/containerd-shim-runc-v2 (8160)] [process-monitor] Exit { exit_code: 0 }
[2023-09-25T01:28:33Z EVENT /usr/sbin/containerd-shim-runc-v2 (8178)] [process-monitor] Fork { ppid: 8166, namespaces: { uts: 4026531838, ipc: 4026531839, mnt: 4026531841, pid: 4026531836, net: 4026531840, time: 4026531834, cgroup: 4026531835 } }
[2023-09-25T01:28:33Z EVENT /usr/sbin/containerd-shim-runc-v2 (8178)] [process-monitor] Exec { filename: /usr/sbin/runc, argc: 4026531838, argv: [ runc, --root, /var/run/docker/runtime-runc/moby, --log, /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/0b6a24ab54de6456ded6eab141e59cfde7f25d2652df928c821fbf304251b743/log.json, --log-format, json, --systemd-cgroup, create, --bundle, /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/0b6a24ab54de6456ded6eab141e59cfde7f25d2652df928c821fbf304251b743, --pid-file, /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/0b6a24ab54de6456ded6eab141e59cfde7f25d2652df928c821fbf304251b743/init.pid, --console-socket, /tmp/pty3126701025/pty.sock, 0b6a24ab54de6456ded6eab141e59cfde7f25d2652df928c821fbf304251b743 ], namespaces: { uts: 4026531841, ipc: 4026531836, mnt: 4026531840, pid: 4026531834, net: 4026531835, time: 0, cgroup: 0 } }
[2023-09-25T01:28:33Z EVENT /usr/sbin/runc (8186)] [process-monitor] Fork { ppid: 8178, namespaces: { uts: 4026531838, ipc: 4026531839, mnt: 4026531841, pid: 4026531836, net: 4026531840, time: 4026531834, cgroup: 4026531835 } }
[2023-09-25T01:28:33Z EVENT /usr/sbin/runc (8186)] [process-monitor] Exec { filename: /proc/self/exe, argc: 4026531838, argv: [ runc, init ], namespaces: { uts: 4026531841, ipc: 4026531836, mnt: 4026531840, pid: 4026531834, net: 4026531835, time: 0, cgroup: 0 } }
[2023-09-25T01:28:33Z EVENT (8188)] [process-monitor] Fork { ppid: 2, namespaces: { uts: 4026531838, ipc: 4026531839, mnt: 4026531841, pid: 4026531836, net: 4026531840, time: 4026531834, cgroup: 4026531835 } }
[2023-09-25T01:28:33Z EVENT /proc/self/exe (8186)] [process-monitor] Exec { filename: /dev/fd/7, argc: 4026531838, argv: [ runc, init ], namespaces: { uts: 4026531841, ipc: 4026531836, mnt: 4026531840, pid: 4026531834, net: 4026531835, time: 0, cgroup: 0 } }
[2023-09-25T01:28:33Z EVENT /usr/lib/systemd/systemd (1)] [process-monitor] Cgroup created { cgroup_path: /system.slice/docker-0b6a24ab54de6456ded6eab141e59cfde7f25d2652df928c821fbf304251b743.scope, cgroup_id: 12235I think the next step to have a proper container support would be writing an uprobe for runc/crun/youki which detects when new container is created and fetches the info about namespaces of the "first process of the container". And keep some mapping between container ID and corresponding namespaces. |
vadorovsky
force-pushed
the
namespaces
branch
2 times, most recently
from
74a3a64
to
7616035
Compare
banditopazzo
approved these changes
Oct 9, 2023
|
|
||
| fn get_process_namespaces(pid: Pid) -> Result<Namespaces, Error> { | ||
| Ok(Namespaces { | ||
| uts: get_process_namespace(pid, "uts").unwrap_or_default(), |
There was a problem hiding this comment.
yes I agree to log the warning as a reminder to understand the reason behind this malformed data.
this is another examples of silent default usages:
https://github.com/Exein-io/pulsar/blob/7616035894404f00cfb5e4ff31f5f0c16a805013/crates/bpf-filtering/src/process_tree.rs#L90-L99
probably should be moved to a warning, what do you think?
vadorovsky
force-pushed
the
namespaces
branch
from
7616035
to
3c51f31
Compare
This is a preparatory change for supporting container monitoring in Pulsar. Containers are just set of namespaces, so the easiest way to recognize that a process belongs to a particular container from information available from BPF programs, is to do that based on namespaces. For now, we simply add information about all namespaces for each executed or forked process.
vadorovsky
force-pushed
the
namespaces
branch
from
3c51f31
to
dde7040
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a preparatory change for supporting container monitoring in Pulsar. Containers are just set of namespaces, so the easiest way to recognize that a process belongs to a particular container from information available from BPF programs, is to do that based on namespaces.
For now, we simply add information about all namespaces for each executed or forked process.