py-spy on host unable to trace python process running in container

[sciurus]

From the host, I tried to trace a python process running in a docker container. It failed with the following error.

# py-spy --pid 3676
Error: No such file or directory (os error 2)

strace shows it dies at open("/usr/local/bin/python2.7", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory). I see that it got that path via readlink("/proc/3676/exe", "/usr/local/bin/python2.7", 256) = 24

python2.7 is only available at that path in the container's mount namespace. From outside the namespace it's in one of the filesystems mounted under /var/lib/docker.

[benfred]

Thanks for the bug report!

I think that this shouldn't be too hard to fix. It seems like I can get the namespace of the target process by reading /proc/PID/ns/mnt, and then call setns before trying to open files from the target process to get around this.

In the meantime, if you've launched your docker container with the SYS_PTRACE capability - you can run py-spy inside the container rather than from the host os.

[benfred]

I managed to replicate this, and it seems like doing something like:

use nix::sched::{setns, CloneFlags};
use std::os::unix::io::AsRawFd;

let file = std::fs::File::open(format!("/proc/{}/ns/mnt", pid))?;
setns(file.as_raw_fd(), CloneFlags::from_bits_truncate(0))?;

before trying to open any files is sufficient enough to get this working.

[vsolovyov]

Even if you didn't launch your docker container with running code and SYS_PTRACE capability, it's possible to launch new container with it, ask docker to use pid namespace of already running container. Something like this should get it working:

docker run -it --pid=container:<your-python-container> --rm --priveleged --privileged --cap-add SYS_PTRACE -v "$(pwd):/py-spy" your.docker.container/with-py-spy bash

You actually can launch some generic python container that has pip there without py-spy preinstalled, install py-spy on the fly and profile your container right there. It's a wonderful piece of technology :)

[alsuren]

There seems to be a patch to rb-spy from January which does setns before reading /proc/PID/exe and then setns back again once you've done the reading. (this is what pyflame does) rbspy/rbspy#68 . This then got reverted (something about libruby.so being troublesome?), and replaced with rbspy/rbspy@3ad0bed which uses /proc/PID/root/MAP_PATH instead.

Are either of these things that we could cherry-pick?

[benfred]

This fixes here: #66

[benfred]

fixed in v0.1.9

[Belval]

This seems to still be an issue in 0.1.10, unless there is an extra step that I am missing.

1. Launch container without any special flags (I might be missing something here)
2. Launch script in docker container
3. py-spy --pid 12345 fails with the above error message

[vsolovyov]

@Belval Have you tried launching it with sudo? I think it needs sudo to access containers.

[Belval]

Unfortunately I can't test it with sudo as I am not a super user on that machine, but I have access to the docker daemon and the container belongs to my user.

[fmhall]

It still doesnt work with sudo

[tachang]

Getting this:

[2019-11-23T18:27:22.803584956Z INFO  remoteprocess::linux] Process 17 appears to be running in a different namespace - setting namespace to match
[2019-11-23T18:27:22.803655874Z WARN  py_spy::python_spy] Failed to set namespace: EINVAL: Invalid argument
[2019-11-23T18:27:22.803706422Z INFO  py_spy::python_spy] Found libpython binary @ /usr/local/lib/libpython3.7m.so.1.0
) = 0
futex(0x559b69e1ea20, FUTEX_WAKE_PRIVATE, 1) = 0
write(2, "Error: ", 7Error: )                  = 7
write(2, "No such file or directory", 25No such file or directory) = 25
write(2, " (os error ", 11 (os error )             = 11
write(2, "2", 12)                        = 1
write(2, ")", 1))                        = 1
write(2, "\n", 1
)                       = 1
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7fe8e5142000, 8192)            = 0
exit_group(1)                           = ?
+++ exited with 1 +++