New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trace processes that run in docker-container. #67

Closed
thieso2 opened this Issue Jan 25, 2018 · 5 comments

Comments

Projects
None yet
4 participants
@thieso2
Copy link

thieso2 commented Jan 25, 2018

Hi - love rbspy already - big thanx. I can see this becoming a very important tool for all the ruby people out there!

I run quite a few things in docker container and love to trace them without having to enter the container (i would also run the container privileged I believe to use rbspy within the container).

To reproduce:
docker-host> docker run ruby ruby -e "sleep(500)"

(in different shell)
docker-host> ps auxww |grep "ruby -e sleep"
root 27179 0.0 0.0 38008 7460 ? Ssl 16:28 0:00 ruby -e sleep(500)
docker-host> sudo ./rbspy snapshot --pid 27179
Error. Causes:

  • Couldn't determine Ruby version
  • Couldn't open ELF file: /usr/local/bin/ruby

This is do to the fact that ruby is of course installed in the container somewhere in the depth of /var/lib/docker ...

@liaden

This comment has been minimized.

Copy link
Collaborator

liaden commented Jan 25, 2018

Profiling the ruby process within a docker container is a future feature that @jvns mentioned working towards. In the short term you might be able to use nsenter to run rbspy. I haven't tried this yet.

On slide 36, he shows some examples: http://www.brendangregg.com/blog/2017-05-15/container-performance-analysis-dockercon-2017.html

@jvns

This comment has been minimized.

Copy link
Collaborator

jvns commented Jan 26, 2018

just made a v0.1.4 release that should make rbspy work with containers. It works in my testing but please reopen if you still have problems with it!

@kirs

This comment has been minimized.

Copy link

kirs commented Apr 1, 2018

hey @jvns thank you for rbspy! I came across this issue because I'm having troubles using rbspy from a docker container. Here are the steps I'm doing:

$ docker exec -i -t fa6a46a29977 /bin/bash
root@fa6a46a29977:/app# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.7  2.8 837244 58044 pts/0    Ssl+ 16:57   0:01 puma 3.11.3 (tcp://0.0.0.0:3000) [app]
root        80  2.0  0.1  18192  3148 pts/1    Ss   17:01   0:00 /bin/bash
root        85  0.0  0.1  36632  2780 pts/1    R+   17:01   0:00 ps aux
root@fa6a46a29977:/app# /bin/rbspy record --pid 1
Press Ctrl+C to stop
Wrote raw data to /root/.cache/rbspy/records/rbspy-2018-04-01-MyTw9IVjaU.raw.gz
Writing formatted output to /root/.cache/rbspy/records/rbspy-2018-04-01-IQ1dYlz1y8.flamegraph.svg
ERROR: No stack counts found
Error. Causes:
- Couldn't determine Ruby version
- Couldn't create process handle for PID
- Permission denied when reading from process. Try again with sudo and check sys_ptrace capability?

Both shell and the ruby process are running from root.

I'm using ruby 2.5.0p0 and rbspy-v0.2.3-x86_64-unknown-linux-musl. The container base is ruby:2.5.

@kirs

This comment has been minimized.

Copy link

kirs commented Apr 3, 2018

Yay, it looks like running the container with --cap-add=SYS_PTRACE flag allows it to use sys_ptrace. Found it in moby/moby#21051.

@jvns

This comment has been minimized.

Copy link
Collaborator

jvns commented Apr 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment