How to deal with uprobe process in docker container from host?

[iAklis]

I run a docker container(docker run -it fedora:26 /bin/bash) which runs bash.

18f3fa58fad7 is the docker container. I installed bcc on hosts.

How could I make tool like bashreadline work for bash in container well as the bash on host.

[yonghong-song]

could you check whether uprobe is installed properly on the host or not? For early version, you can check debugfs /sys/kernel/debug/tracing/uprobe_events, newer kernel you can check using kernel bpftool perf. Maybe you want to try to use debugfs uprobe_events directly to check whether it works or not? May also check whether thee above /bin/bash has "readline" function or not?

[iAklis]

@yonghong-song
Thank you for your reply.

I checked uprobe_events found uprobe_events worked (in uprobe_events). But there is nothing printed.

I can use ltrace to attach the bash(etc pid 31571 below), but uretprobe not worked.

Docker version 18.09.2, build 6247962
Storage Drive: aufs
Host Linux kernel version: 4.15.0-46-generic

[iAklis]

I run the zlim/bcc attached one container for tracing the bash.

docker run -it \
    --pid=container:4b19b2305663 \
    --net=container:4b19b2305663 \
    -v /lib/modules:/lib/modules:ro \
    -v /usr/src:/usr/src:ro \
    --privileged \
    --cap-add sys_admin \
    --cap-add sys_ptrace \
zlim/bcc

But nothing got.

[iAklis]

I discovered was related to Storage Drive.

[yzhao1012]

Re: the last update

@iAklis
Could you provide more details on what is "related to Storage Drive"? I've seen uprobe attached to overlay file path not working, while uprobe attached to the host file is working.

[palmtenor]

@yzhao1012 If by overlay you mean overlayfs then you need torvalds/linux@f0a2aa5

[mgale]

@palmtenor or @yzhao1012 Does anyone have any more information on this topic?
My binary works on the host os but only sees events from the host os, not from the container.
My binary works in the container but only sees events from inside the container, not from the host.

I would like to run the binary inside the container and capture events from the host (outside the container).

[ezxfv]

@mgale uprobes are made of the program file inode, the instruction offset, the list of related actions and the replaced instruction code. When you create a probe, it is added to a specific binary tree. it means you can't use a binary copy, but you can attach to the binary in container's overlayfs, use docker inspect <container id / name> to get all layer, like:

"GraphDriver": {
            "Data": {
                "LowerDir": "/var/snap/docker/common/var-lib-docker/overlay2/7e3b77e99734c319d186787b37cb9fe675d1727e3b2b2e1948f583e5bb0778e4-init/diff:/var/snap/docker/common/var-lib-docker/overlay2/ab98d5ddf7f4d393186c8eae7f2802eaaf7f8298b55dbcfd05dd34b9b3bcd004/diff:/var/snap/docker/common/var-lib-docker/overlay2/8e49943a79719b2ca9bfaa0b1aaf5627ea00b2e0a2777a96acecd1d25954ad21/diff:/var/snap/docker/common/var-lib-docker/overlay2/53bfa27bf1fecc791c82d239e54c42b813d04b87aab61b69007f75812c6703ec/diff:/var/snap/docker/common/var-lib-docker/overlay2/15cfebca848e6be2e298ba90acac9e35f42475ce363510ab0e40ecbcc5f318cc/diff:/var/snap/docker/common/var-lib-docker/overlay2/e08d27cdb032bdc8e5048dfa39b8baf4448792a785739b9d6c31c79e6114e31e/diff:/var/snap/docker/common/var-lib-docker/overlay2/512ea7cc8d67a15fa9ba21fb763fcf279bf1ef9c0df1e7830d53fa5d907a4946/diff:/var/snap/docker/common/var-lib-docker/overlay2/de41f924c7eb3dcb88efc8fde077b96244a5eb15087a5b457ffd7f234aec9c67/diff",
                "MergedDir": "/var/snap/docker/common/var-lib-docker/overlay2/7e3b77e99734c319d186787b37cb9fe675d1727e3b2b2e1948f583e5bb0778e4/merged",
                "UpperDir": "/var/snap/docker/common/var-lib-docker/overlay2/7e3b77e99734c319d186787b37cb9fe675d1727e3b2b2e1948f583e5bb0778e4/diff",
                "WorkDir": "/var/snap/docker/common/var-lib-docker/overlay2/7e3b77e99734c319d186787b37cb9fe675d1727e3b2b2e1948f583e5bb0778e4/work"
            },
            "Name": "overlay2"
        }

for example, my binary is /var/snap/docker/common/var-lib-docker/overlay2/7e3b77e99734c319d186787b37cb9fe675d1727e3b2b2e1948f583e5bb0778e4/diff/src/app, I attached to this binary from host is working