Detect issue with LTTng and Docker and report error when tracing

[christophebedard]

There is an issue with LTTng when using more than one Docker container that share the same mounted home directory. When trying to trace in a second container, LTTng will mistakenly think that there is a session daemon running, so it will happily trace, but no trace data will be recorded, because there isn't actually a session daemon in that container. The fix is to use a different LTTng home, e.g., by setting the LTTNG_HOME environment variable to a unique value. See this LTTng issue: https://bugs.lttng.org/issues/1371

This PR detects this (very small) corner case and reports the issue to the user so that they can set LTTNG_HOME. Example:

1. Tracing works fine in the first container:

container1$ ros2 trace -s my-sesh-container1
UST tracing enabled (59 events)
kernel tracing disabled
context (3 fields)
writing tracing session to: /home/user/.ros/tracing/my-sesh-container1
press enter to start...
press enter to stop...
stopping & destroying tracing session

2. But an error will be reported in the second container:

container2$ ros2 trace -s my-sesh-container2
UST tracing enabled (59 events)
kernel tracing disabled
context (3 fields)
writing tracing session to: /home/user/.ros/tracing/my-sesh-container2
press enter to start...
error: lttng-sessiond seems to exist, but is unreachable. If using two containers with the same HOME directory, set the LTTNG_HOME environment variable to a unique value for each container. See: https://bugs.lttng.org/issues/1371

3. Setting LTTNG_HOME to a unique value (so that container 2 has its own LTTng home) and tracing again will work:

container2$ export LTTNG_HOME="$HOME/.lttng_home2"
container2$ ros2 trace -s my-sesh-container2
UST tracing enabled (59 events)
kernel tracing disabled
context (3 fields)
writing tracing session to: /home/user/.ros/tracing/my-sesh-container2
press enter to start...
press enter to stop...
stopping & destroying tracing session

Note that we must first try to spawn a session daemon is there isn't one, because, when a Docker container is killed/stopped, the ~/.lttng/lttng-sessiond.pid file (which contains the PID of the session daemon that is expected to be running) still exists. This would result in a false positive due to the way we detect cases of the Docker issue. If we manage to spawn a session daemon before checking for an unreachable session daemon, then we avoid the false positive.

Also, note that the LTTNG_HOME directory must exist; LTTng will not create it. Therefore, make sure to tell the user to create the directory.

Signed-off-by: Christophe Bedard christophe.bedard@apex.ai

[mjcarroll]

A small corner case, but worth covering as Docker is pretty popular in our community.

[christophebedard]

• Linux
• Linux-aarch64
• Windows