Declare a dependency on the (Linux-only) LTTng userspace tracer

[christophebedard]

Feature request

Feature description

This is a follow-up to #957 and relates to ros2/ros2_tracing#23

I would like to discuss declaring a dependency on the LTTng userspace tracer (LTTng-UST) on Linux. I'd like to make sure that this change would be welcomed. My goal would be for this to be implemented a fair amount of time before the next release (Humble).

Core ROS 2 packages are instrumented with calls to functions provided by the tracetools package. These functions contain:

1. LTTng tracepoints if LTTng is installed/detected and tracetools is re-compiled/overlayed
2. nothing when building from source if LTTng isn't installed/detected
3. nothing in debians/Linux binaries
4. nothing in any code built for/on non-Linux platforms (macOS, Windows)

and

5. the function calls themselves can be completely removed by building instrumented core packages with -DTRACETOOLS_DISABLED=ON

This LTTng dependency isn't declared. This was initially for simplicity (especially since this is a platform-specific dependency) and because of some concerns about overhead.

This means that this feature currently isn't available out-of-the-box; users need to manually install LTTng and build from source (or at least just tracetools), since it's not installed when installing debians or running rosdep install. This has led to some questions and issues:

• https://gitlab.com/ros-tracing/ros2_tracing/-/issues/94
• https://gitlab.com/ros-tracing/ros2_tracing/-/merge_requests/139

As mentioned in #957 (comment), the LTTng userspace tracer is less "invasive" than the kernel tracer. I don't think we need to declare a dependency on the kernel tracer. Users could install it manually if they want to record kernel data for their analyses.

Also, in practice, the overhead is really negligible if tracepoints are not explicitly enabled: https://github.com/lttng/lttng-ust/blob/1f8a8ec9581af89d98aec47de9ad9e25087cd54a/include/lttng/tracepoint.h#L53-L62. Users/companies that do not want to ship a ROS 2-based product with LTTng can completely disable/remove it by setting the right CMake option (see item 5 above) and building from source, which I assume is something they are already doing.

Some references:

• Comparing overhead of tracepoints compiled in and enabled vs tracepoints compiled out for a pub->sub use-case
  • Section VI: https://arxiv.org/pdf/2201.00393.pdf
• Average overhead of a single LTTng-UST tracepoint: 158 ns (on a "typical" system)
  • Table VI: https://publications.polymtl.ca/3816/1/2018_Gebai_Survey_analysis_kernel_userspace_tracers.pdf
• Impact of a tracepoint that is compiled in but not enabled (suggested default behaviour): a simple condition check for a branch which is optimized for the case where the tracepoint is disabled
  • Section IV-B: https://publications.polymtl.ca/3816/1/2018_Gebai_Survey_analysis_kernel_userspace_tracers.pdf

Implementation considerations

New dependencies for tracetools:

• liblttng-ust-dev: LTTng-UST
• lttng-tools: LTTng CLI (not strictly necessary, but recommended to be able to actually use the tracepoints)

These would of course need to be installed on the Linux ci.ros2.org executors.

If the tracing tests turn out to create problems on ci.ros2.org, skipping them could be an option, but I think it would probably be fine.

• Also, the ros2_tracing README currently recommends installing both kernel and userspace tracers: https://github.com/ros2/ros2_tracing#building. Some logic would need to change in the package supporting the ros2 trace command and the Trace action so that it doesn't assume that it's either all (kernel+userspace tracers) or nothing.
  • this has been implemented: https://gitlab.com/ros-tracing/ros2_tracing/-/merge_requests/273

Any other platform-related considerations?

Pull requests

• Include tracepoints by default on Linux ros2_tracing#31
• Add 'lttng-tools' rosdep rule for RHEL ros/rosdistro#36641
• Install LTTng and related packages on Linux ci#690
• Flip ros2_tracing guide to building without instrumentation ros2_documentation#3404
• Add release note about tracepoints being included by default ros2_documentation#3414

[christophebedard]

@clalancette friendly ping 😃 I see that you self-assigned this. Is there any way I can help push this forward?

[christophebedard]

@clalancette friendly ping

[mjcarroll]

I think that having the tracepoints available by default is a reasonable request.

The value of being able to trace without rebuilding large portions of the workspace is helpful for users trying to diagnose performance issues.
Additionally, as time goes on, it would be nice to be able to build more analysis tooling around the tracing capabilities, and having it easily available would be valuable.

The overhead of the user-space trace points seems minimal compared to the value of having the tracepoints in.

The only concern I can think of would be the availability of lttng-ust on our supported platforms which is currently:

RHEL 9: (2.12?)
Ubuntu Jammy: (2.13.1)[https://packages.ubuntu.com/jammy/liblttng-ust1]
Debian Bullseye: (2.12.1)[https://packages.debian.org/bullseye/liblttng-ust0]
OpenEmbedded: (2.13.5)[http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng]

@cottsay I may need your input on RHEL, it appears to be 2.12, but I'm not sure I was using the correct package listing.

[mjcarroll]

Also of note, the users who are deeply concerned with performance (on the level of lttng serspace tracepoints being an issue) are likely not using our pre-compiled binaries and are instead compiling themselves.

[cottsay]

cottsay I may need your input on RHEL, it appears to be 2.12, but I'm not sure I was using the correct package listing.

It is indeed 2.12.0:

$ dnf list -q --available --showduplicates lttng-ust
Available Packages
lttng-ust.x86_64          2.12.0-6.el9          rhel-9-for-x86_64-appstream-rpms

[christophebedard]

I can work on the main ros2_tracing PR to enable the instrumentation by default.

The only concern I can think of would be the availability of lttng-ust on our supported platforms which is currently:

This is kind of related to ros2/ros2_tracing#16. If all current Linux platforms have LTTng-UST >= 2.12, we should be fine, but things could still be improved.

[mjcarroll]

Yes, 2.12 is going to be the least common denominator for rolling. I think we would have no intention of backporting this into released distributions.

[christophebedard]

I think we would have no intention of backporting this into released distributions.

Agreed

[clalancette]

One thing I will note is that RHEL-9 is aspirational for Rolling at this time. We may or may not get it in before Iron, so Iron may end up supporting RHEL-8 still. I'm fine if we end up disabling tracing by default on RHEL-8, but we should keep that in mind.

[christophebedard]

The PRs have been merged, so I'm closing this.