Performance benchmarking refactor

[adamdbrw]

Improvements to current performance benchmarking package:

• extended to include benchmarks of end-to-end performance of recording (including transport).
• reuse of byte producer for both transport-less writer and end-to-end
• YAML configuration enabling mixed publishers, defined as groups (to simplify adding hundreds or thousands of similar publishers).
• configuration of QoS settings of publishers
• use new configuration in writer_benchmark as well
• replace non-portable and clunky benchmark script with launch / python based experiment batch execution
• add description on how to run it to readme
• count/stats for end-to-end benchmarks
• add end-to-end experiments to launch
• run (more) experiments with new setups. Compare trasport vs no-transport.
• only record interesting topics so that stats are not disturbed (i.e. no logs or parameters) - this was solved with Recorder --regex and --exclude options #604.

[adamdbrw]

@mjeronimo @emersonknapp @Karsten1987 this is ready for a review now

[emersonknapp]

Pretty complicated - very big review, but since it's not direct product code, i'm inclined to merge it if it works as desired

[adamdbrw]

We already used it to run numerous experiments (reports are incoming) and it works for us reliably. I agree it can be difficult to review since it is quite big.

[emersonknapp]

Gist: https://gist.githubusercontent.com/emersonknapp/7bc8c2396115fc7b1d52a2fc8f8c9ea0/raw/7d026faddaa306b579571fb44d4d3ae65f33f992/ros2.repos
BUILD args: --packages-up-to rosbag2_performance
TEST args: --packages-select rosbag2_performance
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/7510

• Linux
• Linux-aarch64
• macOS
• Windows

[adamdbrw]

test failure: since the code of this PR is contained in an independent package that in no way affects rosbag2 recording, I assume that this is a flaky test.
CI job failures: I am not entirely sure, but could it be due to the fact that the benchmarking package requires --cmake-flags -DBUILD_ROSBAG2_BENCHMARKS in colcon build arguments? This however didn't change with this PR, since the previous version of the package required the same flag.

Edit: corrected spelling of the flag.

[emersonknapp]

CI job failures: I am not entirely sure, but could it be due to the fact that the benchmarking package requires --cmake-flags -DBUILD_ROSBAG2_BENCHMARKING in colcon build arguments

I have added that, but the main problem is my mistake: I asked for package rosbag2_performance when it is called rosbag2_performance_benchmarking

• Linux
• Linux-aarch64
• macOS
• Windows

[adamdbrw]

The flag needs to be --cmake-args -DBUILD_ROSBAG2_BENCHMARKS=1. My apologies for giving a wrong spelling and no value in an earlier comment (this is in the README, but I guess the late hour works against me)

[adamdbrw]

I fixed Windows build issue with uint (which is an usual compiler shortcut but not supported by all compilers). Perhaps all will be green this time :]
It seems that there are build issues in compression code (not affected by this PR)

[emersonknapp]

It seems that there are build issues in compression code (not affected by this PR)

Where? If you mean the "Rpr" build - it isn't a good indicator of success, because it only builds against released versions of packages, rather than against the latest version of the code. In this case, rcpputils has a new const-specifier that is merged in master but not released in rolling yet - I would ignore that build and pay attention to "Test rosbag2 / build_and_test (pull_request) " and the ci.ros2.org builds in general, which both build from source

[emersonknapp]

• Linux
• Linux-aarch64
• macOS
• Windows

[y-okumura-isp]

Are there any plans to backport it to Foxy?

[adamdbrw]

@y-okumura-isp I will soon be able to update you on this, since it is a ongoing decision process that also needs feedback from the community. If I can support you in the meanwhile, please let me know - you can find my email in the commit log.

The same applies to double buffering PR (#546) - I saw you were also interested in this one.