Benchmark for different ROS2 clients

[minggangw]

Feature request

Feature description

We are developing rclnodejs, a Node.js client for ROS2. We want to add some Performance and Power (PnP) tests for the current rclcpp, rclpy and rclnodejs (may be more) clients, some cases have been designed here. But I don't think it is proper to put these codes in the repo of rclnodejs, does the community have any plan to hold the benchmark test on a specific repo? any suggestions? Thanks!

[dirk-thomas]

Placing them in this repo sounds like a good option to me. As an alternative we could also create a new repository.

[wjwwood]

Here or in a new repository is also good by me. The best option might be in this repository but in a new package, like test_performance or whatever you want to call it.

For organization, I'd recommend starting the package name or package names with test_ as that's a convention we use for things that do not need to be released. Also, it might be a good idea to split it up by package name, so that it's easy to only build rclnodejs performance tests or only rclcpp and rclnodejs performance tests by simply putting an AMENT_IGNORE in the other package's directory. As opposed to cmake options and optional dependencies. Though those are all just suggestions, others might disagree.

[minggangw]

I agree that putting a package named test_clients_performance under this repo is reasonable, and the structure looks like:

├── test_clients_performance
     ├── test_rclcpp  -- package for C++ client
     ├── test_rclnodejs -- package for Python client
     └── test_rclpy -- package for Node.js client
     └── ...

The next step is how we define these performance tests (I have written some under rclnodejs repo), I suppose that all tests of the clients must follow the same strategy in order to compare the statistics. Do we have any references?

[dirk-thomas]

The next step is how we define these performance tests (I have written some under rclnodejs repo), I suppose that all tests of the clients must follow the same strategy in order to compare the statistics. Do we have any references?

Since we don't have any performance tests at the moment there is no reference. I would suggest you propose something after checking what has been done in ROS 1 or similar projects in the past.

[minggangw]

That' no problem, I will collaborate with our QA team to run the current PnP test designed for rclnodejs, and if the result can give some meaningful data, I will propose some then.