Revamp CMake to make compilation significantly faster #4357

clalancette · 2024-05-21T01:52:44Z

Basic Info

Info	Please fill out this column
Ticket(s) this addresses	N/A
Primary OS tested on	Ubuntu 22.04
Robotic platform tested on	N/A
Does this PR contain AI generated software?	No

Description of contribution in a few bullet points

This very large PR deserves a long explanation, so please bear with me.

I was recently using navigation2 along with the turtlebot3 simulation on ROS 2 Iron, and I noticed that compiling navigation2 took ages. In particular, it looks like compiling nav2_system_tests is extremely slow.

That led me to look into how things are being compiled. Besides being slow, I noticed that navigation2 isn't using many of what we consider the "modern" CMake practices in ROS 2. Thus, this PR implements the following things:

Make sure to generate CMake "targets" in all packages. We do this so that downstream packages only need to link against the exact target that they need, and the individual targets only export the things that are PUBLIC. This is the main source of the speedup of compilation in this PR. The way this is done is by using the EXPORT command to install, and also calling ament_export_targets to generate the appropriate CMake glue.
Removal of ament_target_dependencies in favor of target_link_libraries. ament_target_dependencies was created in the days before target_link_libraries, and as such doesn't have as many features. In particular, ament_target_dependencies works on the concept of packages, while target_link_libraries works on the concept of "targets", which can have arbitrary PUBLIC and PRIVATE linked libraries and header files. I'll note that this switch in focus does have a downside, in that the dependencies that we link against are targets while the dependencies we export (via ament_export_dependencies) are package names.
Switch to installing header files in include/${PROJECT_NAME}/${PROJECT_NAME}. This was a change we pushed in Humble, and ensures that overlays work in all situations. You can see more about why we did this in http://docs.ros.org/en/humble/Releases/Release-Humble-Hawksbill.html#c-headers-are-installed-in-a-subdirectory and Overlaying packages using CMake export targets can fail with merge install underlay ros2/ros2#1150 .
Remove include_directories in favor of target_include_directories. This is best practice nowadays to more precisely control exactly what include directories are used.
Cleanup of dependencies in general, adding them where needed and removing them if not needed.
Cleanup of package.xml files, ensuring that we use depend as needed (which implies build_export_depend as well as build_depend and exec_depend).
Addition of the proper #include in C++ and header files. Not only is this cleaner, it also makes it much easier to determine which dependencies should be PUBLIC, and which ones PRIVATE.

With all of these cleanups in place, on my local machine compilation went from over 22 minutes to 9 minutes.

This PR targets the iron branch, as that is what I was using for my development. But I can retarget this to another branch depending on the development practices for this repository.

Note that I do not expect this PR to go in as-is (which is why it is a draft). I didn't even run any tests on this yet, and it requires both ros/bond_core#94 and BehaviorTree/BehaviorTree.CPP#826 before it will even compile. But I wanted to get the reaction of the developers here to the "shape" of what this will look like, and get early feedback. If this is something that the maintainers don't want, then I'll just close this. If the maintainers are OK with how this generally looks, then I'll probably open a separate PR per package, implementing the changes.

Thoughts?

Description of documentation updates required from your changes

Future work that may be required in bullet points

For Maintainers:

Check that any new parameters added are updated in docs.nav2.org
Check that any significant change is added to the migration guide
Check that any new features OR changes to existing behaviors are reflected in the tuning guide
Check that any new functions have Doxygen added
Check that any new features have test coverage
Check that any new plugins is added to the plugins page
If BT Node, Additionally: add to BT's XML index of nodes for groot, BT package's readme table, and BT library lists

This commit does a number of things: 1. Switches to using target_link_libraries everywhere. This gives us finer-grained control over what dependencies are exported to downstream as public, or private. In the particular case of nav2_util, this actually doesn't matter *too* much, but it will help for other packages. 2. Moves the include directory down one level to include/${PROJECT_NAME}, which is best practice in ROS 2 since Humble. 3. Removes the separate src/CMakeLists.txt. While this isn't strictly necessary, it makes it a lot easier to do the rest of the changes in here. 4. Makes sure to export nav2_util as a CMake target, so downstream users of it can use that target. Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

mergify · 2024-05-21T01:53:25Z

@clalancette, all pull requests must be targeted towards the main development branch.
Once merged into main, it is possible to backport to @iron, but it must be in main
to have these changes reflected into new distributions.

SteveMacenski · 2024-05-21T03:15:04Z

nav2_amcl/CMakeLists.txt

-add_subdirectory(src/motion_model)
-add_subdirectory(src/sensors)
+if(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+  add_compile_options(-Wno-gnu-folding-constant)


Can you explain this added for AMCL specifically?

Yeah, this is a good question. The honest truth is that for reasons I don't totally understand, having the targets in the subdirectory doesn't produce nav2_amcl targets that "worked". In particular, the downstream package linking against it (nav2_system_tests) would fail to properly find the target. Moving this all together into the main CMakeLists.txt seemed to make it work. It bears more investigation into why that is happening.

SteveMacenski · 2024-05-21T03:17:44Z

nav2_behavior_tree/CMakeLists.txt

+    PRIVATE
+    "$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>"
+    "$<INSTALL_INTERFACE:include/${PROJECT_NAME}>")
+  target_link_libraries(${bt_plugin} PUBLIC


Question: Is there much of a speed up by specifying the exact targets vs all the targets in the packages? It is somewhat convenient to be able to specify the dependencies variable and just target the set as a whole without having to think about it much. But if this is a big speed up, that is a worthwhile trade off to me to specify them individually.

So yeah, you get most of the speed-up by specifying the exact targets, and splitting this into PUBLIC and PRIVATE. That is, anything that is marked as PRIVATE will not be exposed to the downstream targets, thus shortening the list of things they have to link against later on.

Unfortunately, as it stands, ament_target_dependencies does not fully support this (in particular, it doesn't support PRIVATE). We could add it to ament_target_dependencies, but since we are trying to slowly move away from it, I didn't think it was particularly worth it.

While thinking about it, I did come up with another way to do this, which would reduce duplication. Inside of nav2_common, we could have a "map" between a package name and target names. And then we could have a custom cmake macro/function here that, given a target and a list of packages, automatically looked up the mapping between a package name and a target name, and called target_link_libraries. The caller would call something like nav2_link_libraries(TARGET mytarget PACKAGES foo bar). The downside is that it is more CMake "magic", which is is something that people have complained about for years with ament. I also don't know what the impact on compile times will be. Nevertheless, if you think this is interesting, I can look into prototyping it.

SteveMacenski · 2024-05-21T03:22:02Z

First of all; thanks!

I certainly don't mind the update (I know this will make @Ryanf55 very happy), I just want the stack to be consistent; which this does since you touch all the packages! Happy for faster compiling and more modern cmake if we can get the full stack over to it.

SteveMacenski · 2024-05-21T03:23:39Z

nav2_behavior_tree/include/nav2_behavior_tree/bt_action_server_impl.hpp

 #include "nav2_util/node_utils.hpp"
+#include "rcl_action/action_server.h"
+#include "rclcpp_lifecycle/lifecycle_node.hpp"


I'm curious how you identified the headers to remove or the C libs to add in in place of others (here and in other files)? I expect this was a bit trial and error as you were adding in the specific targets within each package, but if there was something more automated you used that would be good for me to have in my toolbox as a future reviewer of changes to this codebase after this PR

Good question.

It was completely manual for right now. For further work here, I'm going to look into doing something much more automated, because it's not particularly sustainable with the amount of code that exists in this repository.

Ryanf55

Nice, this is a much needed improvement to bring NAV2 in line with the newer conventions in the ament tutorials. This is similar to a few other pull requests I have done in NAV2. Overall, I approve the intent of this work, but cannot do a detailed review of every line of code change. Nice benchmarks on the system tests. Turns out, not linking everything to everything speeds up the build!

The changes from boost filesystem to std::filesystem were a good change too.

Ryanf55 · 2024-05-21T05:56:16Z

nav2_util/CMakeLists.txt

 set(library_name ${PROJECT_NAME}_core)
-add_subdirectory(src)
+
+add_library(${library_name} SHARED


Why hard-code SHARED? This breaks the ability for people to compile the util as a static lib, right?

I carried this over from what existed already. Actually, another good change we could make here would be to switch these packages over to ament_cmake_ros, which automatically handles making libraries shared by default, but also allows using -DBUILD_SHARED=OFF.

Ryanf55 · 2024-05-21T05:57:39Z

nav2_util/CMakeLists.txt

-add_subdirectory(src)
+
+add_library(${library_name} SHARED
+  src/costmap.cpp


I find it nicer to use target_sources in another CMakeLists in the src directory because then you can just copy/paste the output of ls -1 *.cpp into the CMakeLists, and diffs are isolated to the directories that changed, but I understand you might be trying to keep par with how the ROS core repos do it all at the top-level CMakeLists.

Yeah, my goal was to keep it mostly the same as the core.

In my opinion, the way we split up packages in ROS 2 actually makes it less interesting to have separate CMakeLists.txt in separate directories. That is, most packages are relatively small, and thus having everything in one CMakeLists.txt at the top-level is viable for most packages. It also removes one of my pet peeves with sub-CMakeLists.txt, where they often rely on "magic" variables inherited from the parent.

All of that said, I can move things back down into the individual sub-directories here, depending on maintainer preference. It isn't a core part of speeding things up.

Ryanf55 · 2024-05-21T05:59:11Z

nav2_system_tests/CMakeLists.txt

@@ -52,8 +52,7 @@ set(dependencies
 set(local_controller_plugin_lib local_controller)

 add_library(${local_controller_plugin_lib} SHARED src/error_codes/controller/controller_error_plugins.cpp)
-ament_target_dependencies(${local_controller_plugin_lib} ${dependencies})
-target_compile_definitions(${local_controller_plugin_lib} PUBLIC "PLUGINLIB__DISABLE_BOOST_FUNCTIONS")
+target_link_libraries(${local_controller_plugin_lib} PRIVATE nav2_core::nav2_core nav2_costmap_2d::nav2_costmap_2d_core ${nav_msgs_TARGETS} ${geometry_msgs_TARGETS} pluginlib::pluginlib)


Some of these I would do a line limit and break up to one dependency per line so it fits on an editor without wrap.

Yeah, 100%. When I open up separate PRs, I'll definitely do that and run them through the linters.

Ryanf55 · 2024-05-21T06:02:15Z

nav2_dwb_controller/nav_2d_utils/CMakeLists.txt

+  ${nav_2d_msgs_TARGETS}
+  tf2::tf2
+  ${tf2_geometry_msgs_TARGETS}
+  conversions


Is there an alias target available or one that could be created to reduce the chances of linker errors due to typos on the library name?

So this is one oddity of CMake. Any libraries we depend on from this package haven't yet had their targets created, so we can't use the traditional nav_2d_utils::conversions target like we would in downstream packages. Instead, we just have to use conversions.

While we could add in aliases, I think it is actually more confusing because those aliases clash with the eventual targets that would be created. I guess we could create an alias called internal::conversions, though I'm not sure it is worth it. It might be a better idea to rename the library to something more descriptive, though that could also have downstream implications.

I've gotten ALIAS targets working well in about 120 packages. If you do the ALIAS as ${PROJECT_NAME}::your_target, then the build target name is the same as ament's exported name. From what I've seen, ALIAS targets don't show up in the install interface, and naming them the same is recommended: https://youtu.be/bsXLMQ6WgIk?si=SPk624LzEa2UhLU5&t=3133

I have yet to have issues doing this method, and will routinely create ALIAS targets so my code in test can link to the right library with configure errors if there are typos.

Ryanf55 · 2024-05-21T06:03:28Z

nav2_controller/package.xml

@@ -19,6 +19,10 @@
  <depend>nav_2d_msgs</depend>
  <depend>nav2_core</depend>
  <depend>pluginlib</depend>
+  <depend>lifecycle_msgs</depend>


How is bloom missing these on iron and passing the build?

Yeah, it's a good question.

Most likely, these are being pulled in by something else. In this case, it looks like nav2_util depends on lifecycle_msgs, and so nav2_controller inherits it from there. But since this package directly depends on it, we should technically have this dependency here as well.

Ah ok. I thought bloom was supposed to catch transitive dependencies. I must be remembering wrong. Good catch!

Ah ok. I thought bloom was supposed to catch transitive dependencies. I must be remembering wrong. Good catch!

It's complicated. In point of fact, bloom won't check at all that your dependencies are "correct", it will only check that your dependencies can be resolved into system or ROS packages.

The buildfarm goes further, in that it when it goes to build your package, it will start with an absolutely minimal Ubuntu container, and just apt-get install exactly what you put in your package.xml. If a dependency is missing from direct and indirect dependencies, it will definitely find that. But if you happen to pull in a package via an indirect dependency, things will just work. That's what is happening here.

Thank you, that clears it up. Good finds on catching all the missing ones.

clalancette · 2024-05-21T13:14:48Z

I certainly don't mind the update (I know this will make @Ryanf55 very happy), I just want the stack to be consistent; which this does since you touch all the packages! Happy for faster compiling and more modern cmake if we can get the full stack over to it.

Just to be totally clear on this point; I didn't convert all of navigation2 here, though I did do most of it. If we want to go down this route, then I'll continue working on this, as there are probably 6 or so packages left to do.

SteveMacenski · 2024-05-21T16:06:39Z

The change is definitely welcome!

mergify · 2024-05-23T23:35:08Z

This pull request is in conflict. Could you fix it @clalancette?

ruffsl · 2024-05-24T02:10:48Z

With all of these cleanups in place, on my local machine compilation went from over 22 minutes to 9 minutes.

Thanks so much @clalancette! Build time were getting out of hand, but my hate for CMake left me despondent.

SteveMacenski · 2024-05-24T19:32:18Z

The changes from boost filesystem to std::filesystem were a good change too.

It wasn't in the version of C++ when that part of the code was written 🥲 But yes, kill boost whereever possible!

@clalancette move forward with this as you see fit, I'm not a backseat driver. I'll review / respond as you're ready

clalancette · 2024-05-24T20:23:52Z

@clalancette move forward with this as you see fit, I'm not a backseat driver. I'll review / respond as you're ready

Thanks! I've been actually working on some tooling for this in the background, which is why I've been kind of quiet here. I'm on vacation next week, but I'll pick up work on that tooling (and updating this PR) once I'm back.

clalancette added 22 commits May 17, 2024 20:47

Update nav2_behavior_tree to use modern CMake idioms.

02efe70

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_voxel_grid to modern CMake idioms.

9716133

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_map_server to modern CMake idioms.

55d26a9

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_amcl to modern CMake idioms.

096928b

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Convert nav2_costmap_2d to modern CMake idioms.

3679802

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Small updates to nav2_core.

58d9e36

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Update nav2_behaviors to use modern CMake idioms.

e55df65

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_bt_navigator to modern CMake idioms.

a98e0b3

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Swithc nav2_collision_monitor to modern CMake idioms.

92301f2

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Revamp nav2_constrained_smoother to use modern CMake idioms.

8044841

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav_2d_utils to use modern CMake idioms.

63d3b6f

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_controller to modern CMake idioms.

9d3e628

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_lifecycle_manager to modern CMake idioms.

9e3ed8d

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_mppi_controller to use modern CMake idioms.

2f5fdfe

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_navfn_planner to modern CMake idioms.

eda8ba3

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_planner to modern CMake idioms.

3b87d7c

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_regulated_pure_pursuit_controller to modern CMake idioms.

7e66b91

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_rotation_shim_controller to modern CMake idioms.

55477b5

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_rviz_plugins to modern CMake idioms.

a30efd0

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Rewrite nav2_system_tests to use modern CMake idioms.

2038f2b

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

Switch nav2_smoother to modern CMake idioms.

41ee273

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

SteveMacenski reviewed May 21, 2024

View reviewed changes

Ryanf55 reviewed May 21, 2024

View reviewed changes

clalancette mentioned this pull request Jun 3, 2024

Revamp nav2_util CMakeLists.txt to use modern idioms. #4393

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revamp CMake to make compilation significantly faster #4357

Revamp CMake to make compilation significantly faster #4357

clalancette commented May 21, 2024

mergify bot commented May 21, 2024

SteveMacenski May 21, 2024 •

edited

clalancette May 21, 2024

SteveMacenski May 21, 2024

clalancette May 21, 2024

SteveMacenski commented May 21, 2024 •

edited

SteveMacenski May 21, 2024

clalancette May 21, 2024

Ryanf55 left a comment

Ryanf55 May 21, 2024

clalancette May 21, 2024

Ryanf55 May 21, 2024

clalancette May 21, 2024

Ryanf55 May 21, 2024

clalancette May 21, 2024

Ryanf55 May 21, 2024

clalancette May 21, 2024

Ryanf55 May 21, 2024

Ryanf55 May 21, 2024

clalancette May 21, 2024

Ryanf55 May 21, 2024

clalancette May 24, 2024

Ryanf55 May 24, 2024

clalancette commented May 21, 2024

SteveMacenski commented May 21, 2024

mergify bot commented May 23, 2024

ruffsl commented May 24, 2024

SteveMacenski commented May 24, 2024

clalancette commented May 24, 2024

Revamp CMake to make compilation significantly faster #4357

Are you sure you want to change the base?

Revamp CMake to make compilation significantly faster #4357

Conversation

clalancette commented May 21, 2024

Basic Info

Description of contribution in a few bullet points

Description of documentation updates required from your changes

Future work that may be required in bullet points

For Maintainers:

mergify bot commented May 21, 2024

SteveMacenski May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SteveMacenski commented May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ryanf55 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clalancette commented May 21, 2024

SteveMacenski commented May 21, 2024

mergify bot commented May 23, 2024

ruffsl commented May 24, 2024

SteveMacenski commented May 24, 2024

clalancette commented May 24, 2024

SteveMacenski May 21, 2024 •

edited

SteveMacenski commented May 21, 2024 •

edited