Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Type Description Codegen and Typesupport (rep2011) #727

Merged

Conversation

emersonknapp
Copy link
Collaborator

@emersonknapp emersonknapp commented Mar 15, 2023

Part of ros2/ros2#1159

Bundled with (must be merged together):

Key features:

  • Changes TYPE_HASH_TUPLES to more accurate TYPE_DESCRIPTION_TUPLES in cmake
  • Adds new C interface functions get_type_description, get_individual_type_description_source, and get_type_description_sources
  • Implements above in idl__description.c.em, generating code for type descriptions and raw interface definition sources, embedded in the binaries
  • To avoid duplicate generation in C, adds a dependency by C++ typesupport generators to C generated code to retrieve description and sources pointers
  • Optionally disables embedding that information in the generated code by the CMake flag ROSIDL_GENERATOR_C_DISABLE_TYPE_DESCRIPTION_CODEGEN
  • Refactors rosidl_generator_type_description to output a new structure of type_description_msg plus a flat list of all referenced types and their hashes at the time of generation. These hashes are used at runtime (in Debug builds) to detect ABI breaks when constructing type descriptions the first time

@methylDragon
Copy link
Contributor

methylDragon commented Mar 15, 2023

Gonna drop this in here before I forget.
Copying works, but it causes the linters to go haywire (because the code generated sources and headers trip up the linters (copyright, cpplint, and uncrustify, specifically.)

Unfortunately there isn't a way for files to get excluded from the linters yet, so we might have to just live with this regression for now (or turn off the linting tests)...

@emersonknapp
Copy link
Collaborator Author

@methylDragon There are only two problems that the generated code is allowed to get away with already: line length and missing copyright declaration. I can fix the copyright easily by running ament_copyright --add-missing on the copied sources, and I should be able to find a way around the line length issue.

"living with the regression" isn't an option, nor is turning off the linters for the package - I just hadn't got as far as making all tests happy in this PR yet.

@emersonknapp
Copy link
Collaborator Author

Whoops update - actually it looks like the capability you talked about was implemented in ament/ament_lint#386 and you even personally signed off on it :)

I've added a commit excluding the copied files from the linters and it works. I've got a lingering question of whether we should run linters here. They are linted in the place they're originally created (minus copyright and with no line length limit) so as long as the script's modifications are considered fine then I'm leaning toward just excluding them. Or, we could run a few extra linter checks on just the copied files - some other packages do this for generated files, like https://github.com/ros2/rcutils/blob/rolling/CMakeLists.txt#L172

@methylDragon
Copy link
Contributor

methylDragon commented Mar 16, 2023

Whoops update - actually it looks like the capability you talked about was implemented in ament/ament_lint#386 and you even personally signed off on it :)

I've added a commit excluding the copied files from the linters and it works. I've got a lingering question of whether we should run linters here. They are linted in the place they're originally created (minus copyright and with no line length limit) so as long as the script's modifications are considered fine then I'm leaning toward just excluding them. Or, we could run a few extra linter checks on just the copied files - some other packages do this for generated files, like ros2/rcutils@rolling/CMakeLists.txt#L172

WHOOPS 🤦 , nice catch. I should go update the issue...

Edit: Thanks for updating the issue

@clalancette
Copy link
Contributor

They are linted in the place they're originally created (minus copyright and with no line length limit) so as long as the script's modifications are considered fine then I'm leaning toward just excluding them. Or, we could run a few extra linter checks on just the copied files - some other packages do this for generated files, like https://github.com/ros2/rcutils/blob/rolling/CMakeLists.txt#L172

Yeah, I'm in favor of linting them, even though it is a bit redundant. But I don't have a strong opinion.

Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Copy link
Member

@wjwwood wjwwood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, need to review (and merge into this) a version of #735 before approving this.

const rosidl_runtime_c__type_description__TypeSource__Sequence *
@(td_c_typename)__@(GET_SOURCES_FUNC)()
{
@# TODO(emersonknapp) Implement raw source code embedding/generation. This sequence is left empty for now.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this handled by the later prs?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@@ -0,0 +1,114 @@
// Copyright 2015 Open Source Robotics Foundation, Inc.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this should be updated?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should, I'll do it in emersonknapp#10

Comment on lines 115 to 127
add_executable(
test_descriptions_c
test/rosidl_generator_c/test_descriptions.c
)
ament_add_test(
test_descriptions_c
COMMAND "$<TARGET_FILE:test_descriptions_c>"
GENERATE_RESULT_FOR_RETURN_CODE_ZERO
)
target_link_libraries(test_descriptions_c
${c_generator_target}
rosidl_runtime_c::rosidl_runtime_c
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you trying to avoid C++/gtest? I'm not sure that's a requirement. I mean, it's done now, no need to change it, but I was just curious.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the other tests here are plain C so I was just following pattern rather than go off-script with C++/gtest dependency

Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks generally good to me. I'm holding off on approving it for now until we merge the follow-up PR into this one.

…n-include-nested

Extend TypeDescription codegen with referenced type inclusion, disable option, raw sources
@clalancette
Copy link
Contributor

I did two additional pieces of analysis here:

  1. I built the whole set of PRs under clang just to get thoughts from a static analyzer. I'm happy to report that there are no warnings introduced by this set of changes.
  2. I went back one more time to compare the binary sizes before and after this set of PRs. In the table below, "Before" is "Before this set of PRs", "Old" is "Using this PR prior to merging Extend TypeDescription codegen with referenced type inclusion, disable option, raw sources emersonknapp/rosidl#10", "New" is the current set of PRs, and "New Disabled" is the current set of PRs with codegen disabled (--disable-description-codegen forced on). The numbers in each column are the raw binary sizes, and the numbers in parentheses are the percentage larger than the "Before".
File Before Size Old PRs Size New Size New Size disabled
install/actionlib_msgs/lib/libactionlib_msgs__rosidl_generator_c.so 36568 (0.00) 48432 (32.44) 48376 (32.29) 37560 (2.71)
install/action_msgs/lib/libaction_msgs__rosidl_generator_c.so 52840 (0.00) 80064 (51.52) 88232 (66.98) 59376 (12.37)
install/action_tutorials_interfaces/lib/libaction_tutorials_interfaces__rosidl_generator_c.so 81160 (0.00) 113648 (40.03) 133664 (64.69) 95944 (18.22)
install/builtin_interfaces/lib/libbuiltin_interfaces__rosidl_generator_c.so 21952 (0.00) 32936 (50.04) 29128 (32.69) 26976 (22.89)
install/composition_interfaces/lib/libcomposition_interfaces__rosidl_generator_c.so 74760 (0.00) 110616 (47.96) 126720 (69.50) 87960 (17.66)
install/diagnostic_msgs/lib/libdiagnostic_msgs__rosidl_generator_c.so 73296 (0.00) 110656 (50.97) 114552 (56.29) 81816 (11.62)
install/example_interfaces/lib/libexample_interfaces__rosidl_generator_c.so 304856 (0.00) 463504 (52.04) 489280 (60.50) 369136 (21.09)
install/geometry_msgs/lib/libgeometry_msgs__rosidl_generator_c.so 191368 (0.00) 329616 (72.24) 305136 (59.45) 213240 (11.43)
install/lifecycle_msgs/lib/liblifecycle_msgs__rosidl_generator_c.so 111176 (0.00) 174128 (56.62) 197968 (78.07) 135744 (22.10)
install/logging_demo/lib/liblogging_demo__rosidl_generator_c.so 27488 (0.00) 37432 (36.18) 46688 (69.85) 33936 (23.46)
install/map_msgs/lib/libmap_msgs__rosidl_generator_c.so 156360 (0.00) 261496 (67.24) 275336 (76.09) 182504 (16.72)
install/nav_msgs/lib/libnav_msgs__rosidl_generator_c.so 131008 (0.00) 235224 (79.55) 252792 (92.96) 144448 (10.26)
install/pendulum_msgs/lib/libpendulum_msgs__rosidl_generator_c.so 31752 (0.00) 42768 (34.69) 42032 (32.38) 32984 (3.88)
install/rcl_interfaces/lib/librcl_interfaces__rosidl_generator_c.so 204272 (0.00) 326360 (59.77) 346192 (69.48) 239272 (17.13)
install/rmw_dds_common/lib/librmw_dds_common__rosidl_generator_c.so 31864 (0.00) 41832 (31.28) 40896 (28.35) 33176 (4.12)
install/ros2cli_test_interfaces/lib/libros2cli_test_interfaces__rosidl_generator_c.so 120200 (0.00) 179384 (49.24) 206976 (72.19) 146696 (22.04)
install/rosbag2_interfaces/lib/librosbag2_interfaces__rosidl_generator_c.so 261752 (0.00) 403672 (54.22) 440536 (68.30) 320800 (22.56)
install/rosbag2_storage_mcap_testdata/lib/librosbag2_storage_mcap_testdata__rosidl_generator_c.so 38768 (0.00) 54144 (39.66) 53904 (39.04) 49384 (27.38)
install/rosgraph_msgs/lib/librosgraph_msgs__rosidl_generator_c.so 17064 (0.00) 19112 (12.00) 18960 (11.11) 17456 (2.30)
install/sensor_msgs/lib/libsensor_msgs__rosidl_generator_c.so 207264 (0.00) 353776 (70.69) 376544 (81.67) 234272 (13.03)
install/service_msgs/lib/libservice_msgs__rosidl_generator_c.so 17216 (0.00) 19976 (16.03) 24416 (41.82) 17656 (2.56)
install/shape_msgs/lib/libshape_msgs__rosidl_generator_c.so 37336 (0.00) 52696 (41.14) 53672 (43.75) 42920 (14.96)
install/statistics_msgs/lib/libstatistics_msgs__rosidl_generator_c.so 32264 (0.00) 42352 (31.27) 48312 (49.74) 33608 (4.17)
install/std_msgs/lib/libstd_msgs__rosidl_generator_c.so 181624 (0.00) 280104 (54.22) 275872 (51.89) 213744 (17.68)
install/std_srvs/lib/libstd_srvs__rosidl_generator_c.so 71592 (0.00) 96552 (34.86) 108888 (52.10) 80296 (12.16)
install/stereo_msgs/lib/libstereo_msgs__rosidl_generator_c.so 17952 (0.00) 28656 (59.63) 31464 (75.27) 22216 (23.75)
install/test_msgs/lib/libtest_msgs__rosidl_generator_c.so 323232 (0.00) 593656 (83.66) 606120 (87.52) 380960 (17.86)
install/tf2_msgs/lib/libtf2_msgs__rosidl_generator_c.so 111264 (0.00) 177472 (59.51) 205368 (84.58) 131240 (17.95)
install/trajectory_msgs/lib/libtrajectory_msgs__rosidl_generator_c.so 48168 (0.00) 78488 (62.95) 74064 (53.76) 49000 (1.73)
install/turtlesim/lib/libturtlesim__rosidl_generator_c.so 176856 (0.00) 267600 (51.31) 295568 (67.12) 216184 (22.24)
install/type_description_interfaces/lib/libtype_description_interfaces__rosidl_generator_c.so 74440 (0.00) 120368 (61.70) 124720 (67.54) 82904 (11.37)
install/unique_identifier_msgs/lib/libunique_identifier_msgs__rosidl_generator_c.so 16800 (0.00) 18264 (8.71) 18184 (8.24) 17304 (3.00)
install/visualization_msgs/lib/libvisualization_msgs__rosidl_generator_c.so 138696 (0.00) 302600 (118.18) 276440 (99.31) 140880 (1.57)

As a general comment, I would say that the "new" set of PRs is somewhat better on binary size than the original set for the largest libraries, but worse for smaller libraries. It is clear that disabling codegen helps significantly, so I'm glad we added that option.

So with that said, I'm going to approve this PR and work on reviewing the rest of the set. I'd like to get @wjwwood 's final approval as well, and of course we'll need new CI.

@emersonknapp
Copy link
Collaborator Author

emersonknapp commented Apr 5, 2023

the "new" set of PRs is somewhat better on binary size than the original set for the largest libraries, but worse for smaller libraries. It is clear that disabling codegen helps significantly, so I'm glad we added that option.

Note that the "new" PRs also now contain raw sources, where the "old" ones did not have that data so that muddies the comparison a bit unfortunately.

Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
@emersonknapp
Copy link
Collaborator Author

Gist: https://gist.githubusercontent.com/emersonknapp/3d42343216f83e16ec6b739829a33a15/raw/710780b6ce1a1318b21363581bad84a247f92bb4/ros2.repos
BUILD args:
TEST args:
ROS Distro: rolling
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/11795

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

@clalancette
Copy link
Contributor

Just because of the size of this series, I'm going to run RHEL and Windows Debug as well:

  • RHEL Build Status
  • Windows Debug Build Status

@clalancette
Copy link
Contributor

And we're green! I'm going to go ahead and merge the lot, and cross my fingers for overnight CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants