Foxy | Regression: Default RMW no longer ships with DDS security features

[ruffsl]

Bug report

Not sure if this is the best place to report this, but I think the default rmw for Foxy was re-released without DDS security enabled?

Required Info:

• Operating System:
  • Ubuntu 20.04
• Installation type:
  • binaries
• Version or commit hash:
  • ros-foxy-sros2/focal,now 0.9.4-1focal.20210125.232905 amd64 [installed,automatic]
  • ros-foxy-fastrtps/focal,now 2.0.2-1focal.20201210.051121 amd64 [installed,automatic]
• DDS implementation:
  • Fast-RTPS
• Client library (if applicable):
  • N/A

Steps to reproduce issue

Follow the demo documented here:

https://github.com/ros2/sros2/blob/foxy/SROS2_Linux.md

Expected behavior

Security works and demo nodes connect and exchange messages via Secure DDS.

Actual behavior

$ ros2 run demo_nodes_cpp talker --ros-args --enclave /talker_listener/talker
[INFO] [1614033471.188423658] [rcl]: Found security directory: /home/ruffsl/sros2_demo/demo_keys/enclaves/talker_listener/talker

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'This Fast-RTPS version doesn't have the security libraries
Please compile Fast-RTPS using the -DSECURITY=ON CMake option, at /tmp/binarydeb/ros-foxy-rmw-fastrtps-shared-cpp-1.2.4/src/participant.cpp:248, at /tmp/binarydeb/ros-foxy-rcl-1.1.10/src/rcl/node.c:276'

with this new error message:

  'rcl node's rmw handle is invalid, at /tmp/binarydeb/ros-foxy-rcl-1.1.10/src/rcl/node.c:428'

rcutils_reset_error() should be called after error handling to avoid this.
<<<
[ERROR] [1614033471.193580788] [rcl]: Failed to fini publisher for node: 1
terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
  what():  failed to initialize rcl node: rcl node's rmw handle is invalid, at /tmp/binarydeb/ros-foxy-rcl-1.1.10/src/rcl/node.c:428

Additional information

Discovered while update sros2 demos for latest ros:foxy images:

ros-swg/turtlebot3_demo#34 (comment)

[kyrofa]

@JaimeMartin did you folks change your default config by any chance?

[clalancette]

This looks like probably a packaging bug. If I look at https://build.ros2.org/view/Fbin_uF64/job/Fbin_uF64__fastrtps__ubuntu_focal_amd64__binary/9/consoleFull (the fastrtps build back in May), I see this:

cd obj-x86_64-linux-gnu && cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=None 
-DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_LOCALSTATEDIR=/var 
-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON 
"-GUnix Makefiles" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_AUTOGEN_VERBOSE=ON 
-DCMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu -DCMAKE_INSTALL_PREFIX=/opt/ros/foxy -DINSTALL_EXAMPLES=OFF 
-DSECURITY=ON ..

If I look at the latest build at https://build.ros2.org/view/Fbin_uF64/job/Fbin_uF64__fastrtps__ubuntu_focal_amd64__binary/24/consoleFull , I see this:

cd obj-x86_64-linux-gnu && cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=None 
-DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_LOCALSTATEDIR=/var 
-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON 
"-GUnix Makefiles" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_AUTOGEN_VERBOSE=ON 
-DCMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu -DCMAKE_INSTALL_PREFIX=/opt/ros/foxy
 -DCMAKE_PREFIX_PATH=/opt/ros/foxy ..

Notice that -DSECURITY=ON is missing from that latter line. Probably something needs to be fixed in https://github.com/ros2-gbp/fastrtps-release , though I'm not sure what. @jacobperron @nuclearsandwich any thoughts here?

[clalancette]

It looks like a bad rebase from the last release: ros2-gbp/fastrtps-release@0ed7d05 removed the template that had -DSECURITY=ON, and ros2-gbp/fastrtps-release@1e8b0d3 readded the template without -DSECURITY=ON. We need a PR to the release repo there to restore the -DSECURITY=ON line, but I'll wait for instructions from @nuclearsandwich on the best way to do that.

[kyrofa]

Thanks for the quick triage @clalancette!

[clalancette]

It looks like Dashing probably has the same problem (https://build.ros2.org/view/Dbin_uB64/job/Dbin_uB64__fastrtps__ubuntu_bionic_amd64__binary/19/consoleFull), though Rolling is fine.

[nuclearsandwich]

I'll wait for instructions from @nuclearsandwich on the best way to do that.

The debian/$ROSDISTRO/$PACKAGE branch contains the Debian metadata templates. PRs modifying rules.em in Dashing and Foxy followed by a re-blooming to create a pkgrev/debinc release will fix the packages.

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/preparing-for-foxy-sync-2021-03-01/19178/1

[clalancette]

All right, this has been fixed in the testing repositories for both Foxy and Dashing. A sync is scheduled for Foxy. I'm going to close this issue out, thanks for reporting @ruffsl .

[ruffsl]

@clalancette , do we know yet how this regression slipped under the radar of existing tests and made it's way into a released public sync? Did the system security tests in CI silently fail? What should we do now to prevent this again?

[mikaelarguedas]

Debian packages don't get tested that's why the regression was not caught.
We would either need a registered package (with PR and / or dev jobs) depending on and testing fastrtps security that got triggered by a new commit, or have some type of nightly that pulls packages from debs and build + test test_security on top.

Not sure how the regression actually happened, as far as I recall patches always got reapplied on rebase when making a new release. @nuclearsandwich @clalancette do you know what happened there and how to prevent it from happening in the future ?

As a side note: 49,419 additions and 5,228 deletions. seems like an incredibly large "patch" for an already out LTS release..

[clalancette]

It's a combination of 3 things:

1. We run CI from https://ci.ros2.org on all releases, but that is a build-and-test-from-source, not a test of the Debian packages. As Mikael says, there is no automated testing of the Debian packages.
2. Nobody manually tests the Debian packages in the testing repositories.
3. The way we ensure that -DSECURITY=ON is enabled while building the debian packages is through patches to the release repositories. These are somewhat easy to lose for someone who isn't intimately familiar with the process and how the fastrtps package in particular is setup.

Improving any of those would help prevent this from happening again. Unfortunately, none of them are easy to do.

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/new-packages-for-foxy-fitzroy-2021-03-01/19224/1

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/preparing-for-dashing-sync-2021-03-04/19233/1

[AnushaPulichintha]

Hi, I recently installed foxy on ubuntu 20.04 from binaries. I still have this problem. How do i get the updated packages for fastrtps?

[ruffsl]

You may want to check what package version you have installed with apt. This is what is in the repo as of writing:
http://packages.ros.org/ros2/ubuntu/dists/focal/main/binary-amd64/Packages

Package: ros-foxy-fastrtps
Version: 2.0.2-2focal.20210423.001712
Architecture: amd64
Maintainer: Steven! Ragnarök <stevenragnarok@osrfoundation.org>
Installed-Size: 10752
Depends: libc6 (>= 2.29), libgcc-s1 (>= 3.0), libssl1.1 (>= 1.1.1), libstdc++6 (>= 9), libtinyxml2-6a (>= 5.0.0), ros-foxy-fastcdr, libssl-dev, libtinyxml2-dev, ros-foxy-foonathan-memory-vendor, ros-foxy-ros-workspace
Priority: optional
Section: misc
Filename: pool/main/r/ros-foxy-fastrtps/ros-foxy-fastrtps_2.0.2-2focal.20210423.001712_amd64.deb
Size: 2483772
SHA256: 6743b30df3ac27746175bcb2464299c2d3e3af1be9a8a02179de613c23b4a431
SHA1: c0863876b0dbf48b782f97ee59b04d635cb58359
MD5sum: da6fc30138d3a8b5b859c476dc686e6c
Description: Implementation of RTPS standard.

[AnushaPulichintha]

Hi, I just checked it again. It's working now, seems the problem is with my security keys.
Sorry about that. Thanks for the quick reply.

[alexleel]

I also met this issue, the command I use as following:
git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat clone -b ros-foxy-fastrtps --bare --mirror https://github.com/ros2-gbp/rmw_fastrtps-release /home/build_sros/downloads/git2/github.com.ros2-gbp.rmw_fastrtps-release I checked the version is 1.2.6-1-r0, how can I slove this problem？ which version can work?