[dunfell][gatesgarth][hardknott][honister][kirkstone] Add 'nativesdk' to some python package recipes to fix colcon nativesdk recipe #988

RiZom-91 · 2021-12-29T13:20:13Z

Following discussion on PR #977 , I've made the changes required to fix colcon nativesdk recipe.

windelbouwman

Looks good to me, thanks!

RiZom-91 · 2021-12-30T08:32:34Z

I've tried to continue this work with the SDK. To do so, I have a workspace which contains demos and example_interfaces packages. Below is what I've manually added to the distro config file for the SDK.

TOOLCHAIN_HOST_TASK_append = " \
    nativesdk-ament-package \
    nativesdk-python3-colcon-common-extensions \
    nativesdk-python3-numpy \
    nativesdk-rosidl-adapter \
    nativesdk-rosidl-cli \
    nativesdk-rosidl-cmake \
    nativesdk-rosidl-generator-c \
    nativesdk-rosidl-generator-cpp \
    nativesdk-rosidl-parser \
    nativesdk-rosidl-typesupport-c \
    nativesdk-rosidl-typesupport-cpp \
    nativesdk-rosidl-typesupport-fastrtps-c \
    nativesdk-rosidl-typesupport-fastrtps-cpp \
    nativesdk-rosidl-typesupport-introspection-c \
    nativesdk-rosidl-typesupport-introspection-cpp \
"

TOOLCHAIN_TARGET_TASK_append = " \
    ament-cmake \
    ament-cmake-auto \
    ament-cmake-core \
    ament-cmake-export-definitions \
    ament-cmake-export-dependencies \
    ament-cmake-export-include-directories \
    ament-cmake-export-interfaces \
    ament-cmake-export-libraries \
    ament-cmake-export-link-flags \
    ament-cmake-export-targets \
    ament-cmake-gmock \
    ament-cmake-google-benchmark \
    ament-cmake-gtest \
    ament-cmake-include-directories \
    ament-cmake-libraries \
    ament-cmake-nose \
    ament-cmake-pytest \
    ament-cmake-python \
    ament-cmake-ros \
    ament-cmake-target-dependencies \
    ament-cmake-test \
    ament-cmake-version \
    ament-lint-auto \
    foonathan-memory-staticdev \
"

I'm not sure ament-* recipes should be installed on the target here but find_package() will search on the target sysroot.
Then I've tried to setup the environment as follow.

cd ~/dev/test_ws
source /opt/r2os/0.9.0/environment-setup-corei7-64-r2os-linux
export AMENT_PREFIX_PATH=/opt/r2os/0.9.0/sysroots/corei7-64-r2os-linux/usr/

Using colcon build fails because some cmake modules will try to use target python3 executable instead of the host one. This can be worked around by adding --cmake-args -DPYTHON_EXECUTABLE=/opt/r2os/0.9.0/sysroots/x86_64-r2ossdk-linux/usr/bin/python3 to the previous command.

Another issue is that some libraries like libtinyxml2.so and libyaml.so are searched in yocto work directory instead of the target sysroot directory. I have no clue for this one yet.

stderr: example_interfaces
gmake[2]: *** No rule to make target « ~/dev/r2os/build-ld-core2/tmp/work/corei7-64-r2os-linux/fastrtps/2.3.4-1-r0/recipe-sysroot/usr/lib/libtinyxml2.so », necessary for « libexample_interfaces__rosidl_typesupport_fastrtps_cpp.so ». Stop.
gmake[2]: *** Waiting for unfinished tasks....
gmake[1]: *** [CMakeFiles/Makefile2:412 : CMakeFiles/example_interfaces__rosidl_typesupport_fastrtps_cpp.dir/all] Erreur 2
gmake: *** [Makefile:141 : all] Error 2

Failed <<< example_interfaces [29.5s, exited with code 2]
stderr: topic_statistics_demo
gmake[2]: *** No rule to make target « ~/dev/r2os/build-ld-core2/tmp/work/corei7-64-r2os-linux/rcl-yaml-param-parser/3.1.2-1-r0/recipe-sysroot/usr/lib/libyaml.so », necessary for « display_topic_statistics ». Stop.
gmake[2]: *** Waiting for unfinished tasks....
gmake[1]: *** [CMakeFiles/Makefile2:132 : CMakeFiles/display_topic_statistics.dir/all] Erreur 2
gmake: *** [Makefile:141 : all] Erreur 2

Failed <<< topic_statistics_demo [30.4s, exited with code 2]

windelbouwman · 2021-12-30T10:21:53Z

Nice job! That seems about right.

It seems fastrtps needs patching as well, I was wondering if this patch is required, it seems it does, it should be something like this:

else()
- find_library(TINYXML2_LIBRARY tinyxml2)
+ find_library(TINYXML2_LIBRARIES tinyxml2)
+ add_library(tinyxml2 SHARED IMPORTED)
+ set_property(TARGET tinyxml2 APPEND PROPERTY IMPORTED_LOCATION "${TINYXML2_LIBRARIES}")
+ set(TINYXML2_LIBRARY tinyxml2)
endif()

The same holds for the rcl-yaml-param-parser. I'm not sure about whether this is the right way to do it in cmake, but it might work out.

- find_package(PkgConfig REQUIRED)
- pkg_check_modules(yaml REQUIRED yaml-0.1)
+ find_library(LIBYAML_LIBRARIES NAMES yaml libyaml)
+ add_library(yaml SHARED IMPORTED)
+ set_property(TARGET yaml APPEND PROPERTY IMPORTED_LOCATION "${LIBYAML_LIBRARIES}")

Another thing, the TOOLCHAIN_HOST_TASK_append could be written in the new syntax TOOLCHAIN_HOST_TASK:append. What I'm also wondering about, is why all these packages need to be specified manually. Why are some packages not included as dependent packages?

RiZom-91 · 2021-12-30T22:38:47Z

I've patched both recipes as you suggested and it worked. Thank you! If this is an acceptable change, I could also commit for other ros2 distros.

I've added -DBUILD_TESTING=OFF to the cmake arguments because I was missing ros-testing and launch-testing-ament-cmake and bitbake was complaining these recipes were blacklisted.

I'm not sure about dependencies, I've started to add them one by one at the beginning but the compilation time is very long, so at some point I gave up and all 'related' recipes...

Another issue is that I don't know how to tell ament-cmake to search for packages in the workspace, because some of them are being built from there and are not on the target sysroot.

windelbouwman · 2022-01-01T09:39:07Z

The colcon magical incantation I used in the end was something like:

colcon build \
--merge-install \
--install-base ${MY_BUILD_FOLDER}/install \
--cmake-args \
-DCMAKE_TOOLCHAIN_FILE=${OE_CMAKE_TOOLCHAIN_FILE} \
-DPYTHON_EXECUTABLE=${OECORE_NATIVE_SYSROOT}/usr/bin/python3 \
-DCMAKE_STAGING_PREFIX=${MY_BUILD_FOLDER}/install \
-DCMAKE_PREFIX_PATH=${MY_BUILD_FOLDER}/install/share \
-DBUILD_TESTING=OFF

Couple of notes:

CMAKE_TOOLCHAIN_FILE specifies what toolchain to use, this is common practice for cmake cross compilation.
CMAKE_STAGING_PREFIX is useful during cross compilation, and allows cmake to find libraries in other folder than your sysroot. This is needed when building multiple inter depending packages.
CMAKE_PREFIX_PATH indicates where to find packages, again, handy when building multiple packages. Notice the extra share suffix, without this, it did not work out for me. This took me a while.
PYTHON_EXECUTABLE is manually specified, this should probably not be needed?

superseppl · 2023-05-01T16:46:15Z

What is blocking this PR from merging it into the master?

daizitian · 2023-06-14T07:01:58Z

The colcon magical incantation I used in the end was something like:
colcon build \
--merge-install \
--install-base ${MY_BUILD_FOLDER}/install \
--cmake-args \
-DCMAKE_TOOLCHAIN_FILE=${OE_CMAKE_TOOLCHAIN_FILE} \
-DPYTHON_EXECUTABLE=${OECORE_NATIVE_SYSROOT}/usr/bin/python3 \
-DCMAKE_STAGING_PREFIX=${MY_BUILD_FOLDER}/install \
-DCMAKE_PREFIX_PATH=${MY_BUILD_FOLDER}/install/share \
-DBUILD_TESTING=OFF
Couple of notes:

CMAKE_TOOLCHAIN_FILE specifies what toolchain to use, this is common practice for cmake cross compilation.

CMAKE_STAGING_PREFIX is useful during cross compilation, and allows cmake to find libraries in other folder than your sysroot. This is needed when building multiple inter depending packages.

CMAKE_PREFIX_PATH indicates where to find packages, again, handy when building multiple packages. Notice the extra share suffix, without this, it did not work out for me. This took me a while.

PYTHON_EXECUTABLE is manually specified, this should probably not be needed?

I have successfully compiled colcon into SDK. When compiling according to your colcon build command, I encountered the following problem. Do you have a solution?
`--- stderr: examples_rclcpp_minimal_action_client
CMake Error at /home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/rmw_implementation/cmake/rmw_implementation-extras.cmake:54 (message):
The RMW implementation has been specified as 'rmw_fastrtps_cpp' via the
default when rmw_implementation was built, but it is not available at this
time.

Currently available middlewares:

''
Call Stack (most recent call first):
/home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/rmw_implementation/cmake/rmw_implementationConfig.cmake:41 (include)
/home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/rcl/cmake/ament_cmake_export_dependencies-extras.cmake:21 (find_package)
/home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/rcl/cmake/rclConfig.cmake:41 (include)
/home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/libstatistics_collector/cmake/ament_cmake_export_dependencies-extras.cmake:21 (find_p ackage)
/home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/libstatistics_collector/cmake/libstatistics_collectorConfig.cmake:41 (include)
/home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/rclcpp/cmake/ament_cmake_export_dependencies-extras.cmake:21 (find_package)
/home/yk/out/ros2/sysroots/aarch64-poky-linux/usr/share/rclcpp/cmake/rclcppConfig.cmake:41 (include)
CMakeLists.txt:15 (find_package)

Failed <<< examples_rclcpp_minimal_action_client [1.45s, exited with code 1]`

jsb9945 · 2023-12-02T14:44:54Z

Hello, i build my image by adding the content on the right(#988 (comment))
But using 'colcon build', the error below continues to occur What's the reason?

(wrlinuxsdk-x86_64-aarch64) wkdtjdqls@wkdtjdqls-MS-7C94:~/ros2_ws$ sudo colcon build \

--merge-install
--install-base ${MY_BUILD_FOLDER}/install
--cmake-args
-DCMAKE_TOOLCHAIN_FILE=${OE_CMAKE_TOOLCHAIN_FILE}
-DPYTHON_EXECUTABLE=${OECORE_NATIVE_SYSROOT}/usr/bin/python3
-DCMAKE_STAGING_PREFIX=${MY_BUILD_FOLDER}/install
-DCMAKE_PREFIX_PATH=${MY_BUILD_FOLDER}/install/share
-DBUILD_TESTING=OFF
Starting >>> cpp_pubsub
--- stderr: cpp_pubsub
Traceback (most recent call last):
File "/opt/windriver/wrlinux/21.20/sysroots/cortexa53-wrs-linux/usr/share/ament_cmake_core/cmake/package_templates/templates_2_cmake.py", line 21, in
from ament_package.templates import get_environment_hook_template_path
ModuleNotFoundError: No module named 'ament_package'
CMake Error at /opt/windriver/wrlinux/21.20/sysroots/cortexa53-wrs-linux/usr/share/ament_cmake_core/cmake/ament_cmake_package_templates-extras.cmake:41 (message):

execute_process(/opt/windriver/wrlinux/21.20/sysroots/x86_64-wrlinuxsdk-linux/usr/bin/python3
/opt/windriver/wrlinux/21.20/sysroots/cortexa53-wrs-linux/usr/share/ament_cmake_core/cmake/package_templates/templates_2_cmake.py
/home/wkdtjdqls/ros2_ws/build/cpp_pubsub/ament_cmake_package_templates/templates.cmake)
returned error code 1
Call Stack (most recent call first):
/opt/windriver/wrlinux/21.20/sysroots/cortexa53-wrs-linux/usr/share/ament_cmake_core/cmake/ament_cmake_coreConfig.cmake:41 (include)
/opt/windriver/wrlinux/21.20/sysroots/cortexa53-wrs-linux/usr/share/ament_cmake/cmake/ament_cmake_export_dependencies-extras.cmake:15 (find_package)
/opt/windriver/wrlinux/21.20/sysroots/cortexa53-wrs-linux/usr/share/ament_cmake/cmake/ament_cmakeConfig.cmake:41 (include)
CMakeLists.txt:13 (find_package)

Failed <<< cpp_pubsub [0.08s, exited with code 1]

Summary: 0 packages finished [0.15s]
1 package failed: cpp_pubsub
1 package had stderr output: cpp_pubsub

Thank you.

SergeyFabric · 2023-12-15T12:21:16Z

The colcon magical incantation I used in the end was something like:
colcon build \
--merge-install \
--install-base ${MY_BUILD_FOLDER}/install \
--cmake-args \
-DCMAKE_TOOLCHAIN_FILE=${OE_CMAKE_TOOLCHAIN_FILE} \
-DPYTHON_EXECUTABLE=${OECORE_NATIVE_SYSROOT}/usr/bin/python3 \
-DCMAKE_STAGING_PREFIX=${MY_BUILD_FOLDER}/install \
-DCMAKE_PREFIX_PATH=${MY_BUILD_FOLDER}/install/share \
-DBUILD_TESTING=OFF
Couple of notes:

CMAKE_TOOLCHAIN_FILE specifies what toolchain to use, this is common practice for cmake cross compilation.

CMAKE_STAGING_PREFIX is useful during cross compilation, and allows cmake to find libraries in other folder than your sysroot. This is needed when building multiple inter depending packages.

CMAKE_PREFIX_PATH indicates where to find packages, again, handy when building multiple packages. Notice the extra share suffix, without this, it did not work out for me. This took me a while.

PYTHON_EXECUTABLE is manually specified, this should probably not be needed?

Hi Tnx, for your help here it's helped me a lot! I have to ask something about the compilation with Colcon that I succeeded in compiling in my SDK. But I have some further questions, What are the whole steps to compile a node with SDK and Colcon ? and how to move the crosscompiled code to the device and run it? there is not enough info about these steps.

Make some python packages extend 'nativesdk' to fix adding colcon to SDK

319e2aa

windelbouwman approved these changes Dec 29, 2021

View reviewed changes

Add 'nativesdk' to asio, libtinyxml2 and python3-lark-parser

abadd24

RiZom-91 added 2 commits December 30, 2021 16:54

Patched fastrtps to find thirdparty library TinyXML2

f0b47b0

Patched rcl-yaml-param-parser to find thirdparty library yaml

8c81b50

windelbouwman mentioned this pull request May 10, 2022

Colcon build for an STM32MP1 target #1000

Closed

robwoolley self-assigned this Apr 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dunfell][gatesgarth][hardknott][honister][kirkstone] Add 'nativesdk' to some python package recipes to fix colcon nativesdk recipe #988

[dunfell][gatesgarth][hardknott][honister][kirkstone] Add 'nativesdk' to some python package recipes to fix colcon nativesdk recipe #988

RiZom-91 commented Dec 29, 2021

windelbouwman left a comment

RiZom-91 commented Dec 30, 2021 •

edited

windelbouwman commented Dec 30, 2021 •

edited

RiZom-91 commented Dec 30, 2021

windelbouwman commented Jan 1, 2022

superseppl commented May 1, 2023

daizitian commented Jun 14, 2023

jsb9945 commented Dec 2, 2023

SergeyFabric commented Dec 15, 2023

[dunfell][gatesgarth][hardknott][honister][kirkstone] Add 'nativesdk' to some python package recipes to fix colcon nativesdk recipe #988

Are you sure you want to change the base?

[dunfell][gatesgarth][hardknott][honister][kirkstone] Add 'nativesdk' to some python package recipes to fix colcon nativesdk recipe #988

Conversation

RiZom-91 commented Dec 29, 2021

windelbouwman left a comment

Choose a reason for hiding this comment

RiZom-91 commented Dec 30, 2021 • edited

windelbouwman commented Dec 30, 2021 • edited

RiZom-91 commented Dec 30, 2021

windelbouwman commented Jan 1, 2022

superseppl commented May 1, 2023

daizitian commented Jun 14, 2023

jsb9945 commented Dec 2, 2023

Summary: 0 packages finished [0.15s] 1 package failed: cpp_pubsub 1 package had stderr output: cpp_pubsub

SergeyFabric commented Dec 15, 2023

RiZom-91 commented Dec 30, 2021 •

edited

windelbouwman commented Dec 30, 2021 •

edited

Summary: 0 packages finished [0.15s]
1 package failed: cpp_pubsub
1 package had stderr output: cpp_pubsub