Use cross-compilers to compile workspace instead of an emulated compilation

[emersonknapp]

Description

Currently, the tool performs the source build inside a QEMU-emulated container for the target architecture.

The desired control flow is:

• create the target architecture sysroot by running rosdep in an emulated image
• use that generated sysroot in a native environment with a cross-compiler

This will give better performance by compiling natively, and should allow for more flexible usage, to let users supply their own sysroots or compiler toolchains.

Test Plan

• No failures in existing testing
• Unit tests for new/untested parts of code
• Changes to the e2e test as required by new workflow

Documentation Plan

• Update README setup and installation instructions
• Specify the supported target architectures and platforms
• Have instructions to install the correct cross compilers in the Setup steps in README

Release Plan

• Changes should be merged in master

Acceptance Criteria

• Code has been implemented, reviewed and merged.
• Test plan has been completed
• Release plan has been completed

Once all above items are checked, this story can be moved to done.

Resources

None

[zmichaels11]

Notes:
Require user to apt-install cross compiler

[emersonknapp]

@zmichaels11 the plan is that we will run the compilation in a (native) docker image that has the necessary tooling, so i would cancel your note, and say the user still only needs Docker and Python3 to use this tool

[lukicdarkoo]

Based on your idea we created this:
https://github.com/cyberbotics/epuck_ros2/tree/master/installation/cross_compile

It works only for Raspberry Pi Zero, but it could be also extended to the other platforms (by including additional cross-compilers and *.cmake configurations).

[emersonknapp]

That's great! There is an experimental branch on this repository that is working towards the same goal https://github.com/ros-tooling/cross_compile/tree/emersonknapp/wip-be - but there are still a few issues with finding some includes/libraries in the target sysroot. I'm in the process of figuring out whether these issues are in the toolchain config or in the cmake configuration of the packages themselves. It's slow going but coming along.

[lukicdarkoo]

Oh, nice! I couldn't find a relevant PR, so I thought no progress has been made.

[emersonknapp]

It's very much not in a state for actual review, but it seems like good communication to put it as a draft PR - I've created that here #242

By all means try it out, but no guarantees about working :) and any fixes welcome. I also may force-push to the branch as I keep it up to date with master, but any PRs against the branch should be fine. Changed the branch name to be more informative - https://github.com/ros-tooling/cross_compile/tree/emersonknapp/cross-compile

[asherikov]

I did that some time ago with melodic, here are some notes that might be useful:

1. it is possible to use an actual system image of the target, no need to create a sysroot
2. cmake files generated by catkin include absolute paths (libraries, includes, gtest), which is a problem if you try to crosscompile your packages using base ROS packages installed in the target image. If I am not mistaken I've seen a stale bug report for catkin regarding this issue.
3. I had a weird issue with inclusions, when some of them were removed by cmake as duplicates, which was messing up inclusion order and breaking compilation.
4. 'True' crosscompilation is ~8 times faster than emulated compilation, I believe it is a must.

[emersonknapp]

1. it is possible to use an actual system image of the target, no need to create a sysroot

Can you clarify the distinction you are making here? Something like "a whole ubuntu" rather than a minimal directory of headers and libraries? If that is the case, that's how it it being done, the target system is emulated in Docker, has its rosdeps installed, and then is exported to a directory, it's got the entire (docker-minimal) OS in the sysroot directory.

2. cmake files generated by catkin include absolute paths (libraries, includes, gtest), which is a problem if you try to crosscompile your packages using base ROS packages installed in the target image. If I am not mistaken I've seen a stale bug report for catkin regarding this issue.

Yes, for ROS 1 this problem is difficult - it's less problematic in ROS 2 because ament/colcon creates portable builds. Some individual libraries still don't work correctly, when installed via package manager, due to incorrect CMake configurations most likely - I solved this in practice by copying some files around from the target sysroot into the host root so they could be found. This may help with ROS1 as well - I think I did something similar for a Kinetic cross-compile a few years back.

3. I had a weird issue with inclusions, when some of them were removed by cmake as duplicates, which was messing up inclusion order and breaking compilation.

I guess we'll have to cross this bridge when we get to it and have a concrete example to debug.

4. 'True' crosscompilation is ~8 times faster than emulated compilation, I believe it is a must.

Yep, it's an order of magnitude faster than emulation.

[asherikov]

1. it is possible to use an actual system image of the target, no need to create a sysroot Can you clarify the distinction you are making here? Something like "a whole ubuntu" rather than a minimal directory of headers and libraries? If that is the case, that's how it it being done, the target system is emulated in Docker, has its rosdeps installed, and then is exported to a directory, it's got the entire (docker-minimal) OS in the sysroot directory.

I ripped a disk image from the target machine and used it as a virtual drive. Additional packages can be installed to such image in emulation as well. This approach creates some interesting possibilities, but there are also a few technical issues.

2. cmake files generated by catkin include absolute paths (libraries, includes, gtest), which is a problem if you try to crosscompile your packages using base ROS packages installed in the target image. If I am not mistaken I've seen a stale bug report for catkin regarding this issue. Yes, for ROS 1 this problem is difficult - it's less problematic in ROS 2 because ament/colcon creates portable builds. Some individual libraries still don't work correctly, when installed via package manager, due to incorrect CMake configurations most likely - I solved this in practice by copying some files around from the target sysroot into the host root so they could be found. This may help with ROS1 as well - I think I did something similar for a Kinetic cross-compile a few years back.

I was able to resolve my issues by injecting `${CMAKE_SYSROOT}` in a few places, but this is a hack as well.

[coalman321]

I am a mentor for a high school FIRST Robotics team and we are attempting to cross compile ROS2, specifically Dashing Diademata for our soft float arm target based on a Zylinx ZYNQ processor.

We would be more than happy to help support the development of this in any way possible As we would like to begin implementing it on their test platform. Unfortunately we are not quite sure where to start. Any pointers? We are working on developing a cross root first but not quite sure how it would work with this tool after that point.

[emersonknapp]

I am a mentor for a high school FIRST Robotics team and we are attempting to cross compile ROS2, specifically Dashing Diademata for our soft float arm target based on a Zylinx ZYNQ processor.

We would be more than happy to help support the development of this in any way possible As we would like to begin implementing it on their test platform. Unfortunately we are not quite sure where to start. Any pointers? We are working on developing a cross root first but not quite sure how it would work with this tool after that point.

I think that the place you would start is with #242 - then look at adding new options to the pipeline in ros_cross_compile.py to optionally take in a premade user sysroot instead of creating one via dockerfiles. If you can get your workflow going with a fork off that branch, then that would be a great position to be in.

At a certain point, with custom sysroot and custom toolchain, I'm not sure how much utility this tool actually has - the idea of it is to abstract away some of those details for the most common cases. Maybe the CMake toolchain files just act as a good example for custom cases?

I am open to suggestions, though, for how this can make workflows easier for any case. The best conversation starter is probably a user-centric focus e.g. "What inputs to I provide and what do I expect to happen from there?" - then we could get into how to implement it

[coalman321]

That is a good point. I think the issue here is how to support rather odd targets, that may not have an emulatable system either from having a very small user base, or in our case, due to a closed source build process for the operating system image. Although, I will say that most of our cross root is either custom built dependencies for ROS or published archives of libraries built and publicly hosted for our target, so it would not be impossible to create it automatically.

In reality, our application would probably be better suited for meta-ros as it is based on open-embedded and yocto. Unfortunately the manufacturer of the controller uses significantly out of date versions that are not supported by meta-ros. This left us the only option to cross compile it manually.

[emersonknapp]

If you think that having a tool do it is easier than just doing the process by hand (and maybe it is, one person has to figure it out and then can distribute that solution easily - as in an education environment like yours) then here's my thinking:

The core pipeline I think is fairly universal (correct me if you see it otherwise)

1. Get or create the base sysroot
2. Determine application dependencies
3. Find or build dependencies into the sysroot
4. Install and configure a cross-compile toolchain
5. Build the user application source, against the sysroot, using the toolchain

Some or all of these steps could be implemented as plugins that can be registered with the tool to override behavior for specific target systems (or classes of systems). This would avoid having to hardcode all this logic into the tool itself - operating similar to colcon (for example) which can be extended arbitrarily from outside the core codebase. Note: ros_cross_compile doesn't do plugins yet, but it could, there are good examples of that pattern to copy from.

For example, a "create the base sysroot" stage could in theory be implemented by a YoctoSysrootCreator plugin that would create what you need. In the specific case of yocto, i'm not familiar enough to say whether you should always build entirely within yocto, or if it sometimes makes sense to do a mixed-case workflow, building the sysroot as a monolith, then doing specific application builds after-the-fact using an independent cross-compile workflow.

Maybe as part of identifying how your application gets built for your board, you could think about which of those steps would require a swapped out implementation.

[coalman321]

I definitely agree with the core pipeline and the plugin assessment. That would definitely allow a more configurable system overall. The most "changable" parts of the system would probably be the installing the toolchain and finding / building the sysroot. Other than those steps most of the effort would be pretty much the same. It should also probably still be done in a container to maintain the overall system isolation during complication.

As to the Yocto point though, my understanding is that it builds based on file system layers, where meta-ros acts as a layer in the system image, so I am not really sure if it would be comparable here. From what I have discovered so far, it requires being able to build the entire image which is unfortunately not possible as the manufacturer has removed necessary parts to build it for their ARM platform.

[Spazzinq]

Hello, I am a member of the aforementioned FIRST Robotics team. What kind of implementation were you looking for in terms of plugin structure to choose a sysroot/compiler?

[emersonknapp]

I think the first question is "what kind of end user experience are you hoping to get?" Jumping into implementation before deciding on a workflow won't get us very far. If you could describe in detail what you would be able to provide to the tool, vs what you need it to do for you, then we can decide what would need to be a plugin vs not.

I don't think there is anything that this tool will be able to provide for you that you wouldn't have to first figure out how to do manually - so a step-by-step instruction set for your platform would then highlight the portions that could be automated - at the end of the day this ros_cross_compile tool isn't doing anything all that fancy, it's just automating away a manual process that can be error prone when you have to do it repeatedly or on new machines.

[coalman321]

Yes, you are absolutely correct, we have continued to develop our workflow (time permitting) and are getting closer. The question about what kind of plugin structure was more academic than implementation. The intent was to look into the system and understand it a bit before even attempting to use it in the future.

That aside, would you happen to know why Colcon would be looking for the shim libraries if they were already compiled into the sysroot? looking at what we have, the sysroot seems to be pointing to the correct location, that contains our cross compiled libraries. This is the dockerfile we have been using to develop the process thus far https://github.com/Worthington-Robotics/2021ROS2Build/blob/master/dashing_compile.dockerfile

[emersonknapp]

That aside, would you happen to know why Colcon would be looking for the shim libraries if they were already compiled into the sysroot? looking at what we have, the sysroot seems to be pointing to the correct location, that contains our cross compiled libraries. This is the dockerfile we have been using to develop the process thus far https://github.com/Worthington-Robotics/2021ROS2Build/blob/master/dashing_compile.dockerfile

I'd prefer not to add too much tangential discussion to this ticket - if you have more details to share (I can't tell much from that description) - perhaps you could @ me on a ticket over on your repository

[windelbouwman]

Just wanted to drop this project here into the discussion, as I think it might be relevant: https://github.com/labapart/cross_sysroot

Rough idea of this project:

• download debian packages from ubuntu ports for arm64 / armhf, and extract them into a folder
• Fix some symlinks

This is essentially what debootstrap is doing. Benefit of creating a sysroot using something like this, is that you do not need a qemu / docker setup to create the sysroot. It's essentially extracting the existing arm64 debian archives in a local folder to create a sysroot.

Obvious downside is the slight version mismatch between the target and the sysroot, but doing an apt update on the target should fix that.

[asherikov]

If someone is interested, here is my take on the problem -> https://github.com/asherikov/ccws, where cross compilation is handled as one of possible build profiles (https://github.com/asherikov/ccws#cross-compilation) of a project and is split into several steps roughly corresponding to those suggested by @emersonknapp. I have also found a workaround for the issue with absolute paths in ROS1 cmake files that does not require any patching -- I use https://github.com/proot-me/proot to address this.