Cross compiling instructions

[diwic]

I've started to write up some cross compiling instructions here: https://github.com/diwic/dbus-rs/blob/master/libdbus-sys/cross_compile.md

...but I can't get it to work. Anyone who wants to help out?

[nikonthethird]

Yes I can help out, I am cross-compiling dbus-rs for the ARMv7 architecture. The only thing is, it is quite an ordeal to set up initially. Locating, downloading, extracting, modifying and merging the native libraries sucks.

I have written a guide that I will post in the next comment (hope it's not too long), then you can see how involved the process is.

[nikonthethird]

Cross compiling the DBus crate

This guide assumes you are using Ubuntu for the host system and are cross compiling for a Raspberry Pi. All shell commands of this guide are available in both PowerShell and Bash.

The following tasks have to be completed to set up cross compilation:

• Setting up the Rust compiler for cross compilation.
• Configuring the DBus crate for cross compilation.

Setting up the Rust compiler

If Rust is not installed on your system, head over there to get it up and running. We are trying to cross compile for the ARMv7 architecture, so we need the GCC cross compiler toolchain for it. The following command installs commands prefixed with arm-linux-gnueabihf-, which will be used by the Rust compiler to create the ARMv7 executable.

PowerShell (pwsh) & Bash

sudo apt install gcc-arm-linux-gnueabihf

Naturally, we need to inform the Rust compiler that this toolchain is available. Edit or create the file ~/.cargo/config and add the following section for the ARMv7 target.

~/.cargo/config

[target.armv7-unknown-linux-gnueabihf]
linker = "arm-linux-gnueabihf-gcc"

Next we need to get Rust set up for the ARMv7 target. On a fresh install, only the standard library for your host system is installed (usually x86_64-unknown-linux-gnu, you can see the supported and installed targets by running rustup target list), so to add the ARMv7 target, run the following
command.

PowerShell (pwsh) & Bash

rustup target add armv7-unknown-linux-gnueabihf

Testing the Rust compiler configuration

To test what we just configured, create an empty directory somewhere and start a terminal in it. Then execute the following commands.

PowerShell (pwsh)

# Initialize a new Rust project in the current folder.
cargo init

# Compile the project for ARMv7.
cargo build --target=armv7-unknown-linux-gnueabihf

# Check the generated file. This should say ARM somewhere.
file "./target/armv7-unknown-linux-gnueabihf/debug/$((Get-Item $PWD).Name)"

# If the file could not be generated, check the configuration again.
# To test if the Rust compiler is working without Cargo, you can run
# the rustc compiler directly and test if the file generated this way is correct.
rustc --target=armv7-unknown-linux-gnueabihf -C linker=$((Get-Command arm-linux-gnueabihf-gcc).Source) ./src/main.rs
file ./main

Bash

# Initialize a new Rust project in the current folder.
cargo init

# Compile the project for ARMv7.
cargo build --target=armv7-unknown-linux-gnueabihf

# Check the generated file. This should say ARM somewhere.
file ./target/armv7-unknown-linux-gnueabihf/debug/${PWD##*/}

# If the file could not be generated, check the configuration again.
# To test if the Rust compiler is working without Cargo, you can run
# the rustc compiler directly and test if the file generated this way is correct.
rustc --target=armv7-unknown-linux-gnueabihf -C linker=$(which arm-linux-gnueabihf-gcc) ./src/main.rs
file ./main

Now you have a working Rust cross-compilation toolchain set up. Next, we need to configure the DBus crate for cross compilation.

Configuring the DBus crate for cross compilation

The DBus crate uses a build script that uses pkg_config to locate the native dbus libraries. This works well when compiling for the host system, but not when cross compiling. In this case, we have to do what the Cargo book tells us and generate the output of the build script ourselves.

The crate's build script is specified in Cargo.toml and is normally executed at every build. There are two Cargo keys that have to be returned by us:

• cargo:rustc-link-search, which is the library search path.
• cargo:rust-link-lib, which is the name of a library to link.

There are two ways we can provide these keys to Cargo:

• In a Cargo config file.
• In our own build script.

Usually a Cargo config file should be enough, just specify the names of the libraries and the paths to them and we should be good to go, right? Not so fast. Currently, there is an issue with how Cargo handles relative paths in rustc-link-search: They are resolved relative to the location of the extracted crate, not relative to our project! So we would have to specify the library search paths absolute for the config file to work. Since it is a bit ridiculous to require the repository to be at the same location for each user, we can additionally use another build script to provide the library search paths.

Enough talk, to action! To see how to set up the DBus crate for cross compilation, we create a new project for it. Create an empty directory somewhere and start a terminal in it. Then execute the following commands.

PowerShell (pwsh) & Bash

# Initialize a new Rust project in the current folder.
cargo init

Change the contents of main.rs to the following. This is just so that we actually use something from the DBus crate, otherwise there would be no reference to it in the final executable and nothing for the linker to do.

main.rs

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let _conn = dbus::Connection::get_private(dbus::BusType::Session)?;
    Ok(())
}

Next, add the DBus crate as an dependency by editing Cargo.toml.

Cargo.toml

[dependencies]
dbus = "0.6"

Create a Cargo configuration file, which has to be in a .cargo folder at the root of the repository, that specifies that the default target is ARMv7 and lists all the native ARMv7 libraries that are required to build DBus.

.cargo/config

[build]

# Specifies that the default target is ARMv7.
target = "armv7-unknown-linux-gnueabihf"

[target.armv7-unknown-linux-gnueabihf.dbus]

# Specifies the library search paths. Since they cannot be relative paths,
# we use a build script to provide them.
rustc-link-search = [
    # Provided by the build script.
]

# Specifies the names of the native libraries that are required to build DBus.
rustc-link-lib = [
    "dbus-1",
    "gcrypt",
    "gpg-error",
    "lz4",
    "lzma",
    "pcre",
    "selinux",
    "systemd",
]

Create the build script that will provide the library search paths. The build script will be named build.rs (which is the default) and lives at the root of the project, next to Cargo.toml.

build.rs

use std::env::var;

fn main() {
    // The manifest dir points to the root of the project containing this file.
    let manifest_dir = var("CARGO_MANIFEST_DIR").unwrap();
    // We tell Cargo that our native ARMv7 libraries are inside a "libraries" folder.
    println!("cargo:rustc-link-search={}/libraries/lib/arm-linux-gnueabihf", manifest_dir);
    println!("cargo:rustc-link-search={}/libraries/usr/lib/arm-linux-gnueabihf", manifest_dir);
}

If you tried building the project at this point, you should get a rather long error message with this at the top: linking with arm-linux-gnueabihf-gcc failed. The final (and most tedious) step is finding and downloading the native ARMv7 libraries required by DBus.

How do you even find out which of the native packages are required? If you take a look at this line in the DBus build script, you see that it is looking for "dbus-1", which means libdbus-1.

OK, now which version of libdbus-1 is required? If you have your target system at hand, you can connect to it and run apt show libdbus-1* on it, which should show something like this.

libdbus-1 Information

Package: libdbus-1-3
Version: 1.12.16-1
...
Depends: libc6 (>= 2.28), libsystemd0
...

If you do not have the target system at hand, there is still a way: If you are using the Raspbian release based on Debian buster, head to this link (this is a huge file!) and search for Package: libdbus-1 inside there. You should see the same information.

Now we know that we have to download libdbus1-3 version 1.12.16-1 and it depends on libc6 (which is provided by the cross compilation toolchain) and libsystemd0 (which is not and which we also have to download).

In total, you have to download the following packages (the .deb files). This list contains the versions for the Raspbian release based on Debian buster. They may have changed since, check the versions installed on your target system. Click each of the package names below and download the correct file.

• libdbus-1-3 is at version 1.12.16-1
• libgcrypt20 is at version 1.8.4-5
• libgpg-error0 is at version 1.35-1
• liblz4-1 is at version 1.8.3-1
• liblzma5 is at version 5.2.4-1
• libpcre3 is at version 2:8.39-12
• libselinux1 is at version 2.8-1+b1
• libsystemd0 is at version 241-5+rpi1

Next, you have to extract each of these downloaded .deb files separately into an empty folder.

PowerShell (pwsh) & Bash

dpkg-deb -x /path/to/package.deb /path/to/empty/folder

Enter the folder you have extracted the package into and take a look at the files. The folder structure can be ./lib/arm-linux-gnueabihf or even ./usr/lib/arm-linux-gnueabihf inside this folder. The relevant files are the .so files. Some libraries however have another number after the .so, for example library.so.3. In this case, you have to add a symlink to library.so because that's where the GCC linker will look for it. The symlink must be in the same directory as the file it points to. To create a symlink called library.so that points to library.so.3, you would use the following command.

PowerShell (pwsh)

New-Item -ItemType SymbolicLink -Target library.so.3 -Path library.so

Bash

ln -s library.so.3 library.so

Then take all the contents of the folder you extracted the package into and move them into another folder called libraries, which you create at the root of your Rust project. This is the location we directed the GCC linker to look for the libraries.

Repeat the extraction, symlinking and moving for all the other libraries.

Finally, after all this is done, your libraries folder should look something like this (the version numbers may differ):

• ./lib/arm-linux-gnueabihf/libdbus-1.so
• ./lib/arm-linux-gnueabihf/libdbus-1.so.3
• ./lib/arm-linux-gnueabihf/libdbus-1.so.3.14.15
• ./lib/arm-linux-gnueabihf/libgcrypt.so
• ...
• ./usr/lib/arm-linux-gnueabihf/liblz4.so
• ./usr/lib/arm-linux-gnueabihf/liblz4.so.1
• ...

Finally, you are able to cross compile the project without error messages.

PowerShell (pwsh) & Bash

cargo build
# Should print something like: Finished dev [unoptimized + debuginfo] target(s) in 0.57s

[diwic]

@nikonthethird Awesome, very well written! Would you like to submit a PR that completely replaces my non-working guide with yours?

The final (and most tedious) step is finding and downloading the native ARMv7 libraries required by DBus.

Your way is doing all this manually; I think there should be some way that utilizes debootstrap or a wrapper around it (Debian/Ubuntu has more than one...) that downloads this and extracts it for you.

Some libraries however have another number after the .so, for example library.so.3. In this case, you have to add a symlink to library.so because that's where the GCC linker will look for it.

This symlink is provided by the -dev package, so an alternative option would be to download and extract the -dev package. Do you need it for more than libdbus-1-dev?

[nikonthethird]

So that's what the dev packages do?!? Wow thanks, I will incorporate that into the guide.

I also did not know about debootstrap, I just got cross compilation working my (admittedly stupid) way and took notes of what I did. I will also check that out.

[nikonthethird]

Also, do you think that libdbus-1-dev should be the only dependency to install, and rustc-link-lib should only contain dbus-1?

Because when I tried that, I get link failures about libsystemd0, that's why I had to download all of the non-libc dependencies. And if that's the case, then yes, I think you need the dev packages for all the other ones as well because I had to create the symlinks for all of them. Only when I did all that then linker errors went away...

[diwic]

@nikonthethird

I also did not know about debootstrap, I just got cross compilation working my (admittedly stupid) way and took notes of what I did. I will also check that out.

There are also wrappers around debootstrap (sbuild, pbuilder, cowbuilder), some of these might be easier to work with. I haven't investigated.

Also, do you think that libdbus-1-dev should be the only dependency to install, and rustc-link-lib should only contain dbus-1?

I'm not sure, that's kind of where I got stuck? But if I want to compile this crate normally (i e not cross), then libdbus-1-dev is all I need to install for this to work, and it does not seem to depend on other -dev packages. And pkg-config --libs dbus-1 only outputs -ldbus-1, not all the other libs (like libsystemd0 etc).

I mean, it's great that you found some way to make it work, but it does not make sense; why would you need more packages for cross compiling than what you need for non-cross?

[nikonthethird]

I tried using debootstrap to download the packages from the Raspbian mirror, but it seems to download way too much and requires additional configuration. I also tried to just limit the required dev package to libdbus-1-dev, but then I get errors:

libdbus-1.so: undefined reference to `sd_listen_fds@LIBSYSTEMD_209'
libdbus-1.so: undefined reference to `sd_is_socket@LIBSYSTEMD_209'

Only when adding systemd to rustc-link-lib, this error goes away. And that requires libsystemd-dev.

But I have good news: I have written a PowerShell script that completely configures a new Rust project with the DBus crate for cross compilation: DBusCrossCompile.zip

You have to install PowerShell Core on your Ubuntu host, and then run in any shell:

chmod +x ./DBusCrossCompile.ps1
./DBusCrossCompile.ps1

It creates a folder dbus-cross, which contains a Rust project, then downloads all the dependencies from the Raspbian mirror, fixes some absolute paths to /usr/lib and /lib and compiles the project. This works fine in my Ubuntu VM, the output looks something like this:

PS /home/nikon/Documents> ./DBusCrossCompile.ps1
Created new Rust project in folder dbus-cross
Added DBus crate as dependency to Cargo.toml
Created cargo configuration file.
Created build script.
Updated contents of main.rs.
Downloading Raspbian buster package file...
Downloaded Raspbian buster package file.
Downloaded and extracted dependency libgmp10.
Downloaded and extracted dependency libisl19.
Downloaded and extracted dependency libmpfr6.
Downloaded and extracted dependency libmpc3.
Downloaded and extracted dependency zlib1g.
Downloaded and extracted dependency cpp-8.
Downloaded and extracted dependency gcc-8-base.
Downloaded and extracted dependency libgcc1.
Downloaded and extracted dependency libc6.
Downloaded and extracted dependency libgpg-error0.
Downloaded and extracted dependency libgcrypt20.
Downloaded and extracted dependency liblz4-1.
Downloaded and extracted dependency liblzma5.
Downloaded and extracted dependency libsystemd0.
Downloaded and extracted dependency libdbus-1-3.
Downloaded and extracted dependency libffi6.
Downloaded and extracted dependency libuuid1.
Downloaded and extracted dependency libblkid1.
Downloaded and extracted dependency libpcre3.
Downloaded and extracted dependency libbz2-1.0.
Downloaded and extracted dependency dpkg.
Downloaded and extracted dependency perl.
Downloaded and extracted dependency libdpkg-perl.
Downloaded and extracted dependency pkg-config.
Downloaded and extracted dependency libdbus-1-dev.
Downloaded and extracted dependency libc-dev-bin.
Downloaded and extracted dependency libtinfo6.
Downloaded and extracted dependency bash.
Downloaded and extracted dependency patch.
Downloaded and extracted dependency dctrl-tools.
Downloaded and extracted dependency linux-libc-dev.
Downloaded and extracted dependency libc6-dev.
Downloaded and extracted dependency libgpg-error-dev.
Downloaded and extracted dependency libgcrypt20-dev.
Downloaded and extracted dependency liblz4-dev.
Downloaded and extracted dependency liblzma-dev.
Downloaded and extracted dependency libpcre16-3.
Downloaded and extracted dependency libpcre32-3.
Downloaded and extracted dependency libstdc++6.
Downloaded and extracted dependency libpcrecpp0v5.
Downloaded and extracted dependency libpcre3-dev.
Downloaded and extracted dependency libsepol1.
Downloaded and extracted dependency libsepol1-dev.
Downloaded and extracted dependency libselinux1-dev.
Downloaded and extracted dependency libsystemd-dev.
Downloaded and extracted all required dependencies.
Converted absolute paths in ld scripts to relative paths.
Converted absolute paths in symbolic links to relative paths.                                                                                                           
The project has been set up for cross compilation. Starting a build...
    Updating crates.io index
   Compiling libc v0.2.61
   Compiling libdbus-sys v0.2.0
   Compiling dbus-cross v0.1.0 (/home/nikon/Documents/dbus-cross)
   Compiling dbus v0.6.5
    Finished dev [unoptimized + debuginfo] target(s) in 7.71s

Sorry it had to be PowerShell, but I'm a Windows dev by trade and bash is still arcane magic to me.

Does this approach look better to you? There are no manual steps involved. If you have an idea how to debootstrap the libraries that doesn't require root (debootstrap does) or an even more complicated setup, that would be even better. :)

[nikonthethird]

Also, regarding the issue you had in your attempt at cross-compilation.

...but when I tried this, I got the following error:
cannot find /lib/arm-linux-gnueabihf/libpthread.so.0
cannot find /usr/lib/arm-linux-gnueabihf/libpthread_nonshared.a
...which I, so far, have not been able to resolve. Let me know if you have any ideas!

That's what the step Converted absolute paths in ld scripts to relative paths. of the script does. For example, open up the script under $yourArmLibFolder/usr/lib/arm-linux-gnueabihf/libc.so. It should look like this:

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-littlearm)
GROUP ( /lib/arm-linux-gnueabihf/libc.so.6 /usr/lib/arm-linux-gnueabihf/libc_nonshared.a  AS_NEEDED ( /lib/arm-linux-gnueabihf/ld-linux-armhf.so.3 ) )

The paths are absolute! You have to change them to this:

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-littlearm)
GROUP ( ../../../lib/arm-linux-gnueabihf/libc.so.6 ../../../usr/lib/arm-linux-gnueabihf/libc_nonshared.a  AS_NEEDED ( ../../../lib/arm-linux-gnueabihf/ld-linux-armhf.so.3 ) )

Furthermore, some symlinks also point to absolute locations that have to be fixed, that's also another step in the script. If you get errors that some files could not be found again, it's probably that.

[diwic]

@nikonthethird

The paths are absolute!

Good catch! So with this finding of yours, I adjusted my own instructions slightly and got my version to compile as well! This is because I added "/usr/arm-linux-gnueabihf/lib" to rustc-link-search and the libs in there - which came with the cross linker - contain the correct paths.

Now I should probably verify that the final binary works on the target, but that will be a task for another day...have you done that with your version?

[diwic]

If you have an idea how to debootstrap the libraries that doesn't require root (debootstrap does) or an even more complicated setup, that would be even better. :)

Well, I have used pbuilder in the past, but merely followed instructions without taking the time to fully understand what was going on under the hood. As for needing root, I guess fakeroot can work around that somehow.

I tried using debootstrap to download the packages from the Raspbian mirror, but it seems to download way too much and requires additional configuration.

Indeed its main purpose is to create a rootfs. My guide just assumes you have one ready, e g, you start up your raspberry, run apt install libdbus-1-dev, then turn it off, and move the sd card to the computer and mount it.

[nikonthethird]

Now I should probably verify that the final binary works on the target, but that will be a task for another day...have you done that with your version?

Absolutely, this is the output of the binary created by the script, running on a Raspberry Pi 3.

pi@smokey:~ $ ./dbus-cross 
org.freedesktop.DBus
org.freedesktop.login1
org.freedesktop.timesync1
org.freedesktop.systemd1
org.freedesktop.Avahi
org.bluez
:1.23
:1.24
:1.0
:1.1
:1.2
:1.3
:1.4
fi.epitest.hostap.WPASupplicant
fi.w1.wpa_supplicant1
:1.5
pi@smokey:~ $

The reason I am using this manual approach is that the project I am working on has to be compiled by several people. Only two of them have the target ARM hardware. That's why we added the libraries to the git repository, that way anyone who has the toolchain installed can build the project.

How do you want to proceed with the instructions? I think most people will be happy with plugging an SD card in to compile the project, so your original guide should be just fine?

[diwic]

How do you want to proceed with the instructions? I think most people will be happy with plugging an SD card in to compile the project, so your original guide should be just fine?

But I like yours as well 🙂

So I'm not sure. What do you think? Maybe have both guides and link to yours from mine?

[diwic]

Ok, so I linked to the issue in the existing instructions, but I would be happy to merge the powershell script as well as your guide if you prefer.

Thanks for the help anyhow!

[nikonthethird]

Maybe have both guides and link to yours from mine?

That sounds fine, thank you. I still have to update the guide a little, I'm on holiday until Sunday, but after that I will add the symlink and LD script changes.

[chirping78]

@nikonthethird
Just leave a comment for future reference: your step "Converted absolute paths in ..." can be covered by add a RUSTFLAGS like:
RUSTFLAGS="-C link-arg=--sysroot=$yourArmLibFolder
So there is no need to fix these absolute path in so file.