Darwin (host) to linux (target) cross compilation

[robbertvanginkel]

I'm looking to make a bazel toolchain for usage with rules_go, with the goal of supporting darwin to linux cross compilation with cgo.

This repo has been a great example and help for understanding bazel's toolchain configuration so far. Before I dive in, I was wondering you you'd attempted or considered extending this repo to include toolchains for host darwin with target linux?

[siddharthab]

I have certainly given it some thought. I think we can achieve this with the right sysroot and target triple. I am not sure if we can do this without a sysroot though.

Let me see how far I can get this weekend.

[siddharthab]

Sorry, I could not really finish scoping it during the weekend. I will try again throughout the week.

[siddharthab]

Quick update: I have been trying to work on it a little bit, but progress has been slow. There are still some subtleties that I need to deal with.

[NathanHowell]

This could help build Docker images using rules_docker on macOS

[prestonvanloon]

Also interested in cross compilation. Any update, progress, or areas that need help?

[prestonvanloon]

If it's helpful, here are some logs from my use case. I am trying to use rules_foreign_cc and compile for darwin on linux. This fails ToolchainResolution. I'm hoping that is some configuration issue. What do you think?

INFO: ToolchainResolution: Looking for toolchain of type @rules_foreign_cc//tools/build_defs:cmake_toolchain...
INFO: ToolchainResolution: Looking for toolchain of type @bazel_tools//tools/cpp:toolchain_type...
INFO: ToolchainResolution: Looking for toolchain of type @rules_foreign_cc//tools/build_defs:ninja_toolchain...
INFO: ToolchainResolution:   Considering toolchain @rules_foreign_cc//tools/build_defs/native_tools:preinstalled_ninja...
INFO: ToolchainResolution:   For toolchain type @rules_foreign_cc//tools/build_defs:ninja_toolchain, possible execution platforms and toolchains: {@local_config_platform//:host -> @rules_foreign_cc//tools/build_defs/native_tools:preinstalled_ninja}
INFO: ToolchainResolution: Looking for toolchain of type @rules_foreign_cc//tools/build_defs/shell_toolchain/toolchains:shell_commands...
INFO: ToolchainResolution:   Considering toolchain @llvm_toolchain//:cc-clang-linux...
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:linux, which does not match value @platforms//os:osx from the target platform @io_bazel_rules_go//go/toolchain:darwin_amd64
INFO: ToolchainResolution:   Considering toolchain @rules_foreign_cc//tools/build_defs/native_tools:preinstalled_cmake...
INFO: ToolchainResolution:   Rejected toolchain @llvm_toolchain//:cc-clang-linux, because of target platform mismatch
INFO: ToolchainResolution:   Considering toolchain @llvm_toolchain//:cc-clang-darwin...
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:osx, which does not match value @platforms//os:linux from the execution platform @local_config_platform//:host
INFO: ToolchainResolution:   Considering toolchain @rules_foreign_cc//tools/build_defs/shell_toolchain/toolchains:linux_commands_data...
INFO: ToolchainResolution:   Considering toolchain @local_config_cc//:cc-compiler-armeabi-v7a...
INFO: ToolchainResolution:   For toolchain type @rules_foreign_cc//tools/build_defs:cmake_toolchain, possible execution platforms and toolchains: {@local_config_platform//:host -> @rules_foreign_cc//tools/build_defs/native_tools:preinstalled_cmake}
INFO: ToolchainResolution:     Toolchain constraint @platforms//cpu:cpu has value @platforms//cpu:arm, which does not match value @platforms//cpu:x86_64 from the target platform @io_bazel_rules_go//go/toolchain:darwin_amd64
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:android, which does not match value @platforms//os:osx from the target platform @io_bazel_rules_go//go/toolchain:darwin_amd64
INFO: ToolchainResolution:   Considering toolchain @rules_foreign_cc//tools/build_defs/shell_toolchain/toolchains:windows_commands_data...
INFO: ToolchainResolution:   Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a, because of target platform mismatch
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:windows, which does not match value @platforms//os:linux from the execution platform @local_config_platform//:host
INFO: ToolchainResolution:   Considering toolchain @local_config_cc//:cc-compiler-k8...
INFO: ToolchainResolution:   Considering toolchain @rules_foreign_cc//tools/build_defs/shell_toolchain/toolchains:osx_commands_data...
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:linux, which does not match value @platforms//os:osx from the target platform @io_bazel_rules_go//go/toolchain:darwin_amd64
INFO: ToolchainResolution:     Toolchain constraint @platforms//os:os has value @platforms//os:osx, which does not match value @platforms//os:linux from the execution platform @local_config_platform//:host
INFO: ToolchainResolution:   Rejected toolchain @local_config_cc//:cc-compiler-k8, because of target platform mismatch
INFO: ToolchainResolution:   Considering toolchain @rules_foreign_cc//tools/build_defs/shell_toolchain/toolchains:default_commands_data...
INFO: ToolchainResolution:   No toolchains found
INFO: ToolchainResolution:   For toolchain type @rules_foreign_cc//tools/build_defs/shell_toolchain/toolchains:shell_commands, possible execution platforms and toolchains: {@local_config_platform//:host -> @rules_foreign_cc//tools/build_defs/shell_toolchain/toolchains:linux_commands_data}
ERROR: While resolving toolchains for target //third_party/kafka:librdkafka: no matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type
ERROR: Analysis of target '//third_party/kafka:librdkafka' failed; build aborted: no matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type
INFO: Elapsed time: 0.317s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 157 targets configured)

[ZymoticB]

I've taken a functional cut at this here ZymoticB@fb76b59

With a WORKSPACE like

load("@com_grail_bazel_toolchain//toolchain:rules.bzl", "llvm_toolchain")
load("@com_grail_bazel_toolchain//toolchain:repositories.bzl", "bazel_toolchains_repositories")
bazel_toolchains_repositories("9.0.0")

http_archive(
    name = "org_chromium_sysroot_linux_x64",
    build_file_content = """
filegroup(
  name = "sysroot",
  srcs = glob(["*/**"]),
  visibility = ["//visibility:public"],
)
""",
    sha256 = "84656a6df544ecef62169cfe3ab6e41bb4346a62d3ba2a045dc5a0a2ecea94a3",
    urls = ["https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/2202c161310ffde63729f29d27fe7bb24a0bc540/debian_stretch_amd64_sysroot.tar.xz"],
)

llvm_toolchain(
    name = "llvm_toolchain",
    llvm_version = "9.0.0",
    sysroot = {
        "linux": "@org_chromium_sysroot_linux_x64//:sysroot",
        "darwin": "@org_chromium_sysroot_linux_x64//:sysroot",
    },
)

load("@llvm_toolchain//:toolchains.bzl", "llvm_register_toolchains")

llvm_register_toolchains()

A little bit in the .bazelrc

build:linuxondarwin --host_crosstool_top=@bazel_tools//tools/cpp:toolchain
build:linuxondarwin --crosstool_top=@llvm_toolchain//:osxcross
build:linuxondarwin --cpu linux-cross

I can

bazel build --config linuxondarwin @com_github_mattn_go_sqlite3//:go_default_library

The one suboptimal hunk is that to cross compile C++ you need to -nostdinc++ and link against the cross compiled version of libc++ and libc++abi I've added. This will need annoying patches to get working with cgo I think. I think the correct approach here is something like https://github.com/vsco/bazel-toolchains/blob/master/toolchains/libcxx_library.bzl (to make use of the extracted toolchain directly instead of via crosstool) but I haven't spent the time yet.

[martinxsliu]

@siddharthab Any progress so far, or thoughts on @ZymoticB 's approach? This would also be really useful for us to build Docker images from macs.

[siddharthab]

@martinxsliu Thanks for bringing this up. This issue has largely fallen off my radar because of the complexities involved. I am going to make a push for it this week. 🤞

[rdelfin]

Has there been any further progress on the issue? I'm looking for a similar cross compilation toolchain (in my case, Linux to ARM) and wanted to see if I could use this. Thanks!

[siddharthab]

I have not been able to work on this issue in any significant capacity so far. I keep promising to look into it but have been failing terribly in keeping these promises. I still want to do it, just need to get better at time management.

[ajbouh]

I've been trying to get this working and it's been extremely challenging.

It seems like builtin_sysroot and cxx_builtin_include_directories just don't work for some reason. I can't tell why. Has anyone else made any progress on this?

[ajbouh]

Ok, I've reached the bottom of this hole and am digging myself back out.

It seems like builtin_sysroot and cxx_builtin_include_directories don't "just work" because of multiple overlapping concepts:

• they need to refer to absolute paths
• you still need to include them as flags
• for things to work smoothly, these additional files should be in the same external repository as the toolchain itself

Based on these learnings (and some help from the Bazel community), I got something working based on zig cc over here: https://github.com/ajbouh/bazel-zig-cc

[siddharthab]

After years of keeping this on the backburner, I was able to get to this today, and was able to refactor the codebase to support cross-compilation better. I have a final PR out that adds a test for darwin to linux cross-compile.

I took a simpler route and did not link against the bundled libc++, but chose to link against the available libstdc++ in the user provided sysroot.

I have only tested basic things, so if you find some flags need adjusting, please feel free to send another PR.