Cross-compile for Android from OSX

[ephemer]

I have just pushed a branch called osx-crosscompile that is at the stage of correctly compiling the objects (as verified using nm from the different toolchains). It fails at the linker stage. I don't understand my way around the OSX toolchain just yet, or how to change it to use the Android toolchain.

Basically it's failing because the OSX linker doesn't understand --sysroot, up until that point, we need the --sysroot linker option because otherwise the Android components attempt to link against the OSX /usr/include, which throws all sorts of errors because of wrong architecture etc.

Another thought was that maybe we can use the iOS linker, since it is also aware of armeabi-v7a. Not sure how to do this, or whether it's a good idea.

I'm going to try figure out how to use the linker from a different toolchain tomorrow but given my (non-existent) experience in this field I'm not sure how far I'll get. Any hints or help are most welcome.

[modocache]

Awesome, @ephemer! I've been thinking about this same problem. I have some work up on a branch based off of apple/swift master: https://github.com/modocache/swift/tree/build-script-cross-compile-sdks

Here are my notes:

Currently, the Swift build scripts and CMakeLists files support limited cross-compilation of the stdlib, from OS X x86_64 to:

• iOS {armv7, armv7s, arm64}
• iOS simulator {i386, x86_64}
• tvOS arm64
• tvOS simulator x86_64
• watchOS armv7k
• watchOS simulator i386

(All other targets--Linux armv7, FreeBSD x86_64--can only be compiled on host machines of the same OS and architecture.)

This cross-compilation is achieved using dynamically generated CMake custom targets. Each custom target is added as a dependency upon the main stdlib compilation target.

Each custom target may target a different OS and architecture, so variables like CMAKE_SYSTEM_NAME cannot be used reliably. Instead, the current system succeeds in cross-compilation through a combination of assumptions (iOS, tvOS, and watchOS are all Darwin systems) and manual configuration (for example, isysroot is set based on the compilation target here then here).

People more familiar with CMake than I have indicated that this is an anti-pattern, and that scaling this out to support cross-compilation across OS X/Linux/iOS/Android would be "impossible". Instead, the suggestion is to invoke CMake multiple times. This is what projects like LLVM do. When cross-compiling from OS X to multiple other targets (such as all of the iOS/tvOS/etc targets), CMake should be invoked once for each. Doing so would allow us to reference variables like CMAKE_SYSTEM_NAME, since those would represent the target. Variables like CMAKE_HOST_SYSTEM_NAME could be used to determine information about the host.

[ephemer]

Hi @modocache, I like the look of the changes you made. Seems to make breaking apart host and target a lot more intuitive. Have you tested these for OS X/iOS targets?

[hpux735]

I too am elbow-deep in Swift CMake, but I'm working on adding support for the gold linker in linux. I hope that my changes don't conflict with either of these too much. Do you either of you have an estimate of when you expect these changes to be close to merging in? It sounds like there might be a somewhat delicate three-way merge of CMake changes on the horizon.

[modocache]

@ephemer They currently only work for all host compilation, and for cross compilation from OS X to iOS/tvOS/watchOS targets. I tried applying the same patches to this SwiftAndroid fork but encountered some errors which I should have saved somewhere... 😞

configure_sdk_unix won't crash and burn on OS X, but it sets the SWIFT_SDK_${prefix}_PATH to /, which is then used as the isysroot. To have this work for cross-compilation, I think you'll need to allow users of the build scripts to specify where a valid isysroot can be found.

configure_sdk_darwin, on the other hand, almost certainly will crash and burn on Linux--it executes xcrun, which shouldn't be available on Linux.

[modocache]

@hpux735 Great to hear! No worries here. As I mention above, I think any patches I land to support cross-compilation will be very different from what I have on my https://github.com/modocache/swift/tree/build-script-cross-compile-sdks branch. I don't have a clear picture of what those will look like yet, and I'm happy to rebase onto your changes.

[ephemer]

I think going forward I'll end up basing any further patches on modocache's
work, so if you keep that in mind and maybe even rebase onto his branch
when it's building across the supported targets, my changes here should be
relatively painless to put on top.

@modocache do you plan to continue work on this? Would be great if you get
it to a mergeable state
William Dillon notifications@github.com schrieb am Mo., 18. Jan. 2016 um
20:55:

I too am elbow-deep in Swift CMake, but I'm working on adding support for
the gold linker in linux. I hope that my changes don't conflict with either
of these too much. Do you either of you have an estimate of when you expect
these changes to be close to merging in? It sounds like there might be a
somewhat delicate three-way merge of CMake changes on the horizon.

—
Reply to this email directly or view it on GitHub
#13 (comment).

[hpux735]

Excellent, what do you think @modocache? Should I rebase off of you?

[modocache]

I am actively working on cross-compiling Swift, but I don't think my work on https://github.com/modocache/swift/tree/build-script-cross-compile-sdks is a good approach, so I wouldn't advise rebasing off of that.

I'll probably make a new branch that is based off of apple/swift master, and I'll post a comment here when that happens! 👋

[ephemer]

@modocache the approach in your branch seems more manageable than what's currently in master. I missed what the catch is. Also I'm not sure how much further I'll get without some kind of structured approach like your current branch. Are you suggesting I wait a while before continuing with this? I have a pretty good idea what errors you would have come across with SwiftAndroid btw: I've just spent all afternoon working through them myself

[modocache]

@ephemer The "catch" I'm imagining is that manually keeping track of which platform we're targeting will become increasingly unwieldy. When cross-compiling multiple targets, it is untenable. For example, let's say we're cross-compiling from OS X to iOS and Android. CMake provides a single, global LINK_FLAGS parameter. Should we set -rpath as part of these linker flags? If we're compiling for both iOS and Android then, as far as I can tell, there is no solution here.

Let me know if I'm misunderstanding something--I'm no CMake wizard.

Still, if you can take my https://github.com/modocache/swift/tree/build-script-cross-compile-sdks branch and build something useful out of it, don't let me stop you! 😄 I'm just not convinced it's the right approach.

[ephemer]

Ok, I only looked at your code on GitHub and I guess I missed this
intricacy.

It makes sense if we can just use the CMAKE_SYSTEM_NAME as the target name,
no? The problem there is that iOS SDK targets also have versioning. But
logically speaking that should be an exception rather than the rule.

I'm not sure how the LINK_FLAGS situation would be an issue if you're
invoking CMake multiple times. Wasn't that the whole point of your change,
to remove this issue?

Generally the way you'd split the code up seemed good though. I'm going to
try using your version as a base but do without the manual tracking of the
target names.

Brian Gesiak notifications@github.com schrieb am Mo., 18. Jan. 2016 um
22:09:

@ephemer https://github.com/ephemer The "catch" I'm imagining is that
manually keeping track of which platform we're targeting will become
increasingly unwieldy. When cross-compiling multiple targets, it is
untenable. For example, let's say we're cross-compiling from OS X to iOS
and Android. CMake provides a single, global LINK_FLAGS parameter
https://github.com/apple/swift/blob/6964301977ed0c09a5ea09812093260f99a2f8a6/cmake/modules/AddSwift.cmake#L1691-L1692.
Should we set -rpath as part of these linker flags
https://github.com/apple/swift/blob/6964301977ed0c09a5ea09812093260f99a2f8a6/cmake/modules/AddSwift.cmake#L1640-L1644?
If we're compiling for both iOS and Android then, as far as I can tell,
there is no solution here.

Let me know if I'm misunderstanding something--I'm no CMake wizard.

Still, if you can take my
https://github.com/modocache/swift/tree/build-script-cross-compile-sdks
branch and build something useful out of it, don't let me stop you! [image:
😄] I'm just not convinced it's the right approach.

—
Reply to this email directly or view it on GitHub
#13 (comment).

[modocache]

I'm not sure how the LINK_FLAGS situation would be an issue if you're invoking CMake multiple times. Wasn't that the whole point of your change, to remove this issue?

My change still builds all stdlib targets using a single CMake invocation, like apple/swift master does. I plan on proposing a change to have stdlib targets built via multiple CMake invocations sometime today.

Basically, this is how Swift is built now:

# build-script-impl

# Loop over each deployment target.
# This is cross-compilation for standalone Swift builds--that is,
# this builds LLVM and a Swift compiler executable for the target
# platform. (Note that this is different from the cross-compilation
# in SwiftAndroid, which builds a Swift compiler that runs on the
# host platform, and produces Swift programs that can be run on
# Android.)
for deployment_target in "${NATIVE_TOOLS_DEPLOYMENT_TARGETS[@]}" "${CROSS_TOOLS_DEPLOYMENT_TARGETS[@]}"; do
    # Loop over cmark, llvm, swift (and potentially lldb, llbuild, swiftpm, xctest, foundation)
    for product in "${PRODUCTS[@]}"; do
        # ...
        # Add all STDLIB_DEPLOYMENT_TARGETS to the targets built by
        # the `cmake --build` invocation below.
        build_targets=(all "${SWIFT_STDLIB_TARGETS[@]}")
        # ...
        # Build all stdlib targets at once.
        ${DISTCC_PUMP} "${CMAKE}" --build "${build_dir}" $(cmake_config_opt ${product}) -- ${BUILD_ARGS} ${build_targets[@]}
    done
done

This is how I'd like to change it:

# build-script-impl

# Loop over each deployment target.
for deployment_target in "${NATIVE_TOOLS_DEPLOYMENT_TARGETS[@]}" "${CROSS_TOOLS_DEPLOYMENT_TARGETS[@]}"; do
    # Loop over cmark, llvm, swift (and potentially lldb, llbuild, swiftpm, xctest, foundation)
    for product in "${PRODUCTS[@]}"; do
        # ...
        # Loop over each stdlib target.
        for stdlib_target in "${SWIFT_STDLIB_TARGETS[@]}"; do
            # ...
            # Build the stdlib target.
            ${DISTCC_PUMP} "${CMAKE}" --build "${build_dir}" $(cmake_config_opt ${product}) -- ${BUILD_ARGS} $stdlib_target}
        done
    done
done

I may be missing something, though. Definitely let me know how your approach goes, or if you see something awkward about how I plan on doing it! 👍

[ephemer]

Sounds great to me in theory. Before you get too far here though, I'd like to point you to the discussion on the swift mailing list about this topic, starting here (there are about 6 emails on the topic):
https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160118/000870.html

Given that we're building a compiler which then builds other things itself (which is always going to be tricky with this kind of dependency graph), it's possible the multiple invocation approach isn't ideal, and is something we only do when really needed.

Either way, I see the build-script AND the CMakeLists becoming unmanageable very quickly the more we work on this cross-compilation stuff. My knowledge of both python and bash are at the "just getting by" level, but I have a feeling the bash script is already way overblown in complexity. If you're going to make changes to build-script-impl, it may be worth trying to port them into the new modularised swift_build_support python modules..

[modocache]

@ephemer Absolutely. Have you seen https://bugs.swift.org/browse/SR-237? I'd love to move as much into Python as possible.

Thanks a ton for the link to the mailing lists! I'll read through them now.

[modocache]

From the mailing list discussion:

Unfortunately CMake only supports one C compiler and one linker in one CMake invocation. The "CMake way" for cross-compilation is to invoke CMake once for every part of the build that requires a different set of tools.

Cool, this mirrors my understanding as well. Glad to see I'm not completely out of my depth.

The disadvantage to this approach is that there are many CMake invocations, and many ninja invocations. This is bad for the same reasons why recursive make is bad. [1] So we would like to keep the current scheme for as many targets as possible (OS X native, OS X to iOS cross-compilation, Linux native etc.), and use the multi-CMake scheme only where strictly required.

Interesting. I'm reading the linked article now, to get an idea as to whether we should pursue multiple invocations for OS X-to-Linux cross-compilation only, or whether we should a single CMake invocation no matter what kind of cross-compilation we're doing.

[ephemer]

@modocache I had seen that bug report but missed that you were driving force behind it! Haha, keep up the good work.

From what I've found out the last couple of days, it seems that it really does make sense to run multiple CMake invocations when you want to use a different compiler/linker toolchain, as is most likely the case with e.g. OSX->Android cross compiling. That's because setting the CMAKE_C_COMPILER variable (which currently happens around build-script-impl:1060) causes all the other compiler settings (flags etc) to reset. This may be manageable but sounds like it could be more trouble than it's worth.

My plan is to try to compiling the OSX swift driver etc with the standard Xcode toolchain and use the Android toolchain to build the stdlib. There is probably a missing piece in linking the two invocations here, which may also prove to bring more pain. We'll see.

See here for more reading:
https://cmake.org/Wiki/CMake_Useful_Variables (see the notes around CMAKE_C_COMPILER)
https://cmake.org/Wiki/CMake_Cross_Compiling

[ephemer]

I feel like I'm out of my depth and don't seem to be making any progress on this at the moment. It seems like the inbuilt clang would be fine to build the android sources, we just need to tell it to use the android linker. I'm not experienced enough with compiling and linking generally to know how to go about that but it seems doable.

On a side note, I did come across this https://github.com/taka-no-me/android-cmake, which may be of interest. (it uses the multiple invocation paradigm)

[modocache]

@froody is continuing to work on this; see the mailing list thread here.

[modocache]

Closing this in order to centralize the discussion here: https://bugs.swift.org/browse/SR-1362