cross compilation support (configurable destinations) #1098

weissi · Apr 19, 2017

General

This adds experimental support for cross compilation or more precisely configurable destinations for binaries to SwiftPM. The basic idea is that you hand a JSON file in the format described below to SwiftPM (ie. swift build --destination my-destination.json). SwiftPM will then invoke the swiftc/clang with the appropriate arguments. The --destination switch is optional and if omitted SwiftPM will use the destination as appropriate for the host OS as it previously did.

`destination.json` File Format

{
    "version": 1,
    "sdk": <path to the SDK for the target platform: String>,
    "toolchain-bin-dir": <path to the toolchain's bin dir: String>,
    "target": <LLVM target triple: String>,
    "extra-cc-flags": <extra C compiler arguments: [String]>,
    "extra-swiftc-flags": <extra Swift compiler arguments: [String]>,
    "dynamic-library-extension": <extension of dynamic library files: String>,
    "extra-cpp-flags": <extra C++ compiler flags: [String]>
}

Example: compile for macOS on macOS

{
    "version": 1,
    "sdk": "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/",
    "toolchain-bin-dir": "/Library/Developer/Toolchains/swift-3.1-RELEASE.xctoolchain/usr/bin",
    "target": "x86_64-apple-macosx10.10",
    "extra-cc-flags": [
        "-arch", "x86_64",
        "-mmacosx-version-min=10.10"
    ],
    "extra-swiftc-flags": [],
    "extra-cpp-flags": ["-lc++"],
    "dynamic-library-extension": "dylib"
}

Example: compile for Ubuntu Linux on macOS (SDK required)

{
    "version": 1,
    "sdk": "/Users/johannes/cross-toolchain/ubuntu-xenial.sdk",
    "toolchain-bin-dir": "/Users/johannes/cross-toolchain/swift.xctoolchain/usr/bin",
    "target": "linux-unknown-x86_64",
    "dynamic-library-extension": "so",
    "extra-cc-flags": [
        "-fPIC"
    ],
    "extra-swiftc-flags": [
        "-use-ld=gold"
    ],
    "extra-cpp-flags": ["-lstdc++"]
}

Some Words About Cross Compilation

To cross compile a binary, you will need a cross compilation toolchain which consists of the toolchain itself and an SDK. For example compiling an iPhone binary from your Mac will use /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.3.sdk or similar. If you want to compile a Swift binary for say Ubuntu Xenial on your Mac, you'll need the following pieces of software

a compiler being able to cross compile ✅ (swiftc and clang are able to do that)
the basic Swift libraries (like libswiftCore) for Ubuntu ✅ (that's part of the Swift distribution for Ubuntu)
an SDK for Ubuntu Linux ✅ (that can be assembled from the .deb packages available from Ubuntu)
an ELF linker that runs on macOS ✅ (the gold linker from the GNU binutils can be compiled on macOS)

In other words, all the pieces of software are available for free online, they just need to be assembled in the correct way. An example script to do that all for Ubuntu Xenial can be found in a separate PR.

ddunbar · Apr 19, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r · Apr 19, 2017

Sources/Basic/Result.swift

@@ -57,6 +57,15 @@ public enum Result<Value, ErrorType: Swift.Error> {
        }
    }

+    public func flatMap<U>(_ transform: (Value) -> Result<U, ErrorType>) -> Result<U, ErrorType> {


This method needs doc comments and tests. Can you propose this as a separate change?

done (it's a separate commit so far, @aciidb0mb3r want me to make a separate PR?)

Separate PR would be nice but commit sounds fine too!

here you go: #1100

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r · Apr 19, 2017

Sources/Commands/Options.swift

@@ -33,5 +33,7 @@ public class ToolOptions {
    /// Disables sandboxing when executing subprocesses.
    public var shouldDisableSandbox = false

+    public var destinationDescriptionPath: AbsolutePath?


Please add doc comments.

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r · Apr 19, 2017

Sources/Commands/SwiftTool.swift

@@ -149,6 +149,10 @@ public class SwiftTool<Options: ToolOptions> {
            option: parser.add(option: "--experimental-use-v4-manifest", kind: Bool.self),
            to: { _, on in Versioning.simulateVersionFour = on })

+        binder.bind(
+            option: parser.add(option: "--destination", kind: PathArgument.self),


Unless we want this to be hidden option, we should add help text.

I thought we'd add it as an experimental feature first but happy to add text. @aciidb0mb3r / @ddunbar any opinions?

Ah, then no help text needed.

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r · Apr 19, 2017

Sources/Commands/UserToolchain.swift

@@ -82,7 +69,7 @@ public struct UserToolchain: Toolchain, ManifestResourceProvider {
        }

        // Look for llbuild in bin dir.
-        llbuild = binDir.appending(component: "swift-build-tool")
+        llbuild = lookup(fromEnv: "LLBUILD_EXEC") ?? binDir.appending(component: "swift-build-tool")


Is this needed for this patch?

no, removed

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r · Apr 19, 2017

Sources/PackageLoading/ManifestLoader.swift

@@ -234,7 +234,7 @@ public final class ManifestLoader: ManifestLoaderProtocol {
        // and validates it.

        // Compute the path to runtime we need to load.
-        let runtimePath = resources.libDir.appending(component: String(manifestVersion.rawValue)).asString
+        let runtimePath = resources.libDir.asString


This shouldn't be changed.

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r · Apr 19, 2017

Sources/PackageModel/Destination.swift

+    }
+}
+
+public struct Destination {


This looks like something which should be a part of toolchain and not each target and product.

@aciidb0mb3r the problem is that Product requires the shared library extension which is calculated from the Destination. So I just wired the Destination through everything. Prefer me to wire through the extension only or the UserToolchain?

Ah, for outname property? I think we should just lift that into the Build module. Doing that should allow keeping the destination completely inside the toolchain.

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r · Apr 19, 2017

Sources/PackageModel/Destination.swift

+
+    public var extraCCFlags: [String]
+
+    public var extraSwiftCFlags: [String]


We can use let, right?

yes, but it's a value type so changes only affect the very local variable. I'm happy to make it let but don't think it makes much difference because it's a struct. Let me know what you think.

I think its better to start with let unless you really need var because it is part of the API. If something is let, clients won't have a question "what happens if I change the value". It also helps in case you need to convert the struct to a class.

cool, changed.

aciidb0mb3r · Apr 19, 2017

aciidb0mb3r requested changes Apr 19, 2017

View changes

I think my major feedback is, moving the destination to be part of the UserToolchain and not have it per target and product. Also, please add some brief Swift doc comments where applicable.

weissi · Apr 19, 2017

thanks very much @aciidb0mb3r for your review. Will work on the bigger items tomorrow, fixed some smaller ones.

weissi · Apr 20, 2017

hey @aciidb0mb3r I now moved Destination from the PackageModel module into the Build module and took the outname out into the Build module too. Is this roughly what you were thinking?

weissi · Apr 20, 2017

@aciidb0mb3r I also added a new property "extra-cpp-flags" for flags for C++, I discovered earlier that that was still handled through #if os(macOS) which is bad...

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/PackageLoading/PackageBuilder.swift

@@ -268,6 +268,8 @@ public final class PackageBuilder {
        }
    }

+    private let outputFileName: (String, PackageModel.ProductType) -> RelativePath


The package builder is the convention system that loads a package from disk, it doesn't need to care about the output file name. We should move the computation of output path/name into the class ProductBuildDescription (It is inside BuildPlan.swift file).

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/PackageLoading/PackageBuilder.swift

@@ -785,7 +789,8 @@ public final class PackageBuilder {
                sources: Sources(paths: swiftSources, root: potentialModule.path),
                dependencies: moduleDependencies,
                productDependencies: productDeps,
-                swiftVersion: try swiftVersion())
+                swiftVersion: try swiftVersion()
+                )


Please avoid the diffs that do not contribute any functionality.

sorry about that, didn't spot those, removed.

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/PackageModel/Module.swift

+            type: .systemModule,
+            sources: sources,
+            dependencies: []
+            )


These diffs are not required, right?

sorry about that, didn't spot those, removed.

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/PackageModel/Product.swift

@@ -41,6 +41,9 @@ public class Product {
    /// The name of the product.
    public let name: String

+    /// The output file name of the product.
+    public let outname: RelativePath


As I commented above, we don't need to have the outname inside the target. It can be a build time thing.

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/TestSupport/Resources.swift

-        binDir = AbsolutePath(CommandLine.arguments[0], relativeTo: currentWorkingDirectory).parentDirectory
-      #endif
-        toolchain = try! UserToolchain(binDir)
+        toolchain = try! UserToolchain(destination: Destination.hostDestination())


We do these special things here to run tests inside Xcode, please ensure the tests work inside Xcode too by generating a project using Utilities/bootstrap --generate-xcodeproj

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/Build/Destination.swift

+    }
+}
+
+public struct Destination {


Please add a short description about this structure.

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/Build/Destination.swift

+
+    public let extraSwiftCFlags: [String]
+
+    public let extraCPPFlags: [String]


Please add a doc comment for each property

aciidb0mb3r · Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r · Apr 20, 2017

Sources/Build/Destination.swift

+            let sdkPlatformFrameworksPath = AbsolutePath(platformPath).appending(components: "Developer", "Library", "Frameworks")
+
+            let devDirPath = try Process.checkNonZeroExit(args: "xcode-select", "-p").chomp()
+            let sysFWPath = AbsolutePath(sdkPath).appending(component: "System")


Please avoid abbreviated variable names. We try to follow Swift design guidelines https://swift.org/documentation/api-design-guidelines/

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

aciidb0mb3r approved these changes Apr 25, 2017

View changes

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

aciidb0mb3r · Apr 25, 2017

@swift-ci please smoke test

weissi referenced this pull request Apr 19, 2017
Merged
add script to build Ubuntu cross compilation toolchain #1099

weissi force-pushed the weissi:jw-cross-compilation branch from f18c819 to fe75f07 Apr 19, 2017

weissi force-pushed the weissi:jw-cross-compilation branch 8 times, most recently from 889b7f6 to 455b06e Apr 19, 2017

weissi force-pushed the weissi:jw-cross-compilation branch from 981ce26 to 570295f Apr 24, 2017

aciidb0mb3r force-pushed the weissi:jw-cross-compilation branch 9 times, most recently from 76572bc to 65d4811 Apr 24, 2017

weissi force-pushed the weissi:jw-cross-compilation branch from 65d4811 to 4218b22 Apr 25, 2017

weissi force-pushed the weissi:jw-cross-compilation branch 2 times, most recently from c0d307e to 51c05a6 Apr 25, 2017

weissi force-pushed the weissi:jw-cross-compilation branch from 51c05a6 to a45aa0e Apr 25, 2017

cross compilation support (configurable destinations)

Loading status checks…

23c1611

weissi force-pushed the weissi:jw-cross-compilation branch from a45aa0e to 23c1611 Apr 25, 2017

aciidb0mb3r merged commit ff9879d into apple:master Apr 25, 2017
2 checks passed

2 checks passed

Swift Test Linux Platform (smoke test) Build finished.
Details

Swift Test OS X Platform (smoke test) Build finished.
Details

mmoaay referenced this pull request Mar 27, 2018
Closed
An Update on Swift 3.1.1 For Raspberry Pi Zero/1/2/3 #52

apple/swift-package-manager

Join GitHub today

cross compilation support (configurable destinations) #1098

Conversation

weissi commented Apr 19, 2017 • edited

General

destination.json File Format

Example: compile for macOS on macOS

Example: compile for Ubuntu Linux on macOS (SDK required)

Some Words About Cross Compilation

This comment has been minimized.

ddunbar commented Apr 19, 2017

weissi referenced this pull request Apr 19, 2017

aciidb0mb3r reviewed Apr 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

weissi Apr 19, 2017 • edited

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 19, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r requested changes Apr 19, 2017 View changes

weissi force-pushed the weissi:jw-cross-compilation branch from f18c819 to fe75f07 Apr 19, 2017

This comment has been minimized.

weissi commented Apr 19, 2017

weissi force-pushed the weissi:jw-cross-compilation branch 8 times, most recently from 889b7f6 to 455b06e Apr 19, 2017

This comment has been minimized.

weissi commented Apr 20, 2017

This comment has been minimized.

weissi commented Apr 20, 2017

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

This comment has been minimized.

aciidb0mb3r reviewed Apr 20, 2017 View changes

This comment has been minimized.

This comment has been minimized.

weissi force-pushed the weissi:jw-cross-compilation branch from 981ce26 to 570295f Apr 24, 2017

aciidb0mb3r force-pushed the weissi:jw-cross-compilation branch 9 times, most recently from 76572bc to 65d4811 Apr 24, 2017

weissi force-pushed the weissi:jw-cross-compilation branch from 65d4811 to 4218b22 Apr 25, 2017

This comment has been minimized.

aciidb0mb3r commented Apr 25, 2017

aciidb0mb3r approved these changes Apr 25, 2017 View changes

This comment has been minimized.

aciidb0mb3r commented Apr 25, 2017

weissi commented Apr 19, 2017 •

edited

`destination.json` File Format

aciidb0mb3r reviewed Apr 19, 2017

View changes

weissi Apr 19, 2017 •

edited

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r reviewed Apr 19, 2017

View changes

aciidb0mb3r requested changes Apr 19, 2017

View changes

weissi force-pushed the weissi:jw-cross-compilation branch from `f18c819` to `fe75f07` Apr 19, 2017

weissi force-pushed the weissi:jw-cross-compilation branch 8 times, most recently from `889b7f6` to `455b06e` Apr 19, 2017

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r reviewed Apr 20, 2017

View changes

aciidb0mb3r reviewed Apr 20, 2017

View changes

weissi force-pushed the weissi:jw-cross-compilation branch from `981ce26` to `570295f` Apr 24, 2017

aciidb0mb3r force-pushed the weissi:jw-cross-compilation branch 9 times, most recently from `76572bc` to `65d4811` Apr 24, 2017

weissi force-pushed the weissi:jw-cross-compilation branch from `65d4811` to `4218b22` Apr 25, 2017

aciidb0mb3r approved these changes Apr 25, 2017

View changes

weissi force-pushed the weissi:jw-cross-compilation branch 2 times, most recently from `c0d307e` to `51c05a6` Apr 25, 2017

weissi force-pushed the weissi:jw-cross-compilation branch from `51c05a6` to `a45aa0e` Apr 25, 2017

weissi force-pushed the weissi:jw-cross-compilation branch from `a45aa0e` to `23c1611` Apr 25, 2017

aciidb0mb3r merged commit `ff9879d` into apple:master Apr 25, 2017
2 checks passed