New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cross compilation support (configurable destinations) #1098

Merged
merged 1 commit into from Apr 25, 2017

Conversation

Projects
None yet
3 participants
@weissi
Collaborator

weissi commented Apr 19, 2017

General

This adds experimental support for cross compilation or more precisely configurable destinations for binaries to SwiftPM. The basic idea is that you hand a JSON file in the format described below to SwiftPM (ie. swift build --destination my-destination.json). SwiftPM will then invoke the swiftc/clang with the appropriate arguments. The --destination switch is optional and if omitted SwiftPM will use the destination as appropriate for the host OS as it previously did.

destination.json File Format

{
    "version": 1,
    "sdk": <path to the SDK for the target platform: String>,
    "toolchain-bin-dir": <path to the toolchain's bin dir: String>,
    "target": <LLVM target triple: String>,
    "extra-cc-flags": <extra C compiler arguments: [String]>,
    "extra-swiftc-flags": <extra Swift compiler arguments: [String]>,
    "dynamic-library-extension": <extension of dynamic library files: String>,
    "extra-cpp-flags": <extra C++ compiler flags: [String]>
}

Example: compile for macOS on macOS

{
    "version": 1,
    "sdk": "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/",
    "toolchain-bin-dir": "/Library/Developer/Toolchains/swift-3.1-RELEASE.xctoolchain/usr/bin",
    "target": "x86_64-apple-macosx10.10",
    "extra-cc-flags": [
        "-arch", "x86_64",
        "-mmacosx-version-min=10.10"
    ],
    "extra-swiftc-flags": [],
    "extra-cpp-flags": ["-lc++"],
    "dynamic-library-extension": "dylib"
}

Example: compile for Ubuntu Linux on macOS (SDK required)

{
    "version": 1,
    "sdk": "/Users/johannes/cross-toolchain/ubuntu-xenial.sdk",
    "toolchain-bin-dir": "/Users/johannes/cross-toolchain/swift.xctoolchain/usr/bin",
    "target": "linux-unknown-x86_64",
    "dynamic-library-extension": "so",
    "extra-cc-flags": [
        "-fPIC"
    ],
    "extra-swiftc-flags": [
        "-use-ld=gold"
    ],
    "extra-cpp-flags": ["-lstdc++"]
}

Some Words About Cross Compilation

To cross compile a binary, you will need a cross compilation toolchain which consists of the toolchain itself and an SDK. For example compiling an iPhone binary from your Mac will use /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.3.sdk or similar. If you want to compile a Swift binary for say Ubuntu Xenial on your Mac, you'll need the following pieces of software

  • a compiler being able to cross compile (swiftc and clang are able to do that)
  • the basic Swift libraries (like libswiftCore) for Ubuntu (that's part of the Swift distribution for Ubuntu)
  • an SDK for Ubuntu Linux (that can be assembled from the .deb packages available from Ubuntu)
  • an ELF linker that runs on macOS (the gold linker from the GNU binutils can be compiled on macOS)

In other words, all the pieces of software are available for free online, they just need to be assembled in the correct way. An example script to do that all for Ubuntu Xenial can be found in a separate PR.

@ddunbar

This comment has been minimized.

Member

ddunbar commented Apr 19, 2017

@swift-ci please smoke test

@@ -57,6 +57,15 @@ public enum Result<Value, ErrorType: Swift.Error> {
}
}
public func flatMap<U>(_ transform: (Value) -> Result<U, ErrorType>) -> Result<U, ErrorType> {

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

This method needs doc comments and tests. Can you propose this as a separate change?

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

done (it's a separate commit so far, @aciidb0mb3r want me to make a separate PR?)

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

Separate PR would be nice but commit sounds fine too!

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

here you go: #1100

@@ -33,5 +33,7 @@ public class ToolOptions {
/// Disables sandboxing when executing subprocesses.
public var shouldDisableSandbox = false
public var destinationDescriptionPath: AbsolutePath?

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

Please add doc comments.

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

done

@@ -149,6 +149,10 @@ public class SwiftTool<Options: ToolOptions> {
option: parser.add(option: "--experimental-use-v4-manifest", kind: Bool.self),
to: { _, on in Versioning.simulateVersionFour = on })
binder.bind(
option: parser.add(option: "--destination", kind: PathArgument.self),

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

Unless we want this to be hidden option, we should add help text.

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

I thought we'd add it as an experimental feature first but happy to add text. @aciidb0mb3r / @ddunbar any opinions?

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

Ah, then no help text needed.

@@ -82,7 +69,7 @@ public struct UserToolchain: Toolchain, ManifestResourceProvider {
}
// Look for llbuild in bin dir.
llbuild = binDir.appending(component: "swift-build-tool")
llbuild = lookup(fromEnv: "LLBUILD_EXEC") ?? binDir.appending(component: "swift-build-tool")

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

Is this needed for this patch?

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

no, removed

@@ -234,7 +234,7 @@ public final class ManifestLoader: ManifestLoaderProtocol {
// and validates it.
// Compute the path to runtime we need to load.
let runtimePath = resources.libDir.appending(component: String(manifestVersion.rawValue)).asString
let runtimePath = resources.libDir.asString

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

This shouldn't be changed.

This comment has been minimized.

@weissi

weissi Apr 20, 2017

Collaborator

done

}
}
public struct Destination {

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

This looks like something which should be a part of toolchain and not each target and product.

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

@aciidb0mb3r the problem is that Product requires the shared library extension which is calculated from the Destination. So I just wired the Destination through everything. Prefer me to wire through the extension only or the UserToolchain?

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

Ah, for outname property? I think we should just lift that into the Build module. Doing that should allow keeping the destination completely inside the toolchain.

This comment has been minimized.

@weissi

weissi Apr 20, 2017

Collaborator

done

public var extraCCFlags: [String]
public var extraSwiftCFlags: [String]

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

We can use let, right?

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

yes, but it's a value type so changes only affect the very local variable. I'm happy to make it let but don't think it makes much difference because it's a struct. Let me know what you think.

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 19, 2017

Member

I think its better to start with let unless you really need var because it is part of the API. If something is let, clients won't have a question "what happens if I change the value". It also helps in case you need to convert the struct to a class.

This comment has been minimized.

@weissi

weissi Apr 19, 2017

Collaborator

cool, changed.

@aciidb0mb3r

I think my major feedback is, moving the destination to be part of the UserToolchain and not have it per target and product. Also, please add some brief Swift doc comments where applicable.

@weissi weissi force-pushed the weissi:jw-cross-compilation branch from f18c819 to fe75f07 Apr 19, 2017

@weissi

This comment has been minimized.

Collaborator

weissi commented Apr 19, 2017

thanks very much @aciidb0mb3r for your review. Will work on the bigger items tomorrow, fixed some smaller ones.

@weissi weissi force-pushed the weissi:jw-cross-compilation branch 8 times, most recently from 889b7f6 to 455b06e Apr 19, 2017

@weissi

This comment has been minimized.

Collaborator

weissi commented Apr 20, 2017

hey @aciidb0mb3r I now moved Destination from the PackageModel module into the Build module and took the outname out into the Build module too. Is this roughly what you were thinking?

@weissi

This comment has been minimized.

Collaborator

weissi commented Apr 20, 2017

@aciidb0mb3r I also added a new property "extra-cpp-flags" for flags for C++, I discovered earlier that that was still handled through #if os(macOS) which is bad...

@@ -268,6 +268,8 @@ public final class PackageBuilder {
}
}
private let outputFileName: (String, PackageModel.ProductType) -> RelativePath

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

The package builder is the convention system that loads a package from disk, it doesn't need to care about the output file name. We should move the computation of output path/name into the class ProductBuildDescription (It is inside BuildPlan.swift file).

@@ -785,7 +789,8 @@ public final class PackageBuilder {
sources: Sources(paths: swiftSources, root: potentialModule.path),
dependencies: moduleDependencies,
productDependencies: productDeps,
swiftVersion: try swiftVersion())
swiftVersion: try swiftVersion()
)

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

Please avoid the diffs that do not contribute any functionality.

This comment has been minimized.

@weissi

weissi Apr 20, 2017

Collaborator

sorry about that, didn't spot those, removed.

type: .systemModule,
sources: sources,
dependencies: []
)

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

These diffs are not required, right?

This comment has been minimized.

@weissi

weissi Apr 20, 2017

Collaborator

sorry about that, didn't spot those, removed.

@@ -41,6 +41,9 @@ public class Product {
/// The name of the product.
public let name: String
/// The output file name of the product.
public let outname: RelativePath

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

As I commented above, we don't need to have the outname inside the target. It can be a build time thing.

binDir = AbsolutePath(CommandLine.arguments[0], relativeTo: currentWorkingDirectory).parentDirectory
#endif
toolchain = try! UserToolchain(binDir)
toolchain = try! UserToolchain(destination: Destination.hostDestination())

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

We do these special things here to run tests inside Xcode, please ensure the tests work inside Xcode too by generating a project using Utilities/bootstrap --generate-xcodeproj

}
}
public struct Destination {

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

Please add a short description about this structure.

This comment has been minimized.

@weissi

weissi Apr 20, 2017

Collaborator

done

public let extraSwiftCFlags: [String]
public let extraCPPFlags: [String]

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

Please add a doc comment for each property

This comment has been minimized.

@weissi

weissi Apr 20, 2017

Collaborator

done

let sdkPlatformFrameworksPath = AbsolutePath(platformPath).appending(components: "Developer", "Library", "Frameworks")
let devDirPath = try Process.checkNonZeroExit(args: "xcode-select", "-p").chomp()
let sysFWPath = AbsolutePath(sdkPath).appending(component: "System")

This comment has been minimized.

@aciidb0mb3r

aciidb0mb3r Apr 20, 2017

Member

Please avoid abbreviated variable names. We try to follow Swift design guidelines https://swift.org/documentation/api-design-guidelines/

This comment has been minimized.

@weissi

weissi Apr 20, 2017

Collaborator

done

@weissi weissi force-pushed the weissi:jw-cross-compilation branch from 981ce26 to 570295f Apr 24, 2017

@aciidb0mb3r aciidb0mb3r force-pushed the weissi:jw-cross-compilation branch 9 times, most recently from 76572bc to 65d4811 Apr 24, 2017

@weissi weissi force-pushed the weissi:jw-cross-compilation branch from 65d4811 to 4218b22 Apr 25, 2017

@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

2 similar comments
@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

@weissi weissi force-pushed the weissi:jw-cross-compilation branch 2 times, most recently from c0d307e to 51c05a6 Apr 25, 2017

@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

1 similar comment
@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

@weissi weissi force-pushed the weissi:jw-cross-compilation branch from 51c05a6 to a45aa0e Apr 25, 2017

@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

1 similar comment
@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

@weissi weissi force-pushed the weissi:jw-cross-compilation branch from a45aa0e to 23c1611 Apr 25, 2017

@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

1 similar comment
@aciidb0mb3r

This comment has been minimized.

Member

aciidb0mb3r commented Apr 25, 2017

@swift-ci please smoke test

@aciidb0mb3r aciidb0mb3r merged commit ff9879d into apple:master Apr 25, 2017

2 checks passed

Swift Test Linux Platform (smoke test) Build finished.
Details
Swift Test OS X Platform (smoke test) Build finished.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment