[SR-13312] SPM Destinations are b0rked in 5.2.4

[swift-ci]

Previous ID	SR-13312
Radar	None
Original Reporter	helge (JIRA User)
Type	Bug

Environment

10.15.6, Xcode 11.6

zMBP20:~ helge$ swift --version

Apple Swift version 5.2.4 (swiftlang-1103.0.32.9 clang-1103.0.32.53)

Target: x86_64-apple-darwin19.6.0

Additional Detail from JIRA

Votes	0
Component/s	Package Manager
Labels	Bug
Assignee	None
Priority	Medium

md5: 5d012d9d70763762258577c97106148b

Issue Description:

The `build_ubuntu_cross_compilation_toolchain` included in SPM runs through, but the resulting toolchain doesn't work anymore when compiling the (empty) example app.

I think the issue is that the SDK is not picked up:

zMBP20:my-test-app helge$ swift build -v --destination /tmp/ubuntu/cross-toolchain/ubuntu-xenial-destination.json
xcrun --sdk macosx --find xctest
xcrun --sdk macosx --show-sdk-platform-path
xcrun --sdk macosx --find xctest
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc -L /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/4_2 -lPackageDescription -Xlinker -rpath -Xlinker /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/4_2 -swift-version 5 -I /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/4_2 -target x86_64-apple-macosx10.10 -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -package-description-version 5.2.0 /private/tmp/ubuntu/my-test-app/Package.swift -o /var/folders/r6/6_zwr7ns5g16txb9ly8s04x00000gp/T/TemporaryDirectory.iICq9p/my-test-app-manifest
/var/folders/r6/6_zwr7ns5g16txb9ly8s04x00000gp/T/TemporaryDirectory.iICq9p/my-test-app-manifest -fileno 1 sandbox-exec -p '(version 1)
(deny default)
(import "system.sb")
(allow file-read*)
(allow process*)
(allow sysctl*)
(allow file-write*
    (regex #"^/private/var/tmp/org\.llvm\.clang.*")
    (regex #"^/var/folders/r6/6_zwr7ns5g16txb9ly8s04x00000gp/T/org\.llvm\.clang.*")
    (regex #"^/private/var/folders/r6/6_zwr7ns5g16txb9ly8s04x00000gp/T/org\.llvm\.clang.*")
    (regex #"^/private/var/folders/r6/6_zwr7ns5g16txb9ly8s04x00000gp/C/org\.llvm\.clang.*")
    (subpath "/private/tmp/ubuntu/my-test-app/.build")
)
'
/tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin/swiftc -module-name my_test_app -incremental -emit-dependencies -emit-module -emit-module-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.swiftmodule -output-file-map /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/output-file-map.json -c /private/tmp/ubuntu/my-test-app/Sources/my-test-app/main.swift -I /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug -target x86_64-unknown-linux -swift-version 5 -enable-batch-mode -index-store-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/index/store -use-ld=lld -tools-directory /tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin -Onone -enable-testing -g -j8 -DSWIFT_PACKAGE -DDEBUG -module-cache-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/ModuleCache -parseable-output -color-diagnostics
/private/tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin/swift -frontend -c -primary-file /private/tmp/ubuntu/my-test-app/Sources/my-test-app/main.swift -emit-module-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main~partial.swiftmodule -emit-module-doc-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main~partial.swiftdoc -emit-module-source-info-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main~partial.swiftsourceinfo -emit-dependencies-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main.d -emit-reference-dependencies-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main.swiftdeps -target x86_64-unknown-linux -disable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -I /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug -color-diagnostics -enable-testing -g -module-cache-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/ModuleCache -swift-version 5 -Onone -D SWIFT_PACKAGE -D DEBUG -enable-anonymous-context-mangled-names -module-name my_test_app -o /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main.swift.o -index-store-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/index/store -index-system-modules
/private/tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin/swift -frontend -merge-modules -emit-module /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main~partial.swiftmodule -parse-as-library -sil-merge-partial-modules -disable-diagnostic-passes -disable-sil-perf-optzns -target x86_64-unknown-linux -disable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -I /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug -color-diagnostics -enable-testing -g -module-cache-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/ModuleCache -swift-version 5 -Onone -D SWIFT_PACKAGE -D DEBUG -enable-anonymous-context-mangled-names -emit-module-doc-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.swiftdoc -emit-module-source-info-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.swiftsourceinfo -module-name my_test_app -o /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.swiftmodule
/tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin/swiftc -modulewrap /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.swiftmodule -o /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/my_test_app.swiftmodule.o
/tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin/swiftc -use-ld=lld -tools-directory /tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin -L /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug -o /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my-test-app -module-name my_test_app -emit-executable -Xlinker '-rpath=$ORIGIN' @/private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my-test-app.product/Objects.LinkFileList -target x86_64-unknown-linux -L /private/tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/lib
<unknown>:0: error: error opening input file '/private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/my_test_app.swiftmodule.o' (Don't know how to extract from object fileformat)
<unknown>:0: error: swift-autolink-extract command failed with exit code 1 (use -v to see invocation) 

and indeed, that .o file is a Mach-O, not an ELF one:

zMBP20:my-test-app helge$ file /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/my_test_app.swiftmodule.o
/private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/my_test_app.swiftmodule.o: Mach-O 64-bit object x86_64 

[swift-ci]

Comment by Helge Heß (JIRA)

I think this is the problematic call:

/private/tmp/ubuntu/cross-toolchain/swift.xctoolchain/usr/bin/swift \
  -frontend -c \
  -primary-file /private/tmp/ubuntu/my-test-app/Sources/my-test-app/main.swift \
  -emit-module-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main~partial.swiftmodule \
  -emit-module-doc-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main~partial.swiftdoc \
  -emit-module-source-info-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main~partial.swiftsourceinfo \
  -emit-dependencies-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main.d \
  -emit-reference-dependencies-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main.swiftdeps \
  -target x86_64-unknown-linux -disable-objc-interop \
  -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk \
  -I /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug \
  -color-diagnostics -enable-testing -g \
  -module-cache-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/ModuleCache \
  -swift-version 5 -Onone -D SWIFT_PACKAGE -D DEBUG -enable-anonymous-context-mangled-names \
  -module-name my_test_app \
  -o /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/main.swift.o \
  -index-store-path /private/tmp/ubuntu/my-test-app/.build/x86_64-unknown-linux/debug/index/store \
  -index-system-modules 

Specifically the `-sdk /Applications/Xcode.app/..../MacOSX1.15.sdk`, all the rest seems to be good.

[swift-ci]

Comment by Helge Heß (JIRA)

Steps to reproduce (no Docker or anything required, just a regular Swift install).

First make sure that the `clang_package_url` in the [`build_ubuntu_cross_compilation_toolchain`](https://github.com/apple/swift-package-manager/blob/master/Utilities/build_ubuntu_cross_compilation_toolchain) has a `https` URL, not `http`. Otherwise straight forward:

curl -o ~/Downloads/swift-5.2.4-RELEASE-ubuntu16.04.tar.gz \
    https://swift.org/builds/swift-5.2.4-release/ubuntu1604/swift-5.2.4-RELEASE/swift-5.2.4-RELEASE-ubuntu16.04.tar.gz
curl -o ~/Downloads/swift-5.2.4-RELEASE-osx.pkg \
    https://swift.org/builds/swift-5.2.4-release/xcode/swift-5.2.4-RELEASE/swift-5.2.4-RELEASE-osx.pkg
./build_ubuntu_cross_compilation_toolchain /tmp \
  ~/Downloads/swift-5.2.4-RELEASE-osx.pkg \
  ~/Downloads/swift-5.2.4-RELEASE-ubuntu16.04.tar.gz

mkdir my-test-app && cd my-test-app && swift package init --type=executable
swift build --destination /tmp/cross-toolchain/ubuntu-xenial-destination.json
 

[swift-ci]

Comment by Helge Heß (JIRA)

Some more research on this.

Actually the call outputting the `main.swift.o` seems to be good, it produces:

.build/x86_64-unknown-linux/debug/my_test_app.build/main.swift.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), with debug_info, not stripped 

The next call is the one producing the `my_test_app.swiftmodule`, no idea whether that is OK.

But the subsequent call which produces `my_test_app.swiftmodule.o` emits a Mach-O:

/tmp/cross-toolchain/swift.xctoolchain/usr/bin/swiftc \
  -modulewrap /Users/helge/Dropbox/dev/Tests/awstoolchain/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.swiftmodule \
  -o /Users/helge/Dropbox/dev/Tests/awstoolchain/my-test-app/.build/x86_64-unknown-linux/debug/my_test_app.build/my_test_app.swiftmodule.o

Gives:

.build/x86_64-unknown-linux/debug/my_test_app.build/my_test_app.swiftmodule.o: Mach-O 64-bit object x86_64

which the subsequent linking step can't process:

<unknown>:0: error: error opening input file '/.build/x86_64-unknown-linux/debug/my_test_app.build/my_test_app.swiftmodule.o' (Don't know how to extract from object fileformat)

[swift-ci]

Comment by Helge Heß (JIRA)

Adding

  -target x86_64-unknown-linux

to the `-modulewrap` call seems to fix the issue. (the linking one)

The linking itself fails, but that may be a different issue:

$ /tmp/cross-toolchain/swift.xctoolchain/usr/bin/swiftc -use-ld=lld -tools-directory /tmp/cross-toolchain/swift.xctoolchain/usr/bin -L /Users/helge/Dropbox/dev/Tests/awstoolchain/my-test-app/.build/x86_64-unknown-linux/debug -o /Users/helge/Dropbox/dev/Tests/awstoolchain/my-test-app/.build/x86_64-unknown-linux/debug/my-test-app -module-name my_test_app -emit-executable -Xlinker '-rpath=$ORIGIN' @/Users/helge/Dropbox/dev/Tests/awstoolchain/my-test-app/.build/x86_64-unknown-linux/debug/my-test-app.product/Objects.LinkFileList -target x86_64-unknown-linux -L /private/tmp/cross-toolchain/swift.xctoolchain/usr/lib
/tmp/cross-toolchain/swift.xctoolchain/usr/bin/ld.lld: error: cannot open Scrt1.o: No such file or directory
...

The missing crt .o files live in `ubuntu-xenial.sdk/usr/lib/x86_64-linux-gnu`, the missing libgcc etc in `ubuntu-xenial.sdk/usr/lib/gcc/x86_64-linux-gnu/5`. By adding the latter to `-L` I can get rid of the libgcc linker errors, but `-L` of course doesn't help with the `.o` files.

[swift-ci]

Comment by Helge Heß (JIRA)

The modulewrap invocation (missing the target) seems to be fixed in 5.3. Would be nice to know when this was first b0rked.

There are still the linking errors though.

[swift-ci]

Comment by Helge Heß (JIRA)

To fix the linking errors, usr/lib/swift and swift_static also (or only?) need to be copied into the target toolchain as well (the script only copies them into the host toolchain, maybe they have been looked up there before).

In addition (and I think this is just a hack-a-round to the original bug), the SDK must be explicitly passed in the `extra-swiftc-flags`, like so (thanks go to @drexin):

"extra-swiftc-flags": [
  "-use-ld=lld", 
  "-tools-directory", "${BUILD_DIR}/${CROSS_TOOLCHAIN_NAME}/$xc_tc_name/usr/bin",
 "-sdk", "${BUILD_DIR}/${CROSS_TOOLCHAIN_NAME}/$linux_sdk_name"
],

(note that the JSON already has a proper "sdk" property, which presumably is supposed to do this automatically - that the extra works might be pure luck [override the macOS sdk in the lookup sequence])