Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-14035] Compiler toolchain build error on M1 mac #56426

Closed
johnno1962 opened this issue Jan 11, 2021 · 17 comments
Closed

[SR-14035] Compiler toolchain build error on M1 mac #56426

johnno1962 opened this issue Jan 11, 2021 · 17 comments
Assignees

Comments

@johnno1962
Copy link
Contributor

johnno1962 commented Jan 11, 2021

Previous ID SR-14035
Radar rdar://problem/73014167
Original Reporter @johnno1962
Type Bug
Additional Detail from JIRA
Votes 0
Component/s Compiler, Project Infrastructure
Labels Bug
Assignee @shahmishal
Priority Medium

md5: 10a69735b885909bbf12d3e950153532

Issue Description:

HI Apple,

I've tried any number of things but have been unable to build a toolchain using swift/main & Xcode 12.3 and the swift/utils/build-toolchain script. It seems to be something to do with supporting multiple architectures for macOS and simulator w/lipo. This warning may be relevant:

warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: lib/swift/watchsimulator/i386/libswiftCompatibility51.a will be fat and ar(1) will not be able to operate on it

@typesanitizer
Copy link

typesanitizer commented Jan 11, 2021

@swift-ci create

@johnno1962
Copy link
Contributor Author

johnno1962 commented Apr 21, 2021

Still unable to build a toolchain on an M1 using a fresh clone. Error messages include:

FAILED: lib/swift/macosx/libswiftCompatibilityDynamicReplacements.a
cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/CompatibilityDynamicReplacements && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibilityDynamicReplacements.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibilityDynamicReplacements.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibilityDynamicReplacements.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibilityDynamicReplacements.a
fatal error: /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibilityDynamicReplacements.a and /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibilityDynamicReplacements.a have the same architectures (arm64) and can't be in the same fat output file
[22/1588][ 1%][1.247s] cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/Compatibility50 && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibility50.a
FAILED: lib/swift/macosx/libswiftCompatibility50.a
cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/Compatibility50 && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility50.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibility50.a
fatal error: /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility50.a and /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility50.a have the same architectures (arm64) and can't be in the same fat output file
[23/1588][ 1%][1.256s] cd /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/stdlib/toolchain/Compatibility51 && /Applications/Xcode124.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo -create -output /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/./lib/swift/macosx/libswiftCompatibility51.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/x86_64/libswiftCompatibility51.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64/libswiftCompatibility51.a /Volumes/Elements/swift-convert/build/buildbot_osx/swift-macosx-arm64/lib/swift/macosx/arm64e/libswiftCompatibility51.a
FAILED: lib/swift/macosx/libswiftCompatibility51.a

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 15, 2021

I've finally been able to build a toolchain for myself on an M1 Mac. It took a good while 🙂. There are about 4 problems that need to be addressed by someone who know more about CMake than I do. The first is the error above where lipo fails when trying to merge the archives for the three architectures x86_64, arm64 and arm64e as they all seem to have been built for arm64. It's odd that this problem should be specific to build on an arm64 machine as I have built "fat" toolchains recently on an Intel machine but I couldn't investigate it any further. In the end I changed SUPPORTED_OSX_ARCHS in the file cmake/modules/DarwinSDKs.cmake to only include arm64 and not attempt a "fat" build (besides it takes a good deal less time!)

The next problem seems to be related to llvm where you get an undefined symbol for ___isOSVersionAtLeast in the source file stdlib/public/runtime/ImageInspectionMachO.cpp. I tried various things but in the end defined my own version of this function returning true in this file.

The third relatively minor problem is with the Swift package manager where I have filed a straightforward PR apple/swift-package-manager#3500 and, with fixes for these problems the toolchain build proceeds most of the way through until you get to another place where you have to provide a symbol ___isOSVersionAtLeast in indexstore-db/lib/LLVMSupport/Support/Path.cpp and you will be rewarded with a toolchain (if not the symbols archive.)

As you can see these are relatively minor problems which someone with knowledge of the CMake files could make easy work of. I'm not sure what the cause of the llvm problem would be. I'm amazed nobody on the compiler team has made the switch to M1 yet as they are just so blazingly fast – or do they know there is something even better on the horizon 🙂.

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 15, 2021

Any ideas @compnerd?

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 17, 2021

I've preformed a new clone and replicated these changes though the following were also necessary:

The missing symbol has changed in the last month from _*isOSVersionAtLeast to *_isPlatformVersionAtLeast

So, in the interim, I've added to swift/stdlib/public/runtime/ImageInspectionMachO.cpp:

extern "C" bool __isPlatformVersionAtLeast();
bool __isPlatformVersionAtLeast()}}{{{}}
{{ return true;}}
{{}}}

You need to copy build/buildbot_osx/lldb-macosx-arm64/bin/lldb-tblgen from an x86_64 build for lldb to build.

The benchmarks fail to run with a undefined swift symbol that doesn't demangle so I've had to:

dyld: Symbol not found: _SayxSicig16_Differentiation14DifferentiableRzlWJrUSpSr

cat >build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Onone

#!/bin/bash

exit 0

^D

cp build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Onone build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_O

cp build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Onone build/buildbot_osx/benchmarks-macosx-arm64/bin/Benchmark_Osize

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 18, 2021

The complete diff for the various changes I had to make to get a toolchain were:

diff --git a/benchmark/scripts/build_script_helper.py b/benchmark/scripts/build_script_helper.py  
index 53bf7b19f68..8c9f8a5d995 100755

-   -   -   a/benchmark/scripts/build_script_helper.py  
            +++ b/benchmark/scripts/build_script_helper.py  
            @@ -54,10 +54,11 @@ def main():  
            os.makedirs(bin_dir)

swiftbuild_path = os.path.join(args.toolchain, "usr", "bin", "swift-build")

-   perform_build(args, swiftbuild_path, "debug", "Benchmark_Onone", "-Onone")

-   perform_build(args, swiftbuild_path, "release", "Benchmark_Osize", "-Osize")

-   perform_build(args, swiftbuild_path, "release", "Benchmark_O", "-O")  
    +# perform_build(args, swiftbuild_path, "debug", "Benchmark_Onone", "-Onone")  
    +# perform_build(args, swiftbuild_path, "release", "Benchmark_Osize", "-Osize")  
    +# perform_build(args, swiftbuild_path, "release", "Benchmark_O", "-O")

if **name** == "**main**":  
+ exit(0)  
main()  
diff --git a/benchmark/scripts/test_Benchmark_Driver.py b/benchmark/scripts/test_Benchmark_Driver.py  
index 7690aacf31b..b18f0d1147d 100644

-   -   -   a/benchmark/scripts/test_Benchmark_Driver.py  
            +++ b/benchmark/scripts/test_Benchmark_Driver.py  
            @@ -1071,4 +1071,5 @@ class TestBenchmarkDoctor(unittest.TestCase):

if **name** == "**main**":  
+ exit(0)  
unittest.main()  
diff --git a/cmake/modules/DarwinSDKs.cmake b/cmake/modules/DarwinSDKs.cmake  
index 98b79e1f279..73813d48ca9 100644

-   -   -   a/cmake/modules/DarwinSDKs.cmake  
            +++ b/cmake/modules/DarwinSDKs.cmake  
            @@ -14,7 +14,7 @@ set(SUPPORTED_TVOS_ARCHS "arm64")  
            set(SUPPORTED_TVOS_SIMULATOR_ARCHS "x86_64;arm64")  
            set(SUPPORTED_WATCHOS_ARCHS "armv7k;arm64_32")  
            set(SUPPORTED_WATCHOS_SIMULATOR_ARCHS "i386;x86_64;arm64")  
            -set(SUPPORTED_OSX_ARCHS "x86_64;arm64;arm64e")  
            +set(SUPPORTED_OSX_ARCHS "arm64")

is_sdk_requested(OSX swift_build_osx)  
if(swift_build_osx)  
diff --git a/stdlib/public/runtime/ImageInspectionMachO.cpp b/stdlib/public/runtime/ImageInspectionMachO.cpp  
index b21ca9d0e05..bd953673e30 100644

-   -   -   a/stdlib/public/runtime/ImageInspectionMachO.cpp  
            +++ b/stdlib/public/runtime/ImageInspectionMachO.cpp  
            @@ -172,5 +172,10 @@ int swift::lookupSymbol(const void \*address, SymbolInfo \*info) {  
            return 1;  
            }

+extern "C" bool \_\_isPlatformVersionAtLeast();  
+bool \_\_isPlatformVersionAtLeast() {  
+ return true;  
+}  
+  
\#endif // defined(**APPLE**) && defined(**MACH**) &&  
// !defined(SWIFT_RUNTIME_MACHO_NO_DYLD)

   

@edymtt
Copy link
Contributor

edymtt commented Jul 19, 2021

The lipo issue should be addressed by the following PRs

[5.5] #38445
[main] #38415

@johnno1962
Copy link
Contributor Author

johnno1962 commented Jul 24, 2021

Thanks very much Eric![]( I bought the CMake book but I never would have been able to resolve this in a million years)

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@johnno1962
Copy link
Contributor Author

johnno1962 commented May 23, 2022

A year on, I've just tried again and basically utils/build-toolchain basically no longer works (on an M1 at least). The issues I mentioned above are still there along with a few new undefined symbols related to half-floats and the bootstrapping process to bring Swift sources into the compiler doesn't seem to work, failing to find a usable version of swiftc. I don't have an intel machine to test if it "just works" on that architecture.

@shahmishal shahmishal assigned edymtt and unassigned shahmishal May 23, 2022
@shahmishal
Copy link
Member

shahmishal commented May 23, 2022

@edymtt Looks like you had a fix for this previously, I wonder if something changed recently.

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 23, 2022

edymtt's fix made a big difference but there are many other issues, at least on my machine. My clone is from a week ago.

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 26, 2022

An update on building a toolchain on M1, I've been able to get the build past the previous road blocks I mentioned by performing the whole build under rosetta!

I've not been able to get a toolchain just yet as it trips up at the very last trying to build swift-driver multi-architecture but it does seem to indicate this isn't a holding it wrong issue or specific to my particular clone. The problem is very easy to replicate if you run build-toolchain on an M1. I'm sure someone with the right llvm and CMake knowledge could clear this up in an instant.

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 27, 2022

To finish the week on a high note, I've been rewarded with a toolchain built on an M1 by running the whole thing inside Rosetta2. I had to edit the build.ninja file for libYams.a inside the swift-driver project to correct the -arch for target arm64 then build on the command line, edit that build out and retry the toolchain script. Seems like an obscure CMake problem for this configuration. Note, the "rosetta" trick only works with recent versions of CMake (I used 3.23.1 and Xcode 13.4)

@gottesmm
Copy link
Member

gottesmm commented May 31, 2022

@johnno1962 I am assuming that normal-ish builds work on M1? Are you only seeing this issue with build-toolchain?

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 31, 2022

Yes, I believe the build-script with and without --xcode works now. It's strange as the build-toolchain script is just a lightweight wrapper for calling build-script.

@gottesmm
Copy link
Member

gottesmm commented May 31, 2022

@johnno1962 I imagine that the issue here is that build-toolchain I don't think on our CI is being run on M1 machines. So I imagine that some incompatibilities have snuck in. @shahmishal we really should have a bot running build-toolchain on an Apple Silicon machine (maybe once a day) like the status bot.

@johnno1962
Copy link
Contributor Author

johnno1962 commented May 31, 2022

I couldn't agree more but the first step would be investigating the problems I've been seeing for a year and getting it working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants