Fix the symbol visibility in Swift compatibility lib into default instead of hidden, solve auto-linking issue and match Apple's behavior #70627

dreampiggy · 2023-12-26T09:19:59Z

Changes

TLDR: Change the Overrides.cpp 2 symbols to explicit using the default visibility, instead of implicit hidden visibility from the compiler options (-fvisibility=hidden)

This matches Apple's Xcode toolchain bundled libswiftCompatibility50.a binary.

Background

Swift's compatibility static lib (libswiftCompatibility50.a) is used for backport Swift runtime features to Swift ABI-stable OS, like iOS 12.4. The new compiler language feature may need some runtime compatibility layer to run on the lower swift runtime host.

However, we faced a issue when using the apple/swift with Xcode's bundled closed-source swift. The issues is because of the visibility for this symbol __ZN5swift34swift50override_conformsToProtocolEPKNS_14TargetMetadataINS_9InProcessEEEPKNS_24TargetProtocolDescriptorIS1_EEPFPKNS_18TargetWitnessTableIS1_EES4_S8_E

The crash is like this:

"Symbol not found: __ZN5swift34swift50override_conformsToProtocolEPKNS_14TargetMetadataINS_9InProcessEEEPKNS_24TargetProtocolDescriptorIS1_EEPFPKNS_18TargetWitnessTableIS1_EES4_S8_E",
"Referenced from: <42049861-CE9C-3353-ADD2-76C05302E30B> /Volumes/VOLUME/*/App.app/Frameworks/AppStorageCore.framework/AppStorageCore",
"Expected in:     <4A119B38-492C-3E7C-B249-E8F49F9D5B99> /Volumes/VOLUME/*/App.app/Frameworks/EEAtomic.framework/EEAtomic"

Our team is trying to solve this issue and found out the reason in the upstream.

Issues

After some investigating of the dynamic library, we surprisingly found that this missing symbol is indeed missing. Following the crash message from dyld, this two dylibs contains a dependency relation (AppStorageCore.framework will load EEAtomic):

nm EEAtomic.framework.dSYM/Contents/Resources/DWARF/EEAtomic | grep __ZN5swift34swift50override_conformsToProtocolEPKNS_14TargetMetadataINS_9InProcessEEEPKNS_24TargetProtocolDescriptorIS1_EEPFPKNS_18TargetWitnessTableIS1_EES4_S8_E
000000000000c384 t __ZN5swift34swift50override_conformsToProtocolEPKNS_14TargetMetadataINS_9InProcessEEEPKNS_24TargetProtocolDescriptorIS1_EEPFPKNS_18TargetWitnessTableIS1_EES4_S8_E

nm AppStorageCore.framework/AppStorageCore | grep __ZN5swift34swift50override_conformsToProtocolEPKNS_14TargetMetadataINS_9InProcessEEEPKNS_24TargetProtocolDescriptorIS1_EEPFPKNS_18TargetWitnessTableIS1_EES4_S8_E
                 U __ZN5swift34swift50override_conformsToProtocolEPKNS_14TargetMetadataINS_9InProcessEEEPKNS_24TargetProtocolDescriptorIS1_EEPFPKNS_18TargetWitnessTableIS1_EES4_S8_EApp

In theory, the AppStorageCore.framework use undefined symbol for that is OK. But the EEAtomic.framework must export the symbol.

Behavior differences

We start to search the toolchain codebase, and found that the symbol exists in toolchain's libswiftCompatibility50.a. The symbols introduced from stdlib/toolchain/Compatibility50/Overrides.cpp

Xcode toolchain

We use llvm-objdump -Ct to compare between the binaries of these two toolchains.

llvm-objdump -Ct /Applications/Xcode-15.0.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphoneos/libswiftCompatibility50.a

The results shows it's a external symbol

0000000000000000 g     F __TEXT,__text swift::swift50override_conformsToProtocol(swift::TargetMetadata<swift::InProcess> const*, swift::TargetProtocolDescriptor<swift::InProcess> const*, swift::TargetWitnessTable<swift::InProcess> const* (*)(swift::TargetMetadata<swift::InProcess> const*, swift::TargetProtocolDescriptor<swift::InProcess> const*))

Swift.org toolchain

However, when compared to the Xcode toolchain, the Swift.org open-sourced toolchain use external hidden symbol

0000000000000000 g     F __TEXT,__text .hidden swift::swift50override_conformsToProtocol(swift::TargetMetadata<swift::InProcess> const*, swift::TargetProtocolDescriptor<swift::InProcess> const*, swift::TargetWitnessTable<swift::InProcess> const* (*)(swift::TargetMetadata<swift::InProcess> const*, swift::TargetProtocolDescriptor<swift::InProcess> const*))

Compare

So why this use hidden visibility ?

Compiler insert `-fvisibility=hidden`

From the toolchain build log (using ./build-script -verbose-build), we can see the Overrides.cpp compile unit use the hidden visibility as defaults

So, we must explicit declare the visibility in source code. Or it will use hidden but not default visibility.

This MR just used to change for this. After changes, our new built toolchain can successfully run the same use case without dyld runtime crash. And solve the problem.

I think this issue is suitable to upstream, so I fire this PR here. Hope someone can help to review and merge-in, or provide better solution.

…visibility, not hidden This symbol is needed for auto-linking. When A.dylib depends on the B.dylib, B should export the symbol and A will linked at runtime This matches what Apple's xctoolchain shipped compatibility static library

dreampiggy · 2023-12-26T10:11:23Z

@swift-ci Please test

Kyle-Ye · 2023-12-26T10:15:38Z

@swift-ci please smoke test

dianqk · 2024-01-02T02:37:24Z

Perhaps you could use SWIFT_RUNTIME_EXPORT? Probably similar to https://github.com/apple/swift/blob/f08f86c71617bacbc61f69ce842e284b27036598/stdlib/toolchain/CompatibilityPacks/Metadata.cpp#L132.

Looking at the https://github.com/apple/swift/blob/f08f86c71617bacbc61f69ce842e284b27036598/stdlib/toolchain/Compatibility51/CMakeLists.txt, I'm curious as to why -fvisibility=hidden is being passed.

Kyle-Ye · 2024-01-02T03:56:16Z

I'm curious as to why -fvisibility=hidden is being passed.

+1 for this. Did not get any result when searching "visibility=hidden" in the project. @dreampiggy

dreampiggy · 2024-01-02T06:55:44Z

Does non-Apple (means, Linux and Windows) use this backport compatibility override ? I guess this is no need for that SWIFT_RUNTIME_EXPORT

The SWIFT_RUNTIME_EXPORT is for Swift Runtime dynamic lib (libswiftCore.dylib), not for that compatibility override .a ar archive.

I also found that Compatibility56/Overrides.cpp use the _attribute__((visibility("hidden"))), so I think it's ok to use the non-macro version ?

dreampiggy · 2024-01-02T06:56:41Z

I'm curious as to why -fvisibility=hidden is being passed.

I'm curious about this as well. Even that some compile unites in Compatibility51/56 explicitly marked _attribute__((visibility("hidden"))) in source code. (See: Compatibility56/Overrides.cpp)

It seems that this compile unit should not pass -fvisibility=hidden in the history.

dianqk · 2024-01-02T12:06:46Z

If I understand correctly, https://github.com/apple/swift/blob/1e172385e4f26e6bff8489d9f0b468f947e78146/stdlib/CMakeLists.txt#L89-L91 affects toolchain builds.

The SWIFT_RUNTIME_EXPORT is for Swift Runtime dynamic lib (libswiftCore.dylib), not for that compatibility override .a ar archive.

I'm not sure.

I also found that Compatibility56/Overrides.cpp use the _attribute__((visibility("hidden"))), so I think it's ok to use the non-macro version ?

If you can't use SWIFT_RUNTIME_EXPORT, I think you can create a new macro. This should be easier to maintain and not require duplicate comments.

Kyle-Ye · 2024-01-02T12:11:52Z

If I understand correctly,

https://github.com/apple/swift/blob/1e172385e4f26e6bff8489d9f0b468f947e78146/stdlib/CMakeLists.txt#L89-L91

affects toolchain builds.

But stdlib/toolchain/Compatibility50/CMakeLists.txt does not contain such directives.

So libswiftCompatibility50.a should not be affected IMO.

dreampiggy · 2024-01-04T10:35:25Z

Can change this attribute using the SWIFT_RUNTIME_EXPORT

But anyway, is there someone who maintains the CMake config can explain why this compile units use visibility hidden as defaults ?

gitliubo · 2024-09-26T14:10:55Z

Xcode Version 14.2 ,MacBook Pro (Retina, 15-inch, Mid 2015),how to fix it?

dreampiggy requested a review from a team as a code owner December 26, 2023 09:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix the symbol visibility in Swift compatibility lib into default instead of hidden, solve auto-linking issue and match Apple's behavior #70627

Fix the symbol visibility in Swift compatibility lib into default instead of hidden, solve auto-linking issue and match Apple's behavior #70627

Uh oh!

dreampiggy commented Dec 26, 2023 •

edited

Loading

Uh oh!

dreampiggy commented Dec 26, 2023

Uh oh!

Kyle-Ye commented Dec 26, 2023

Uh oh!

dianqk commented Jan 2, 2024

Uh oh!

Kyle-Ye commented Jan 2, 2024

Uh oh!

dreampiggy commented Jan 2, 2024

Uh oh!

dreampiggy commented Jan 2, 2024 •

edited

Loading

Uh oh!

dianqk commented Jan 2, 2024

Uh oh!

Kyle-Ye commented Jan 2, 2024

Uh oh!

dreampiggy commented Jan 4, 2024

Uh oh!

gitliubo commented Sep 26, 2024

Uh oh!

Uh oh!

Fix the symbol visibility in Swift compatibility lib into default instead of hidden, solve auto-linking issue and match Apple's behavior #70627

Are you sure you want to change the base?

Fix the symbol visibility in Swift compatibility lib into default instead of hidden, solve auto-linking issue and match Apple's behavior #70627

Uh oh!

Conversation

dreampiggy commented Dec 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Background

Issues

Behavior differences

Xcode toolchain

Swift.org toolchain

Compare

Compiler insert -fvisibility=hidden

Uh oh!

dreampiggy commented Dec 26, 2023

Uh oh!

Kyle-Ye commented Dec 26, 2023

Uh oh!

dianqk commented Jan 2, 2024

Uh oh!

Kyle-Ye commented Jan 2, 2024

Uh oh!

dreampiggy commented Jan 2, 2024

Uh oh!

dreampiggy commented Jan 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dianqk commented Jan 2, 2024

Uh oh!

Kyle-Ye commented Jan 2, 2024

Uh oh!

dreampiggy commented Jan 4, 2024

Uh oh!

gitliubo commented Sep 26, 2024

Uh oh!

Uh oh!

dreampiggy commented Dec 26, 2023 •

edited

Loading

Compiler insert `-fvisibility=hidden`

dreampiggy commented Jan 2, 2024 •

edited

Loading