Executable fails to link when using Regex with --static-swift-stdlib (Linux)

[patrickfreed]

Describe the bug
The new Regex type cannot be used together with --static-swift-stdlib. I've confirmed this on Swift 5.7.1 and on the latest main snapshot (2022-11-03). Sometimes clean builds work, but subsequent incremental builds fail with a linker error. Similar behavior is observed when using just -Xswiftc -static-executable.

Steps To Reproduce
Steps to reproduce the behavior:

1. Create an executable that uses the Regex type in some way (e.g. try Regex("abc"))
2. Execute swift build --static-swift-stdlib
3. If step 2 didn't fail, make a change to the executable's source and issue the same build command

Expected behavior
Building and linking should succeed.

Screenshots
Here's some example output:

error: link command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld.gold: error: /home/patrick/.swiftly/toolchains/5.7.1/usr/lib/swift_static/linux/libswiftCore.a(UnicodeScalarProps.cpp.o): multiple definition of '_swift_stdlib_getScript'
/usr/bin/ld.gold: /home/patrick/.swiftly/toolchains/5.7.1/usr/lib/swift_static/linux/libswift_StringProcessing.a(UnicodeScalarProps.c.o): previous definition here
/usr/bin/ld.gold: error: /home/patrick/.swiftly/toolchains/5.7.1/usr/lib/swift_static/linux/libswiftCore.a(UnicodeScalarProps.cpp.o): multiple definition of '_swift_stdlib_getScriptExtensions'
/usr/bin/ld.gold: /home/patrick/.swiftly/toolchains/5.7.1/usr/lib/swift_static/linux/libswift_StringProcessing.a(UnicodeScalarProps.c.o): previous definition here
/home/patrick/.swiftly/toolchains/5.7.1/usr/lib/swift_static/linux/libswift_StringProcessing.a(_StringProcessing.o):_StringProcessing.o:function $s17_StringProcessing5RegexV5MatchV6outputxvg: error: undefined reference to '$s12_RegexParser16TypeConstructionO5tuple2ofypxn_tSKRzyp7ElementRtzlFZ'

clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

Environment (please fill out the following information)

• OS: Ubuntu 20.04
• Swift 5.7.1, Swift version 5.7-dev (LLVM d5f117e38620783, Swift d8b7ece)

[tomerd]

cc @MaxDesiatov @etcwilde

[ephemer]

FWIW I just found this issue after posting a longer breakdown on the forums here. In short, I can confirm this is breaking Android builds as well currently.

[etcwilde]

Sorry. Haven't had a chance to look yet. It's definitely good that the error is catching though. It's UB otherwise.
We shouldn't be linking the same .o file into both libraries. I'll have to poke around the build and see why it's liking UnicodeScalarProps.c.o into both libraries and remove it from one of them.

[ephemer]

FWIW @etcwilde I noticed the Android build (which links via ldd in my setup) mentions the symbol is originating from UnicodeScalarProps.c.o in one of the .a files but from a similarly-named .cpp file in the other .a file:

ld.lld: error: duplicate symbol: _swift_stdlib_getScript
>>> defined at UnicodeScalarProps.c
>>>            UnicodeScalarProps.c.o:(_swift_stdlib_getScript) in archive ../arm64-v8a/usr/lib/swift_static/android/libswift_StringProcessing.a
>>> defined at UnicodeScalarProps.cpp
>>>            UnicodeScalarProps.cpp.o:(.text._swift_stdlib_getScript+0x0) in archive ../arm64-v8a/usr/lib/swift_static/android/libswiftCore.a

ld.lld: error: duplicate symbol: _swift_stdlib_getScriptExtensions
>>> defined at UnicodeScalarProps.c
>>>            UnicodeScalarProps.c.o:(_swift_stdlib_getScriptExtensions) in archive ../arm64-v8a/usr/lib/swift_static/android/libswift_StringProcessing.a
>>> defined at UnicodeScalarProps.cpp
>>>            UnicodeScalarProps.cpp.o:(.text._swift_stdlib_getScriptExtensions+0x0) in archive ../arm64-v8a/usr/lib/swift_static/android/libswiftCore.a

I imagine this will be the same for other linux builds.

[etcwilde]

So, this is way worse than I thought, and is definitely UB

So the one from the stdlib is defined in UnicodeScalarProps.cpp, but it's exposes as an extern "C" function, so it will look like a C symbol.

https://github.com/apple/swift/blob/9936a60333676d1e5e4af34fd50e50163f60b418/stdlib/public/stubs/Unicode/UnicodeScalarProps.cpp#L492-L495

Meanwhile, we also have:

https://github.com/apple/swift-experimental-string-processing/blob/335a0c23aeb755fbdbc5a0a32c726f6daf35f303/Sources/_CUnicode/UnicodeScalarProps.c#L70-L87

So, yeah, there are two, different, implementations of these functions. One uses C calling conventions, the other Swift. If you did manage to get a binary out of this, I would tell you not to touch it because it's going to be buggy in hard-to-debug ways. Thanks for bringing this up.

[etcwilde]

I imagine this will be the same for other linux builds.

Yeah, this is UB everywhere. Fixing it should work, yes.

[ephemer]

@etcwilde good to know! I did get a binary out of this by using the dodgy hack described in my link above to the Swift Forums.

I then ran into another issue, which may or may not be related, namely that the static Android build (which should be similar to the linux build in most ways) fails when looking for symbols from the objc runtime, which shouldn't be linked at all on Linux. Again, not sure if that's related to the issue here – it seems unlikely, but maybe?

[etcwilde]

Wait, no, those are different. I'll have to take a closer look later.

[etcwilde]

I wouldn't trust anything that you get out though. If you have two of the same symbol, it's UB and can do anything it likes.

[ephemer]

Thanks a lot for the advice! That's good to know.

I didn't plan to trust it, but even if I wanted to, the resulting binary fails to link at runtime due to the unsatisfied link error looking for the objc runtime (objc_retainAutoreleasedReturnValue). Just an FYI 🙏🏼

[etcwilde]

This should have made it into the 5.7 branch: swiftlang/swift-experimental-string-processing#544

[ephemer]

@etcwilde that's interesting. I don't really understand how certain commits are chosen for a particular release. I am using swift-5.7-RELEASE.xctoolchain and Android SDK build made from that same tag, so looks like this fix you linked above is indeed still needed 🙏🏼 Thanks so much for looking into this!

By the way, the objc runtime issue was unrelated. For that I also needed to link -lDispatchStubs which presumably just stubs out those objc methods. So the fix here is the main thing stopping us from using static linking on Android. Exciting times! Thanks again

[etcwilde]

I don't really understand how certain commits are chosen for a particular release.

Before the release branch is cut, everything going onto main will end up in the release.
After the release branch is cut, we have to remember to cherry-pick everything from main onto the release branch.
After the release is released, only certain bug fixes get cherry-picked back to a given release for a dot-release.

[koliyo]

I have been using the swift:5.9 docker image, and after I implicitly got the updated 5.9.1, I did get this error. Downgrading to 5.9.0 on linux solved it for me, seems to be a regression.

Update: I do not get the multiple symbol, but I do get lots of undefined reference, eg:

/usr/lib/swift_static/linux/libswift_StringProcessing.a(_StringProcessing.o):_StringProcessing.o:function $s17_StringProcessing15MatchingOptionsV14Representation33_684D3EBC598AF48CD1B258716A9A78E6LLV5applyyy12_RegexParser3ASTV0C14OptionSequenceVF: error: undefined reference to '$s12_RegexParser3ASTV22MatchingOptionSequenceV6addingSayAC0dE0VGvg'
  /usr/lib/swift_static/linux/libswift_StringProcessing.a(_StringProcessing.o):_StringProcessing.o:function $s17_StringProcessing15MatchingOptionsV14Representation33_684D3EBC598AF48CD1B258716A9A78E6LLV5applyyy12_RegexParser3ASTV0C14OptionSequenceVF: error: undefined reference to '$s12_RegexParser3ASTV14MatchingOptionV4KindO15caseInsensitiveyA2GmFWC'
  /usr/lib/swift_static/linux/libswift_StringProcessing.a(_StringProcessing.o):_StringProcessing.o:function $s17_StringProcessing15MatchingOptionsV14Representation33_684D3EBC598AF48CD1B258716A9A78E6LLV5applyyy12_RegexParser3ASTV0C14OptionSequenceVF: error: undefined reference to '$s12_RegexParser3ASTV14MatchingOptionV4KindO24allowDuplicateGroupNamesyA2GmFWC'

[Azoy]

I'm going to close this as the duplicate symbol was resolved in 5.7.2. @koliyo if you are still seeing this, please file a new issue, thanks! It could be completely unrelated 🤔