Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-13605] Build script ‘swift/utils/build-parser-lib’ fails with undefined symbols #404

Open
swift-ci opened this issue Sep 25, 2020 · 12 comments
Assignees
Labels
bug

Comments

@swift-ci
Copy link
Contributor

@swift-ci swift-ci commented Sep 25, 2020

Previous ID SR-13605
Radar None
Original Reporter TikiMcFee (JIRA User)
Type Bug
Status Reopened
Resolution

Attachment: Download

Environment
  • macOS 10.15.6 (19G2021)

  • XCode Version 12.0 (12A7209)

  • Toolchain: Swift Development Snapshot 2020-09-23 (a)

  • /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2020-09-23-a.xctoolchain
Additional Detail from JIRA
Votes 0
Component/s SwiftSyntax
Labels Bug
Assignee @akyrtzi
Priority Medium

md5: 2b3cfd0512fa4210543251067beeaf09

Issue Description:

Hey there folks. I greatly appreciate your time. If there's anything at all I can supply to help solve this - or to help produce a better bug/support ticket - please let me know!

The context

I am attempting to build a custom version of `lib_InternalSwiftSyntaxParser.dylib` to embed in iOS as described in the repositories below:

https://github.com/apple/swift-syntax
https://github.com/apple/swift

My configuration is:

# swift-DEVELOPMENT-SNAPSHOT-2020-09-23-a installed at /Library/Developer/Toolchains
$ echo $TOOLCHAINS
swift

$ xcrun --find swift
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2020-09-23-a.xctoolchain/usr/bin/swift

$ xcrun --toolchain default --find swift
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift

$ xcrun --toolchain swift --find swift
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2020-09-23-a.xctoolchain/usr/bin/swift

# showing that --toolchain with bad input goes back to default
$ xcrun --toolchain random_string_that_shouldnt_do_anything --find swift
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift

So far, I have successfully built the main swift repository with

$ ./swift/utils/build-script --clean

... and I am using the built `.dylib` from that effectively in a macOS application.

The crash

$ ./swift/utils/build-parser-lib --release --no-assertions --build-dir /tmp/parser-lib-build-ios --host iphoneos --architectures arm64

[ ... many many lines of build information later ...]

[1/2] Linking CXX shared library lib/lib_InternalSwiftSyntaxParser.dylib
FAILED: lib/lib_InternalSwiftSyntaxParser.dylib
: && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -arch arm64 -miphoneos-version-min=10.0 -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -Wno-nested-anon-types -Werror=switch -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.0.sdk -dynamiclib -Wl,-headerpad_max_install_names -Wl,-exported_symbols_list,"/tmp/parser-lib-build-ios/arm64/obj/tools/swift/tools/libSwiftSyntaxParser/libSwiftSyntaxParser.exports" -Wl,-dead_strip -target arm64-apple-ios10.0 -Xlinker -dead_strip -Xlinker -compatibility_version -Xlinker 1 -o lib/lib_InternalSwiftSyntaxParser.dylib -install_name @rpath/lib_InternalSwiftSyntaxParser.dylib tools/swift/tools/libSwiftSyntaxParser/CMakeFiles/libSwiftSyntaxParser.dir/c-include-check.c.o tools/swift/tools/libSwiftSyntaxParser/CMakeFiles/libSwiftSyntaxParser.dir/libSwiftSyntaxParser.cpp.o lib/libswiftParse.a lib/libswiftSyntaxParse.a lib/libswiftParse.a lib/libswiftSyntaxParse.a lib/libswiftAST.a lib/libswiftBasic.a lib/libLLVMSupport.a -lm lib/libLLVMDemangle.a lib/libswiftDemangling.a ./lib/libclangBasic.a lib/libswiftSyntax.a && :
Undefined symbols for architecture arm64:
 "clang::MacroInfo::dump() const", referenced from:
 swift::ClangNode::dump() const in libswiftAST.a(Decl.cpp.o)
 "clang::Type::dump() const", referenced from:
 swift::UnexpectedClangTypeError::dump() in libswiftAST.a(ExtInfo.cpp.o)
 "clang::Decl::dump() const", referenced from:
 swift::ClangNode::dump() const in libswiftAST.a(Decl.cpp.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
ERROR: command terminated with a non-zero exit status 1, aborting

(You can find the full, gorey, exciting log at https://gist.github.com/tikimcfee/75c24cdce8cc352645da78da4b015daf)

There are a couple of things I noticed in the build log that have me questioning some kind of cache or local configuration. Specifically, lines that mention build tools from a different toolchain:

-- Found libtool - /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool
[...]
-- CMake (/usr/local/Cellar/cmake/3.18.2/bin/cmake) Version: 3.18.2
-- CMake Make Program (/usr/local/bin/ninja) Version: 1.10.1
-- C Compiler (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc) Version: 12.0.0.12000032
-- C++ Compiler (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++) Version: 12.0.0.12000032

I'm guessing I'm doing something wrong with the invoking of this script, but I've also tried a number of path hacks within the actual build script to try to point to a different toolchain or compiler - all of which are things I am completely unfamiliar with and am stabbing at in the dark with print debugging.

@typesanitizer
Copy link

@typesanitizer typesanitizer commented Sep 25, 2020

@akyrtzi
Copy link
Member

@akyrtzi akyrtzi commented Sep 25, 2020

Thanks for reporting this!

The toolchain is not relevant for this, the issue needs to be fixed by changes in the swift repo. The parser library build aims to avoid linking in the libclangAST.a library, since that is not necessary for its purpose, but occasionally symbol references leak out and trigger linker errors.

On another note, given that you want to use the parser library for iOS, you should know that the app store does not accept apps with embedded `.dylibs`, so an open task that we need to make it applicable for iOS apps is to build it as a `.framework`

@swift-ci
Copy link
Contributor Author

@swift-ci swift-ci commented Sep 25, 2020

Comment by Ivan Lugo (JIRA)

Hey there @akyrtzi, thanks much for the info! I grok the issue here - seems like I'll need to continue testing with the current macOS implementation if I want to use this directly. I completely forgot about the .dylib usage limitation on iOS, and I again appreciate your word on it. Definitely puts a bit of a damper on the mobile use case there.

I mentioned my unfamiliarity with the build process, but I'd be happy to assist in either experimenting with or testing out symbol fixes and framework builds. If it's something you think I'd be able to pick up quickly enough to be dangerous and throw up a PR, I'll happily give it some more cycles.

Furthering the tangent, I've been looking around at other solutions for Swift source parsing and analysis, and found open repositories like SourceKitten that use other (what I guess can be called) analysis backends. My goal is using the above would the same: knowing fairly high level information about a source file - declarations and identifiers, file positioning with trivia (for manual rendering of the file), and 'structures' like beginnings and ends of code components (classes, structs, etc). Do you have any thoughts on the tool above with respects to its ability to provide useful source information, remaining compatible with the language, or using it across devices?

EDIT: Did a bit more reading and realized SourceKitten also relies on SourceKit and other mac-only libraries. Drat![]( Now I'm on the look out for something iOS compatible at all)

@akyrtzi
Copy link
Member

@akyrtzi akyrtzi commented Sep 25, 2020

> I mentioned my unfamiliarity with the build process, but I'd be happy to assist in either experimenting with or testing out symbol fixes and framework builds. If it's something you think I'd be able to pick up quickly enough to be dangerous and throw up a PR, I'll happily give it some more cycles.

I can look into the linker issue sometime next week. I also have a TODO task to get it built as a framework.

> Did a bit more reading and realized SourceKitten also relies on SourceKit and other mac-only libraries.

SwiftSyntax is way much easier to embed but it only gives you syntactic information. With SourceKit you'd have the full power of the Swift compiler but it would involve essentially building a swift toolchain for use from iOS. I'm not sure if there is a convenient way to do that or not, if this is a direction you want to explore I'd suggest asking in the forums for some direction.

@swift-ci
Copy link
Contributor Author

@swift-ci swift-ci commented Sep 25, 2020

Comment by Ivan Lugo (JIRA)

Lots to learn here. I'll watch for linker news in the mean time. The iOS toolchain rings some bells across various blogs and articles I've seen, so I'll definitely look into it. I've done some cross positing with this as I originally though it was something I was doing wrong, so I'll pivot on those topics.

For now, I'll shift toward making the mobile app a bit more tethered. I've got an idea to throw some of the results from the mac to the mobile device via something like the multipeer kit, and if it can handle the heavy lifting for the time being, it'd be a neat showcase anyway of having devices work together.

Cheers again for all your info and help.

@akyrtzi
Copy link
Member

@akyrtzi akyrtzi commented Oct 15, 2020

Linker errors are fixed via apple/swift#34187

@swift-ci
Copy link
Contributor Author

@swift-ci swift-ci commented Oct 16, 2020

Comment by Ivan Lugo (JIRA)

Fantastic. I just pulled everything down and within a few minutes I had a fresh iOS-capable library to embed. This simplifies a ton of my current project work.

If I could inquire about one more thing: I've disabled bitcode support for my project target, and it seems to build just fine without it. Is it possible to build these libraries with support for this? I'm done some more grepping about to find bitcode comments in the build scripts, but again I'm a bit out of my depth.

Cheers and many, many thanks for your fix. I'm looking forward to running this tomorrow!

@akyrtzi
Copy link
Member

@akyrtzi akyrtzi commented Oct 16, 2020

It's a bit unclear to me, are you asking about building the library with bitcode? Not sure about that, there could be some CMake variable to set.

@swift-ci
Copy link
Contributor Author

@swift-ci swift-ci commented Oct 22, 2020

Comment by Ivan Lugo (JIRA)

Yep! (sorry for the late response) I did take a bit of dive through the cmake flags and flipped a few switches without much success. That's totally out of scope, though, and I don't have any real reason to need it so far, but I figured knowing ahead of time would be good.

@swift-ci
Copy link
Contributor Author

@swift-ci swift-ci commented Nov 10, 2020

Comment by Ivan Lugo (JIRA)

Hey again!

I've come back to say that everything was working great for the last month or so, but I seem to have hit the same problem again. It looks like sometime in the last few weeks, more of those symbols became visible to the build script. Taking your lead, I attempted to short-circuit the methods that were found to be problematic and I was able to correctly build the dynamic library as before. I've attached a git patch here to show the changes I made (if this is more appropriate as a pull request, please let me know!)

diff.patch

So there's that, and now I've hit something brand new:

ld: unknown option: -no_adhoc_codesign

The above error occurs when building with the latest trunk toolchain, Development Snapshot 2020-11-08 (a). Some context for it is

-L/usr/lib/swift -Xlinker -add_ast_path -Xlinker /Users/ivan.lugo/Library/Developer/Xcode/DerivedData/LookAtThat_AppKit-ebjswrdgtfgsvwgyyusmcptsrhya/Build/Intermediates.noindex/LookAtThat_AppKit.build/Debug/LookAtThat_AppKit.build/Objects-normal/x86_64/LookAtThat_AppKit.swiftmodule -l_InternalSwiftSyntaxParser -Xlinker -no_adhoc_codesign -Xlinker -dependency_info -Xlinker /Users/ivan.lugo/Library/Developer/Xcode/DerivedData/LookAtThat_AppKit-ebjswrdgtfgsvwgyyusmcptsrhya/Build/Intermediates.noindex/LookAtThat_AppKit.build/Debug/LookAtThat_AppKit.build/Objects-normal/x86_64/LookAtThat_AppKit_dependency_info.dat

My best guess for the above, using https://keith.github.io/xcode-man-pages/ld.1.html#no_adhoc_codesign as a reference, leads me to believe some breaking changes were added to support the new Apple Silicon paradigm. Could be wrong though.

Any advice for this new set of interesting problems?

@swift-ci
Copy link
Contributor Author

@swift-ci swift-ci commented Dec 4, 2020

Comment by Ivan Lugo (JIRA)

Hey again![]( Quick ping to you @akyrtzi, seeing if this dropped in the backlog, or if there are some easy workarounds. Thanks as always)

@akyrtzi
Copy link
Member

@akyrtzi akyrtzi commented Dec 5, 2020

Thanks for looking into it, I fixed the linker errors here with only the minimum necessary changes: apple/swift#34976

> ld: unknown option: -no_adhoc_codesign

I'm not sure where this flag is coming from, it doesn't look like it's coming from the Swift compiler; are you sure it's not coming from project changes?

Also BTW, why are you "building with the latest trunk toolchain"? The parser library+SwiftSyntax are independent of a particular toolchain, you can use them while building your project with the toolchain from latest Xcode.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug
Projects
None yet
Development

No branches or pull requests

4 participants