Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-12933] lldb: Couldn't IRGen expression; with -no-serialize-debugging-options #4464

Closed
swift-ci opened this issue Jun 4, 2020 · 20 comments
Closed
Assignees
Labels
bug LLDB for Swift

Comments

@swift-ci
Copy link

@swift-ci swift-ci commented Jun 4, 2020

Previous ID SR-12933
Radar rdar://problem/63760482
Original Reporter steipete (JIRA User)
Type Bug
Status Resolved
Resolution Done
Environment

Version 11.5 (11E608c)

macOS 10.15.5 (19F96)

Swift trunk toolchain May 26, 2020

Additional Detail from JIRA
Votes 4
Component/s LLDB for Swift
Labels Bug
Assignee @adrian-prantl
Priority Medium

md5: f093e1b28bc880b1bc4508f349570aa1

Issue Description:

We get Couldn't IRGen expression, no additional error despite setting -no-serialize-debugging-options via SR-12783.
This happens when we provide a binary SDK with dSYM files.

It seems that the header import paths are stored in the dSYM.
The folders are mentioned in the lldb log. If the dSYM files are deleted, then debugging works (and our build folder is not listed in the logs)

PSPDFKit 9.4pre, no-serialize, dSYMs available, master toolchain (May 26):
https://gist.github.com/steipete/515646a9840c91a61b73d1ab9f255bb3

Full debugging flow:
https://steipete.com/posts/couldnt-irgen-expression/

Example to reproduce:

https://www.dropbox.com/s/pofzs6ly5syd86t/PSPDFKit-9.4pre-no-serialize.dmg?dl=1 (large file, can't upload here)

  1. Open PSPDFKit 9.4.0-dev for iOS/Examples/SwiftExample.xcodeproj

  2. Set breakpoint to AppDelegate.swift, window.makeKeyAndVisible()

  3. Run. Observe IRGen error

  4. Delete PSPDFKit-dSYMs and PSPDFKitUI-dSYMs

  5. Delete derived data

  6. Run. Observe <UIWindow... is printed

Note: It DOES WORK if you unpack in \tmp - seems like lldb can't find the dSYMs then? (Always delete derived data when moving)

it seems that this is fixed in master trunk:

  1. Restore. (delete derived data, restore dSYM)

  2. Install Swift Master Trunk and enable

  3. Set iOS Deployment Target to 13 (SR-12932)

  4. Run. Observe <UIWindow... is printed

Questions:

  • Should it work like that? Is the dSYM bound to the machine it was built?

  • I cannot find any binary size differences with -no-serialize-debugging-options; what does this change?

  • How does lldb find dSYM files? Same name+folder as the framework? Why does this fail in /tmp?

I reported thus bug as FB7718242 on Feedback Assistant as well, but with much less understanding what's going on.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 4, 2020

Based on the description, you are running intro the issue discussed in https://bugs.swift.org/browse/SR-12783. The short answer is, if you are planning to ship a dSYM to another machine, you need to make sure to compiler with -no-serialize-debugging-options. This will avoid hardcoding any search paths into the serialized Swift modules that are stored in the .dSYM bundle.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 4, 2020

Looks like I hit reply too early. You already tried that.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 4, 2020

I just double-checked the compiler sources and -no-serialize-debugging-options is the right option:

It gets lowered to SerializeOptionsForDebugging here:
https://github.com/apple/swift/blob/4e0fad869c440489053753d059c57a4ea9dd66e6/lib/Frontend/ArgsToFrontendOptionsConverter.cpp#L175

and that option controls whether the Clang search paths are serialized in the .swiftmodule or not here:

https://github.com/apple/swift/blob/4e0fad869c440489053753d059c57a4ea9dd66e6/lib/Serialization/Serialization.cpp#L911

Now evidently this isn't working as expected, so let's figure out why!

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Jun 4, 2020

Comment by Peter Steinberger (JIRA)

Thanks for looking into this, Adrian! I did verify that my xcconfig settings work, if I add a typo to the flag, I get compile errors. Can I send you anything else to help?

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 4, 2020

> Should it work like that? Is the dSYM bound to the machine it was built?

With -no-serialize-debugging-options it is supposed to work, however, since no Clang search paths are serialized, you may need to tell LLDB where to find the Clang header dependencies on the other machine manually. This can be done by setting the target.swift-extra-clang-flags option. Starting with Swift 5.2, LLDB is generally capable of also importing the Clang dependencies straight from the debug info, so even if you don't do that, debugging should still work. Because this is currently only a fallback mechanism, you will still see clang compiler errors about the missing headers, but debugging should still work.

The goal for LLDB is to make this code path the default rather than a fallback, which will allow us to make -no-serialize-debugging-options the default. The Swift 5.3 branch is still missing a few features (most notably apinotes support) that are necessary to complete this transition.

> I cannot find any binary size differences with -no-serialize-debugging-options; what does this change?

I'll investigate. You are supposed to see a difference in the binary .swiftmodule file, and by extension in the .dSYM (which contains a copy of all Swift modules).

> How does lldb find dSYM files? Same name+folder as the framework? Why does this fail in /tmp?

LLDB looks

  1. next to the executable/dylib
  2. through any custom .dSYM location mechanism ( https://lldb.llvm.org/use/symbols.html )
  3. through Spotlight, using the UUID of the executable/dylib. Note that Spotlight doesn't index /tmp.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 4, 2020

Thank you so much for filing this report! I was able to identify a bug in dsymutil that would misdirect LLDB to the .swiftmodule files on disk (the ones on the machine the .dSYM was built with), and disregard the ones included in the .dSYM itself. That could explain part of what we're seeing.

> Can I send you anything else to help?

Yes. I was unable to reproduce your report that -no-serialize-debugging-options has no effect. I did the following experiment:

$ touch /tmp/t.swift && swiftc -g /tmp/t.swift -o /tmp/a.out
$ touch /tmp/t.swift && swiftc -Xfrontend -no-serialize-debugging-options -g /tmp/t.swift -o /tmp/b.out
$ strings /tmp/a.out.dSYM/Contents/Resources/DWARF/a.out > /tmp/1
$ strings /tmp/b.out.dSYM/Contents/Resources/DWARF/b.out > /tmp/2
$ opendiff /tmp/1 /tmp/2

I can see that the Clang option block (for example -working-directory) is missing in b.out. So I wonder if your setup is somehow different from my experiment, or if perhaps you were diffing the object files instead of the .swiftmodules or .dSYMs when you reached the conclusion that the option doesn't do anything.

There is also a second thing I would curious about: In response to https://bugs.swift.org/browse/SR-12783, I changed LLDB to surface the underlying Clang errors instead of just saying Couldn't IRGen expression, no additional error. I understand that this may be a lot to ask, but if you reproduce the issue with a recent downloadable toolchain — can you see the underlying error, and what is it?

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Jun 5, 2020

Comment by Franz Busch (JIRA)

@adrian-prantl I was just trying to debug this in our application as well since we are running into this problem constantly. We use Carthage and share the artefacts with ROME. That means one dev is building all the artefacts and then uploading them to an S3 bucket where everybody pulls them from. As far as I understood from the above we should be using -no-serialize-debugging-options to make this work.

What I tried now was to just build all deps locally again and then try to debug but I am still running into this IRGen issue. So I pulled the latest toolchain from June 4th and tried to debug again. Now I get a more expressive error but don't know how to properly interpret this:

warning: Swift error in fallback scratch context: error: module 'Swift' was created for incompatible target x86_64-apple-ios13.0-simulator: /Users/franzbusch/Library/Developer/Xcode/DerivedData/ModuleCache.noindex/Swift-2XSJ1TSBVFM44.swiftmodule

note: This error message is displayed only once. If the error displayed above is due to conflicting search paths to Clang modules in different images of the debugged executable, this can slow down debugging of Swift code significantly, since a fresh Swift context has to be created every time a conflict is encountered.

Can't construct shared Swift state for this process after repeated attempts.
Giving up.  Fatal errors:
error: module 'Swift' was created for incompatible target x86_64-apple-ios13.0-simulator: /Users/franzbusch/Library/Developer/Xcode/DerivedData/ModuleCache.noindex/Swift-2XSJ1TSBVFM44.swiftmodule

One other interesting thing is that this is happening a lot if our static libs and their associated test bundle but in our main target we are able to po properly. However, we have to set a breakpoint in didFinishLaunching that does po application.
Is there anything more we can provide you around this IRGen issue to help you debug it, we have been stuck with this for months now.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 5, 2020

Peter, I'm sorry. I just realized that you had the answer to my second question already in your fantastic blog post:

warning: Swift error in fallback scratch context: error: module 'Swift' was created for incompatible target x86_64-apple-ios13.0-simulator: /Users/steipete/Library/Developer/Xcode/DerivedData/ModuleCache.noindex/Swift-3K8REJ00QGV2U.swiftmodule

That is very valuable information. This means that this is not failing because of Clang search path options, which explains, why -no-serialize-debugging-options does nothing to fix the problem for you!

The error message itself points at something else going wrong:

  1. The Swift stdlib module is not supposed to be built from a .swiftinterface (in which case it would end up in the module cache) but it's supposed to get loaded from Xcode's prebuilt module cache.

  2. I also think the error diagnostic is likely wrong, since I'm assuming that x86_64-apple-ios13.0-simulator is actually the correct triple for your target, right?

I'll see if I can reproduce this.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 5, 2020

The dsymutil issue I mentioned is fixed here: https://reviews.llvm.org/D81205

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 5, 2020

One more thing I noticed in https://gist.github.com/steipete/515646a9840c91a61b73d1ab9f255bb3:

SwiftASTContext("PSPDFKit")::DeserializeAllCompilerFlags() – Found 1 AST file data entries.
Unable to load Swift AST for module "PSPDFKit" from library "/Users/steipete/Library/Developer/Xcode/DerivedData/SwiftExample-gkpyxmqpselkitgjpzrvuzwcnruj/Build/Products/Debug-iphonesimulator/SwiftExample.app/Frameworks/PSPDFKit.framework/PSPDFKit".
The module file format is too old to be used by this version of the debugger.

At the moment, you can not yet load a dSYM that contains a .swiftmodule produced by an older Swift compiler into an LLDB that was linked against a newer Swift compiler. LLDB may be able to find some of the types used in the public interface of PSPDFKit.framework in other dSYMs, and in the PSDPDFKit .swiftinterface but it will not be able to peek at the internals of the library, since most of the Swift types will be missing. That's just not supported (but being worked on).

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 11, 2020

There is one more thing that could play into this: Does the problem also reproduce when you bump the deployment target to iOS 13?

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Jun 12, 2020

Comment by Peter Steinberger (JIRA)

Adrian: Thanks so much for debugging this and for fixing the issue in dsymutil!

Apologies for the late reply. I've been running into a bunch of issues (https://twitter.com/steipete/status/1270756342295334914) trying to build our project with the Swift Trunk toolchain, but Slava Pestov [mentioned|https://twitter.com/slava_pestov/status/1271150466404155399] that a fix for Catalyst will land in a few days (most other issues I was able to work around or modify build settings)

Thanks also for explaining how lldb performs the dSYM lookup - this explains the behavior I've seen and I updated my blog post with it. In there I did a bin diff of the binaries and not the dSYM, which explains why I didn't find any differences in size with/without the flag - I looked at the wrong spot.

I'll wait for trunk to be fixed and then build a new version to test. Curiously we dropped iOS 11 with our upcoming 9.4 release and also enabled library evolution, and with the draft I can't reproduce the lldb issue anymore. Given how much trouble this made to reliably reproduce, I need to try this in a fresh VM again - will do both after our release next week.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jun 12, 2020

That's great, by then I may actually have found the time to fix the issue I'm suspecting regarding the deployment target. I'll ping the Jira ticket once that fix has landed.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jul 9, 2020

The deployment target issues are fixed (see SR-12932). Does this still reproduce with the bugfixes from SR-12932?

(Unfortunately Xcode 12 beta 2 shipped before the last fix went in. You may see a "cannot load Swift stdlib" error there).

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Jul 20, 2020

Comment by keith siilats (JIRA)

I ran into this when trying to attach the Firebase binaries with carthage in Xcode 11.5 and 11.6. I think its only the performance one that does this. I tried the lldb log file but don't see anything obvious there. I also read your blog post but I dont see and Firebase related dsym files in the Carthage Build IOS folder, although there are some facebook ones that have older dates. Could anyone try to replicate with Cartfile containing:

binary "https://dl.google.com/dl/firebase/ios/carthage/FirebaseAnalyticsBinary.json"

binary "https://dl.google.com/dl/firebase/ios/carthage/FirebaseCrashlyticsBinary.json"

#binary "https://dl.google.com/dl/firebase/ios/carthage/FirebasePerformanceBinary.json"

binary "https://dl.google.com/dl/firebase/ios/carthage/FirebaseProtobufBinary.json"

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Jul 26, 2020

Comment by Peter Steinberger (JIRA)

We're still working on getting everything built with Xcode 12b3; there's another lldb bug where it doesn’t work when the iOS deployment target is set to anything below iOS 13. FB8128033, also reported on https://developer.apple.com/forums/thread/651375.

There's also a crasher in Apple's security helper on the DTK that currently breaks our build script (FB8127506), but that's something we can work around.

I hope that by b4 FB8128033 is resolved and I can report back if this fix worked.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Jul 27, 2020

Unfortunately there is very little information in FB8128033 that would help diagnose what the root cause is. Based on the symptoms, it could be the bug that was fixed in #1548 (not yet in beta 3) but it's difficult to tell with neither a log nor a reproducer. [It is that bug if the types log under SwiftASTContextForExpressions::LogConfiguration shows an iOS target triple, where is should be using an ios-simulator target triple]. I understand that sharing a reproducer can be difficult. Perhaps you can try with a downloadable 5.3 toolchain https://swift.org/download/#snapshots from Jul 27+? Otherwise I need more information to work with.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Aug 10, 2020

All the issues I'm aware of in this area should be resolved in beta 4.

@swift-ci
Copy link
Author

@swift-ci swift-ci commented Aug 20, 2020

Comment by Peter Steinberger (JIRA)

Many thanks for looking into this issue!

I can no longer reproduce the issues here as of Xcode 12b4 - debugging works as expected, no matter if the dSYM is available or not.

@adrian-prantl
Copy link
Member

@adrian-prantl adrian-prantl commented Aug 20, 2020

I'm happy to hear that we finally got to the bottom of this. Thanks a lot for you patience and the excellent bug reports!

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 7, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug LLDB for Swift
Projects
None yet
Development

No branches or pull requests

2 participants