[ModuleTrace] Track direct dependencies through #import and @_exported import

[varungandhi-apple]

Still missing:

• Test cases.
  • Test case(s) for SPI (maybe SPI change should be a separate PR?)
  • Test cases for overlay-overlay dependency hazards. (Current status: works on my machine)
  • Test cases for import chains. I really wish unit-testing this part of the compiler were easier lolsob. (Current status: I think it should work?)
• Some need to break up last commit into smaller commits. Seems difficult, a lot of things depend on each other.
• It's not clear that the depth-first traversal can be refactored out but maybe it's possible. This is fine. We can refactor later. There's also a bunch of special cases, don't want to block landing the fix on an elegant design.
• It might be possible to factor out some of the logic in getABIDependenciesForSwiftModule. Seems ok.

Just opening the PR to get initial feedback on whether the approach looks right.

[varungandhi-apple]

SPI PR separated out: #32771 (please ignore the first two commits here)

[beccadax]

Sorry—I wrote a bunch of review comments a couple days ago, but it looks like I forgot to post them. You’ve made changes since, so I’m not sure how many are still applicable.

[varungandhi-apple]

@swift-ci please smoke test

[shahmishal]

@swift-ci please smoke test

[varungandhi-apple]

Interesting! We found a slightly different cycle on Windows:

<unknown>:0: note: error: invariant violation: unexpected cycle in import graph!

loaded_module_trace_append (Swift) @ 0x222246b2db0

depends on Swift (Swift) @ 0x22222f02708

depends on SwiftShims (Clang) @ 0x222246b2150

depends on visualc (Clang) @ 0x222246b26a0

depends on Swift (Swift) @ 0x22222f02708

What is going on here? 🤔🤔🤔

[varungandhi-apple]

• Hamish pointed out the cycles involved in the stdlib may have to do with implicit private imports of the stdlib by Clang modules. 70f8c19
• Robert pointed out that the Darwin (Swift) -> SwiftOverlayShims (Clang) -> Darwin (Swift) dependency is likely due to SwiftOverlayShims (Clang) depending on Darwin (Clang) but us automatically using the Swift module.

Trying to think about how we want to handle this properly...

[varungandhi-apple]

Mostly just some cleanup/reorganization and more comments explaining why the different checks are necessary, removing a single one makes one or more of the tests fail. 😅

[varungandhi-apple]

Current status is DNM because there is an issue building llbuildSwift, noticed in #32980. Looking into it.

[varungandhi-apple]

So one of the issues was with iterator invalidation. I'm still not entirely sure why that was happening (I would've expected a temporary to be created which lasts the duration of the loop) but creating a separate copy of the set seems to fix the issue. From what I can tell, the set itself was not being modified, because we do check for module equality and those checks were passing.

[varungandhi-apple]

Source compat suite was passing with copying code in #32980, just re-running it with latest changes without the copy. I did check locally that llbuildSwift and SwiftPM built fine.

[varungandhi-apple]

Fixed #import -> #include to make Windows less sad.

[varungandhi-apple]

Fixed #include -> #import to make macOS less sad.

[varungandhi-apple]

@swift-ci please smoke test

[beccadax]

It looks like you may have some unfinished business at a few points. Otherwise, I think this looks okay.

[varungandhi-apple]

Going to merge this, since fixing rdar://64993153 is kinda' pressing. Will submit follow-up PRs for the TODOs.