swift-syntax compiles much more slowly with WMO enabled

[allevato]

Description

This is a pretty open-ended issue, but I wanted to file some observations to see if folks have insight about possible improvements we can make here.

I observed recently that WMO makes swift-syntax compile an order of magnitude (!) slower than without WMO. From some recent testing (iMac Pro 2017, 18-core 2.3 GHz Core Intel Xeon W, 128GB RAM):

• Not optimized (SPM debug build, swift build -v --target SwiftSyntax): 17.34 sec
• Optimized but no WMO (SPM debug build + -O, swift build -v --target SwiftSyntax -Xswiftc -O): 37.85 sec
• Optimized and WMO (SPM release build, swift build -c release -v --target SwiftSyntax): 263.27 sec

Granted, SwiftSyntax is a big module with lots of generated code, so it's going to take some time to compile, but the WMO situation looks a bit grim. A big part of the problem could just be forcing so much complex code into a single frontend invocation instead of parallelizing it across multiple cores, which is unavoidable with WMO. I did a build with -stats-output-dir / -profile-stats-events / -trace-stats-events (the logging made the build take >900 sec) and now have a 2GB CSV file and some flamegraphs which I'll try to analyze...

I was wondering if this could cause problems with macros in the future; once they start using standalone executables with SwiftSyntax as a package dependency, folks with a strong focus on build performance will likely want their macro binaries compiled with optimizations, and SwiftPM's default for release mode is to enable WMO. Could the current performance be a major bottleneck if it's not addressed?

But even excluding that use case, I wonder if there is any tuning that's possible to close this gap?

Steps to Reproduce

No response

[bnbarham]

rdar://103933042

[artemcm]

@allevato, there is -num-threads from WMO which will parallelize the compiler backend across the number of available cores, but AFAICT SPM is already passing that in:
https://github.com/apple/swift-package-manager/blob/main/Sources/Build/BuildPlan.swift#L990

I can't speak to what this will mean to macros, but otherwise, as you pointed out, this is largely a circumstance of a single (largely single-threaded) swift-frontend invocation having to handle a whole lot of code. In contrast, non-WMO code-path will use all 18 cores of your iMac Pro so it, unfortunately, not too surprising that you get an order-of-magnitude difference in build time.

[allevato]

Yeah, my build log shows -num-threads 36 when I do a standard release build.

I attempted to de-parallelize a debug build as much as possible to do a comparison, and this command line swift build -v --target SwiftSyntax -Xswiftc -j1 -Xswiftc -disable-batch-mode resulted in a build time of 87.39s. Not great, but still quite a bit different lower than the WMO build, so it seems like WMO in particular is still resulting in a lot more work (or the distribution of work during WMO vs. multiple single frontend invocations is different enough to produce the slower timings).

Given SwiftSyntax's size, I don't think there's an obvious right answer here and users may just need to rely on their intermediates being cached (most users won't be modifying SwiftSyntax even if they're developing their own macros), but it could still be a pain point any time someone cleans their build products folder. I suppose I mainly intended this issue to be a discussion area for any small wins that could be eked out to reduce the overall burden.

[moshe-foreflight]

@allevato How did you generate the flame graph?

Edit: 🤦‍♂️ I need to read.

[moshe-foreflight]

@bnbarham would you consider offering a binary package release along with the source code, to speed up compilation for projects that depend on SwiftSyntax?