[SR-280] swiftc - Can't compile project with +1500 Swift files

[swift-ci]

Previous ID	SR-280
Radar	rdar://problem/23878192
Original Reporter	tgebarowski (JIRA User)
Type	Bug
Status	Closed
Resolution	Done

Attachment: Download

Environment

OS X 10.11.2, Xcode 7.2 (7C68)
Apple Swift version 2.1.1 (swiftlang-700.1.101.15 clang-700.1.81)

Affects also latest compiled version from github.

Additional Detail from JIRA

Votes	0
Component/s	Compiler
Labels	Bug, Driver
Assignee	@belkadan
Priority	Medium

md5: 2ca795674d6ea9ba33f18cec8c203ce4

relates to:

• SR-7399 Cannot build with many/long source file paths

Issue Description:

Background

I'm working on a mixed Swift/Objective-C project with a large code base, which has more than 1500 Swift files and several hundred Objective-C files. The project is written in Swift 1.2 and is currently build successfully on Xcode 6.4. I started to migrate it to Swift 2.1 and Xcode 7.2, but it seems that swiftc has a bug which does not allow for successfuly migration/compilation. Note, that the project has to support iOS 7, so I cannot use Swift dynamic frameworks and all Swift files has to be compiled in the same target.

When trying to compile such a project, compilaion fails just after successful termination of Compile Swift sources step (with swiftc), no MergeSwitfModules step is invoked and -Swift.h umbrella header is not generated. Because there is no umbrella header the project compilation fails.

It seems that swiftc is not generating umbrella header when it has more than 1500 Swift files to process. Reducing number (to about 1100) of Swift files solves the problem, same as concatenating Swift classes into one big file.

When enabling -whole-module-optimization option in swiftc (which helped me with some problems under Xcode 6.4 and MergeSwiftModules crashes), the compiler process is returning immediatelly and is not processing any Swift files.

How to reproduce?

The bug is easy to reproduce especially with -whole-module-optimization, by following these steps:

• Run attached script (./generate_pure_swift_class.py) to generate 2500 pure swift classes (simple classes having two dummy methods)

• Try to compile all swift files:

swiftc -num-threads 1 -g -whole-module-optimization -parseable-output *.swift

• Compiler exits immediatelly without performing any steps and without any warning or error. Note that when setting -num-threads to 0 or not setting at all compilation seems to work.

Potential problem

I spent some time on investigating the issue with -whole-module-optimization and immediate compiler exit. After checking swiftc source code I found out that the culprit is posix_spawn invoked from Task::execute() (lib/Basic/Unix/TaskQueue.inc:142), it returns error code 7, which suggests [EFAULT] aka Path, argv, or envp point to an illegal address.

I checked both argv and envp and they contain correct entries, doesn't look like corrupted. I think that reason for posix_spawn() failure can be too long list of arguments stored in argvp (about 5900 entries, which is much bigger than original list of input arguments passed to swiftc - about 2000 args). Strangely this worked correctly in Swift 1.2.

Maybe posix_spawn is passed more arguments to a spawned process (since what was done for Swift 1.2), because some object dependency graph is passed together with input files to avoid unnecessary recompilation or something?

The situation could be solved by blocking posix_spawn (#undef HAVE_POSIX_SPAWN) and using fork, but this is more like a workaround and needs recompilation of swift.

Note: Both problems (not generating Swift umbrella header and immediate exit of swiftc in -whole-module-optimization mode) are also reproducible with Xcode 7.2, when creating new project and adding about 1.2 - 1.5k Swift files to it. Tested both on pure Swift files and Swift files inheriting from NSObject.

[belkadan]

I guess the right answer here would be to pass a list of files down to the frontend, instead of trying to pass them all individually, or to have a mode where we pass them via stdin. We may also want to try supporting something like GCC's @file command-line syntax, but that's (a) more complicated, and (b) potentially dangerous.

[swift-ci]

Comment by Tomasz Gebarowski (JIRA)

Hi Jordan, thanks for answering! I would probably opt for using pipe/stdin to pass parameters to a spawned process. I've noticed that you opened an internal radar for that issues, any idea if that is a serious issue for you and is going to be fixed anytime soon? It's a bit blocker for our project. Setting -num-threads 0 seems to help, but only from command line. Xcode keeps overriding this value when invoking swiftc so setting

defaults write com.apple.dt.Xcode IDEBuildOperationMaxNumberOfConcurrentCompileTasks 0

doesn't help as the value is simply ignored by Xcode. Any idea how to force Xcode to use -num-threads 0 when invoking swiftc?

[belkadan]

You might be able to add -num-threads 0 to your OTHER_SWIFT_FLAGS build setting for now. It's probably a coincidence that that helps, though.

I can't speak to schedule at this point, just that you're not the only one with such a big target (unless the other reports are also from your team). If someone else gets to it first, that's also great.

[swift-ci]

Comment by Tomasz Gebarowski (JIRA)

I'm the only person from my team who reported this problem, so I guess that more projects are affected.
I tried appending -num-threads 0 to OTHER SWIFT FLAGS, but unfortunately Xcode adds its -num-threads just after values
from OTHER SWIFT FLAGS and it gets overwritten. I solved the problem temporarily by writing a simple wrapper for swiftc in Python (which replaced the swiftc symlink in Xcode archive) that invokes swiftc under the hood and appends -num-threads 0 as last argument. This seems to work, at least the code compiles.

Unfortunately it fails at the linking stage. It seems that -whole-module-optimization is not working at all in Xcode 7.1 and Xcode 7.2.
There is already a radar for that:
rdar://23472326

Even the simplest Xcode project fails at linking stage when -whole-module-optimization is turned on. I will keep investigating this second problem, maybe there is some workaround for that as well.

Anyway thanks for help!

[swift-ci]

Comment by Tomasz Gebarowski (JIRA)

Small update, I managed to bypass this bug by modifying the way swiftc is invoked from Xcode. With this approach I managed to successfully compile my Xcode project with +1500 files in it (with -whole-module-optimization turned on)

The wrapper script I used is on my github account, more info can be found there as well:
https://github.com/tgebarowski/swiftc-wrapper

[belkadan]

It looks like you're just silently throwing away arguments. I can't recommend that.

[belkadan]

Ah, I see what it's doing. Okay, I can't call it supported, and it may break other things in Xcode, but it shouldn't be dangerous.

[swift-ci]

Comment by Tomasz Gebarowski (JIRA)

I am not dropping any arguments, but this is also not a fix, it's just a way to bypass incorrect compiler behavior. Maybe someone facing same problem will find it useful. With this workaround I managed to successfully compile,archive and run my project on both device and simulator. At least it's not blocking me and I can migrate my app to Swift 2 and wait either for (A) getting if fixed in next Xcode release or (B) dropping iOS 7 support in my app and dividing it to smaller components.

[belkadan]

I've added a -filelist parameter for input files in ad945426. I'd also like to remove the O(N) arguments from

• swiftmodule inputs to the merge-module build phase

• /output/ files for multithreading single-frontend builds (WMO)

• object file inputs to the linker, at least on OS X

[belkadan]

Finshed the rest of the work in commits up to bbc14af.

[belkadan]

...but asking Chris to review and waiting for a buildbot cycle before cherry-picking the later commits to the 2.2 branch.

[belkadan]

Test fix-up in 9cb775a.

[belkadan]

Reviewed by ChrisW and cherry-picked to the 2.2 branch!

[belkadan]

It should also be in the next development snapshot from the master branch, if you want to test sooner than Xcode 7.3.

[swift-ci]

Comment by Tomasz Gebarowski (JIRA)

Thanks Jordan for such a quick reaction. When I have time I will test the snapshot and let you know if the problem is fixed in the snapshot build.

[modocache]

Any updates here, tgebarowski (JIRA User)? If you've been able to compile your 1500+ file Swift/Objective-C project using Xcode 7.3, then I think we can close this issue.

[swift-ci]

Comment by Tomasz Gebarowski (JIRA)

Hi Brian.

The project compilation started to work with Xcode 7.3 with -Owholemodule (where the fix made by Jordan was first introduced). Compilation with -Onone was still not working, but I think that this was another issue. So yes, I think that you can close this.

Regards,
Tomasz

[modocache]

Cool, thanks! Post a link to the other issue you mentioned, if you can find it. 🙂

[swift-ci]

Comment by Vladislav Alexeev (JIRA)

I can confirm Xcode 9.2 still invokes swift and it passes all swift files through command line instead of using -filelist option. I understand there is support for -filelist in Swift, but how do people make Xcode work properly with large projects?

[belkadan]

This bug was about communication between the swiftc driver and the subprocesses it spawns, not about any tools that need to invoke swiftc themselves. SR-4517 talks about adding general response file support to the driver, but even just acceping a filelist option would be a reasonable task on its own. (This is tracked by rdar://problem/33635183 within Apple.)

[swift-ci]

Comment by Vladislav Alexeev (JIRA)

Thanks Jordan for references and explanation!

[pmusolino]

Is this problem fixed? On Xcode 9.2 I have the same problem

[belkadan]

See my comment above.

[swift-ci]

Comment by Vitaliy (JIRA)

Is this problem fixed? On Xcode 10.0 I have the same problem ((

[belkadan]

See my comment above.