Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-280] swiftc - Can't compile project with +1500 Swift files #42902

Closed
swift-ci opened this issue Dec 17, 2015 · 25 comments
Closed

[SR-280] swiftc - Can't compile project with +1500 Swift files #42902

swift-ci opened this issue Dec 17, 2015 · 25 comments

Comments

@swift-ci
Copy link
Collaborator

@swift-ci swift-ci commented Dec 17, 2015

Previous ID SR-280
Radar rdar://problem/23878192
Original Reporter tgebarowski (JIRA User)
Type Bug
Status Closed
Resolution Done

Attachment: Download

Environment

OS X 10.11.2, Xcode 7.2 (7C68)
Apple Swift version 2.1.1 (swiftlang-700.1.101.15 clang-700.1.81)

Affects also latest compiled version from github.

Additional Detail from JIRA
Votes 0
Component/s Compiler
Labels Bug, Driver
Assignee @belkadan
Priority Medium

md5: 2ca795674d6ea9ba33f18cec8c203ce4

relates to:

  • SR-7399 Cannot build with many/long source file paths

Issue Description:

Background

I'm working on a mixed Swift/Objective-C project with a large code base, which has more than 1500 Swift files and several hundred Objective-C files. The project is written in Swift 1.2 and is currently build successfully on Xcode 6.4. I started to migrate it to Swift 2.1 and Xcode 7.2, but it seems that swiftc has a bug which does not allow for successfuly migration/compilation. Note, that the project has to support iOS 7, so I cannot use Swift dynamic frameworks and all Swift files has to be compiled in the same target.

When trying to compile such a project, compilaion fails just after successful termination of Compile Swift sources step (with swiftc), no MergeSwitfModules step is invoked and -Swift.h umbrella header is not generated. Because there is no umbrella header the project compilation fails.

It seems that swiftc is not generating umbrella header when it has more than 1500 Swift files to process. Reducing number (to about 1100) of Swift files solves the problem, same as concatenating Swift classes into one big file.

When enabling -whole-module-optimization option in swiftc (which helped me with some problems under Xcode 6.4 and MergeSwiftModules crashes), the compiler process is returning immediatelly and is not processing any Swift files.

How to reproduce?

The bug is easy to reproduce especially with -whole-module-optimization, by following these steps:

  • Run attached script (./generate_pure_swift_class.py) to generate 2500 pure swift classes (simple classes having two dummy methods)

  • Try to compile all swift files:

swiftc -num-threads 1 -g -whole-module-optimization -parseable-output *.swift

  • Compiler exits immediatelly without performing any steps and without any warning or error. Note that when setting -num-threads to 0 or not setting at all compilation seems to work.

Potential problem

I spent some time on investigating the issue with -whole-module-optimization and immediate compiler exit. After checking swiftc source code I found out that the culprit is posix_spawn invoked from Task::execute() (lib/Basic/Unix/TaskQueue.inc:142), it returns error code 7, which suggests [EFAULT] aka Path, argv, or envp point to an illegal address.

I checked both argv and envp and they contain correct entries, doesn't look like corrupted. I think that reason for posix_spawn() failure can be too long list of arguments stored in argvp (about 5900 entries, which is much bigger than original list of input arguments passed to swiftc - about 2000 args). Strangely this worked correctly in Swift 1.2.

Maybe posix_spawn is passed more arguments to a spawned process (since what was done for Swift 1.2), because some object dependency graph is passed together with input files to avoid unnecessary recompilation or something?

The situation could be solved by blocking posix_spawn (#undef HAVE_POSIX_SPAWN) and using fork, but this is more like a workaround and needs recompilation of swift.

Note: Both problems (not generating Swift umbrella header and immediate exit of swiftc in -whole-module-optimization mode) are also reproducible with Xcode 7.2, when creating new project and adding about 1.2 - 1.5k Swift files to it. Tested both on pure Swift files and Swift files inheriting from NSObject.

@belkadan
Copy link
Contributor

@belkadan belkadan commented Dec 17, 2015

I guess the right answer here would be to pass a list of files down to the frontend, instead of trying to pass them all individually, or to have a mode where we pass them via stdin. We may also want to try supporting something like GCC's @file command-line syntax, but that's (a) more complicated, and (b) potentially dangerous.

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Dec 18, 2015

Comment by Tomasz Gebarowski (JIRA)

Hi Jordan, thanks for answering! I would probably opt for using pipe/stdin to pass parameters to a spawned process. I've noticed that you opened an internal radar for that issues, any idea if that is a serious issue for you and is going to be fixed anytime soon? It's a bit blocker for our project. Setting -num-threads 0 seems to help, but only from command line. Xcode keeps overriding this value when invoking swiftc so setting

defaults write com.apple.dt.Xcode IDEBuildOperationMaxNumberOfConcurrentCompileTasks 0

doesn't help as the value is simply ignored by Xcode. Any idea how to force Xcode to use -num-threads 0 when invoking swiftc?

@belkadan
Copy link
Contributor

@belkadan belkadan commented Dec 18, 2015

You might be able to add -num-threads 0 to your OTHER_SWIFT_FLAGS build setting for now. It's probably a coincidence that that helps, though.

I can't speak to schedule at this point, just that you're not the only one with such a big target (unless the other reports are also from your team). If someone else gets to it first, that's also great.

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Dec 18, 2015

Comment by Tomasz Gebarowski (JIRA)

I'm the only person from my team who reported this problem, so I guess that more projects are affected.
I tried appending -num-threads 0 to OTHER SWIFT FLAGS, but unfortunately Xcode adds its -num-threads just after values
from OTHER SWIFT FLAGS and it gets overwritten. I solved the problem temporarily by writing a simple wrapper for swiftc in Python (which replaced the swiftc symlink in Xcode archive) that invokes swiftc under the hood and appends -num-threads 0 as last argument. This seems to work, at least the code compiles.

Unfortunately it fails at the linking stage. It seems that -whole-module-optimization is not working at all in Xcode 7.1 and Xcode 7.2.
There is already a radar for that:
rdar://23472326

Even the simplest Xcode project fails at linking stage when -whole-module-optimization is turned on. I will keep investigating this second problem, maybe there is some workaround for that as well.

Anyway thanks for help!

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Jan 4, 2016

Comment by Tomasz Gebarowski (JIRA)

Small update, I managed to bypass this bug by modifying the way swiftc is invoked from Xcode. With this approach I managed to successfully compile my Xcode project with +1500 files in it (with -whole-module-optimization turned on)

The wrapper script I used is on my github account, more info can be found there as well:
https://github.com/tgebarowski/swiftc-wrapper

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 4, 2016

It looks like you're just silently throwing away arguments. I can't recommend that.

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 4, 2016

Ah, I see what it's doing. Okay, I can't call it supported, and it may break other things in Xcode, but it shouldn't be dangerous.

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Jan 5, 2016

Comment by Tomasz Gebarowski (JIRA)

I am not dropping any arguments, but this is also not a fix, it's just a way to bypass incorrect compiler behavior. Maybe someone facing same problem will find it useful. With this workaround I managed to successfully compile,archive and run my project on both device and simulator. At least it's not blocking me and I can migrate my app to Swift 2 and wait either for (A) getting if fixed in next Xcode release or (B) dropping iOS 7 support in my app and dividing it to smaller components.

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 13, 2016

I've added a -filelist parameter for input files in ad945426. I'd also like to remove the O(N) arguments from

  • swiftmodule inputs to the merge-module build phase

  • /output/ files for multithreading single-frontend builds (WMO)

  • object file inputs to the linker, at least on OS X

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 14, 2016

Finshed the rest of the work in commits up to bbc14af.

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 14, 2016

...but asking Chris to review and waiting for a buildbot cycle before cherry-picking the later commits to the 2.2 branch.

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 15, 2016

Test fix-up in 9cb775a.

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 20, 2016

Reviewed by ChrisW and cherry-picked to the 2.2 branch!

@belkadan
Copy link
Contributor

@belkadan belkadan commented Jan 20, 2016

It should also be in the next development snapshot from the master branch, if you want to test sooner than Xcode 7.3.

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Jan 20, 2016

Comment by Tomasz Gebarowski (JIRA)

Thanks Jordan for such a quick reaction. When I have time I will test the snapshot and let you know if the problem is fixed in the snapshot build.

@modocache
Copy link
Mannequin

@modocache modocache mannequin commented Aug 20, 2016

Any updates here, tgebarowski (JIRA User)? If you've been able to compile your 1500+ file Swift/Objective-C project using Xcode 7.3, then I think we can close this issue.

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Aug 20, 2016

Comment by Tomasz Gebarowski (JIRA)

Hi Brian.

The project compilation started to work with Xcode 7.3 with -Owholemodule (where the fix made by Jordan was first introduced). Compilation with -Onone was still not working, but I think that this was another issue. So yes, I think that you can close this.

Regards,
Tomasz

@modocache
Copy link
Mannequin

@modocache modocache mannequin commented Aug 20, 2016

Cool, thanks! Post a link to the other issue you mentioned, if you can find it. 🙂

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Mar 15, 2018

Comment by Vladislav Alexeev (JIRA)

I can confirm Xcode 9.2 still invokes swift and it passes all swift files through command line instead of using -filelist option. I understand there is support for -filelist in Swift, but how do people make Xcode work properly with large projects?

@belkadan
Copy link
Contributor

@belkadan belkadan commented Mar 15, 2018

This bug was about communication between the swiftc driver and the subprocesses it spawns, not about any tools that need to invoke swiftc themselves. SR-4517 talks about adding general response file support to the driver, but even just acceping a filelist option would be a reasonable task on its own. (This is tracked by rdar://problem/33635183 within Apple.)

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Mar 16, 2018

Comment by Vladislav Alexeev (JIRA)

Thanks Jordan for references and explanation!

@pmusolino
Copy link

@pmusolino pmusolino commented May 24, 2018

Is this problem fixed? On Xcode 9.2 I have the same problem

@belkadan
Copy link
Contributor

@belkadan belkadan commented May 24, 2018

See my comment above.

@swift-ci
Copy link
Collaborator Author

@swift-ci swift-ci commented Sep 18, 2018

Comment by Vitaliy (JIRA)

Is this problem fixed? On Xcode 10.0 I have the same problem ((

@belkadan
Copy link
Contributor

@belkadan belkadan commented Sep 18, 2018

See my comment above.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants