Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory issue with large modules on Linux for Swift 4.2 #559

Open
Fryie opened this issue Oct 1, 2018 · 0 comments
Open

Memory issue with large modules on Linux for Swift 4.2 #559

Fryie opened this issue Oct 1, 2018 · 0 comments

Comments

@Fryie
Copy link

@Fryie Fryie commented Oct 1, 2018

We have a server-side project (using SPM) and have a script to generate documentation for it. However, when trying to migrate to Swift 4.2 (from 4.1.2), we noticed that in our docker container on CI, the documentation couldn't be generated anymore for one of the modules (the other modules in our project worked fine). Basically, what happened was that the documentation generation started becoming very slow after ca. 30-40 files and then ended up being killed. That problem does not appear on macOS.

This behaviour suggests some kind of memory issue, either a leak, or just an inefficient use of memory (maybe on large modules, some intermediate information could be stored on disk?).

To reproduce this issue I created a silly repository with a single module that has 2001 source files—all of them very simple, one Base class and 2000 children inheriting from it. Our repository has of course fewer files, but they are more complicated; I would suspect that the underlying issue is the same.

Running sourcekitten doc --spm-module Dummy on a Linux machine seems to eat up loads of memory. I tested it on a Vagrant machine and sourcekitten was using about 84% of memory while >98% of CPU time was spent swapping. At some point the process is simply killed. Running htop shows that the memory usage is completely off the charts (it takes less than 5s to grow above 500MB).

In the linked repository, the issue can be reproduced by running everything inside a container (just by using make linux42). The container seems to be able to handle a little bit more than my Vagrant VM does, but at some point it gives up nonetheless.

This doesn't happen on my mac (just test with make mac): While the generation is slow and memory also increases, the progress seems more steady instead of everything becoming progressively slower, and towards the end Activity Monitor shows that < 600MB are being used by sourcekitten, while < 200MB are being used by SourceKit, but this takes a very long time to build up, as opposed to the Linux situation.

We can also test the issue on the Swift 4.1 release (make linux41). If I try that, I find out that sourcekitten will be able to process a slightly larger amount of files before grinding to a halt (on my machine, it's about 245 files on Swift 4.2 and about 300 on Swift 4.1 until things start to go bad). This might explain why in our project, it still worked fine with Swift 4.1, but now it doesn't anymore.

Now, of course I'm not sure if I'm on the right track here and/or if this is a sourcekitten issue or a bug in SourceKit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.