New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A lot of memory is required to compile Clippy #1062

Open
shepmaster opened this Issue Jun 30, 2016 · 16 comments

Comments

Projects
None yet
7 participants
@shepmaster
Copy link
Member

shepmaster commented Jun 30, 2016

I'm attempting to add Clippy support to my implementation of the Rust Playground, but I'm unable to compile Clippy on the free tier of an Amazon EC2 instance (a little less than 1 GiB of RAM).

The last value from ps that I captured before the process was killed:

root      8027 87.0 63.5 956676 647664 ?       Sl   01:39   0:13 /root/.multirust/toolchains/nightly-2016-06-24-x86_64-unknown-linux-gnu/bin/rustc root/.cargo/registry/src/github.com-1ecc6299db9ec823/clippy_lints-0.0.77/src/lib.rs --crate-name clippy_lints --crate-type lib -C opt-level=3 -C metadata=44756caa45bd5b13 -C extra-filename=-44756caa45bd5b13 --out-dir /tmp/cargo-install.Np8ev9N0Fp6c/release/deps --emit=dep-info,link -L dependency=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps -L dependency=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps --extern regex_syntax=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps/libregex_syntax-6ded9701a0454251.rlib --extern unicode_normalization=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps/libunicode_normalization-5de3a9c8fd8ddf4e.rlib --extern matches=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps/libmatches-5d5580ffd528031c.rlib --extern quine_mc_cluskey=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps/libquine_mc_cluskey-bfa4946e33f51135.rlib --extern rustc_serialize=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps/librustc_serialize-3561541d79c18212.rlib --extern toml=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps/libtoml-1a75b37a708f335b.rlib --extern semver=/tmp/cargo-install.Np8ev9N0Fp6c/release/deps/libsemver-60d1aa0e68346373.rlib --cap-lints allow

Any suggestions on how reasonable it is to require 600+ MiB of RAM to compile that crate? Any ideas how to work around it?

@Manishearth

This comment has been minimized.

Copy link
Member

Manishearth commented Jun 30, 2016

cargo-clippy takes up a lot of memory since it links to rustc as a binary. clippy_lints shouldn't be taking this much memory. Could you run this with time-passes and paste the output? cargo rustc -p clippy_lints -- -Ztime-passes

@shepmaster

This comment has been minimized.

Copy link
Member

shepmaster commented Jun 30, 2016

I apologize, I missed a key piece of information in my original comment. I'm doing this via cargo install clippy, so I don't have a Cargo.toml lying around. Would you like me to run that command in a checkout of this repository?

@Manishearth

This comment has been minimized.

Copy link
Member

Manishearth commented Jun 30, 2016

Yes please :)

Also, cargo install clippy will take a lot of memory since it links with rustc. I'm not sure if this can be mitigated, cc @oli-obk

However, your memory hog is within clippy_lints itself, which should not be affected by this.

@shepmaster

This comment has been minimized.

Copy link
Member

shepmaster commented Jun 30, 2016

   Compiling clippy_lints v0.0.77 (file:///tmp/x/rust-clippy/clippy_lints)
time: 0.108; rss: 67MB  parsing
time: 0.019; rss: 68MB  configuration
time: 0.000; rss: 68MB  recursion limit
time: 0.000; rss: 68MB  crate injection
time: 0.000; rss: 68MB  plugin loading
time: 0.000; rss: 68MB  plugin registration
time: 0.818; rss: 163MB expansion
time: 0.000; rss: 163MB maybe building test harness
time: 0.000; rss: 163MB checking for inline asm in case the target doesn't support it
time: 0.006; rss: 163MB complete gated feature checking
time: 0.018; rss: 163MB assigning node ids
time: 0.009; rss: 164MB collecting defs
time: 0.019; rss: 173MB external crate/lib resolution
time: 0.015; rss: 173MB early lint checks
time: 0.003; rss: 173MB AST validation
time: 0.056; rss: 177MB name resolution
time: 0.029; rss: 186MB lowering ast -> hir
time: 0.003; rss: 187MB indexing hir
time: 0.003; rss: 187MB attribute checking
time: 0.002; rss: 180MB language item collection
time: 0.003; rss: 180MB lifetime resolution
time: 0.000; rss: 180MB looking for entry point
time: 0.000; rss: 180MB looking for plugin registrar
time: 0.014; rss: 186MB region resolution
time: 0.002; rss: 186MB loop checking
time: 0.002; rss: 186MB static item recursion checking
time: 0.000; rss: 186MB load_dep_graph
time: 0.091; rss: 189MB type collecting
time: 0.001; rss: 189MB variance inference
time: 0.110; rss: 197MB coherence checking
time: 0.108; rss: 198MB wf checking
time: 0.135; rss: 198MB item-types checking
time: 2.527; rss: 220MB item-bodies checking
time: 0.000; rss: 220MB drop-impl checking
time: 0.126; rss: 221MB const checking
time: 0.013; rss: 221MB privacy checking
time: 0.002; rss: 221MB stability index
time: 0.010; rss: 221MB intrinsic checking
time: 0.004; rss: 221MB effect checking
time: 0.036; rss: 221MB match checking
time: 0.015; rss: 221MB liveness checking
time: 0.083; rss: 221MB rvalue checking
time: 0.240; rss: 243MB MIR dump
time: 0.158; rss: 244MB MIR passes
time: 0.225; rss: 245MB borrow checking
time: 0.001; rss: 245MB reachability checking
time: 0.014; rss: 245MB death checking
time: 0.019; rss: 246MB stability checking
time: 0.000; rss: 246MB unused lib feature checking
time: 0.141; rss: 246MB lint checking
time: 0.000; rss: 246MB resolving dependency formats
time: 0.196; rss: 250MB Prepare MIR codegen passes
  time: 0.107; rss: 254MB       write metadata
  time: 0.931; rss: 274MB       translation item collection
  time: 0.034; rss: 277MB       codegen unit partitioning
fatal runtime error: out of memory
@Manishearth

This comment has been minimized.

Copy link
Member

Manishearth commented Jun 30, 2016

hello MIR, we meet again.

@eddyb @nagisa Looks like we got nuked from orbit. Thoughts?

@shepmaster

This comment has been minimized.

Copy link
Member

shepmaster commented Jun 30, 2016

@Manishearth you really need to step up your emoji game. 🚀🌏💥 or something.

@eddyb

This comment has been minimized.

Copy link
Member

eddyb commented Jun 30, 2016

@Manishearth It's old trans still though.

@oli-obk

This comment has been minimized.

Copy link
Collaborator

oli-obk commented Jun 30, 2016

running with -Zorbit takes 5% more maximum memory on my machine.

but compiling in release mode with -Zorbit shaves off 10% of the maximum memory (from ~800 MB to ~725 MB). Regular trans in release mode takes ~755MB

There's also a chance that using a 32 bit host for compiling reduces memory usage.

@Manishearth

This comment has been minimized.

Copy link
Member

Manishearth commented Jun 30, 2016

Yeah, this isn't MIR, because it happens in old-trans.

So eddyb looked at the generated LLVM IR and it seems like rustc is doing a LOT of monomorphizations. We have a ton of hashmaps, and also each LateLintPass impl will impl a bajillion default visitor functions, again.

I suspect this is a cumulative effect, but if someone wants to bisect the lints we use and look for any particular things that stand out (remember that the mod will have to be commented out too, not just the register_lint) that would be nice.

@eddyb

This comment has been minimized.

Copy link
Member

eddyb commented Jun 30, 2016

@DemiMarie

This comment has been minimized.

Copy link

DemiMarie commented Aug 4, 2016

Could trait objects be the solution?

@llogiq

This comment has been minimized.

Copy link
Collaborator

llogiq commented Aug 4, 2016

I suspect that pulling passes together will have a bigger effect. At the moment, we have a lot of passes, which is nice for locality and reasoning about the code but apparently not so nice for rustc's memory habits...

@Manishearth

This comment has been minimized.

Copy link
Member

Manishearth commented Aug 4, 2016

So mentioned this yesterday on irc and totally forgot to file an issue.

Most of our passes implement just one of the check_foo methods, usually check_expr.

We create traits for ExprLint, etc. These only have the lint name method and check_expr. We then define a ExprLintPass(Box<ExprLint>) trait object wrapper which implements LateLintPass with check_expr forwarded to the inner lint. This means that all our expr lints will only need one copy of the defaulted late lint pass methods, which is great!

We could alternatively use a struct with fields for each pass to avoid the virtual dispatch ("pulling them together"), though that gets more cumbersome to deal with. I suspect the virtual dispatch won't cost much, because lints involve virtual dispatch anyway and overall take a trivial amount of time to run.

@llogiq

This comment has been minimized.

Copy link
Collaborator

llogiq commented Aug 4, 2016

Perhaps code generation can help up pull it together? If we change the *LintPass methods to plain functions and have a script collect them and call them in the right order, we'll be golden.

@nagisa

This comment has been minimized.

Copy link

nagisa commented Aug 4, 2016

Cc @nikomatsakis

I has completely forgotten about this issue.

Has anybody actually ran clippy with -Z time-passes or some such to see
where the extra memory use comes from? How much more is used compared to
non MIR (in percent, not absolute units)?

On Jun 30, 2016 7:21 AM, "Manish Goregaokar" notifications@github.com
wrote:

hello MIR, we meet again.

@eddyb https://github.com/eddyb @nagisa https://github.com/nagisa
Looks like we got nuked from orbit.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1062 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AApc0smOJv6X7lX71M1r4-78dWAkpInaks5qQ0RhgaJpZM4JBulv
.

@Manishearth

This comment has been minimized.

Copy link
Member

Manishearth commented Aug 4, 2016

Yeah, with time-passes it seems like the translation step chokes.

llogiq added a commit that referenced this issue Aug 24, 2016

refactored misc to reduce passes
related to #1062

Before: 960MB After: 956MB

So while this reduces code size somewhat, I don't see much memory
improvement here. Still, it's probably worthwile to consider reducing
our passes at least within modules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment