rustc runs out of memory while trying to compile a large static (lazy_static!) HashMap

[ctrlcctrlv]

    Blocking waiting for file lock on build directory
   Compiling crash v0.1.0 (/home/fred/Workspace/rtuts/crash)
error: Could not compile `crash`.

Caused by:
  process didn't exit successfully: `rustc --edition=2018 --crate-name crash src/main.rs --color always --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=71bdac3a949cee3b -C extra-filename=-71bdac3a949cee3b --out-dir /home/fred/Workspace/rtuts/crash/target/debug/deps -C incremental=/home/fred/Workspace/rtuts/crash/target/debug/incremental -L dependency=/home/fred/Workspace/rtuts/crash/target/debug/deps --extern lazy_static=/home/fred/Workspace/rtuts/crash/target/debug/deps/liblazy_static-7468e7fc38fb612c.rlib` (signal: 9, SIGKILL: kill)

shell returned 101

My machine has 48GB of memory, yet still can't compile the example project, which creates a static hash map at compile time.

I tried both nightly and stable, same thing.

The input source code is only 19M, only containing u32s and strings. This shouldn't happen, GCC could do this no problem.

Project crash.zip

[JuanPotato]

Hey I'm doing almost your exact thing. I solved it by using a phf. It still crashed when I had all the Unicode entries added in one file, so I had the data in another file and read it from there in a loop while adding entries.

https://github.com/JuanPotato/charname

[jonas-schievink]

GCC is not a Rust compiler, so that comparison doesn't really work. It's also a single function containing 277000 lines, so it's not exactly surprising that it uses a lot of time and memory to build.

That said, this should of course not blow up as drastically. According to -Ztime-passes, the memory usage spike happens after the misc checking 2 phase, which should be the borrow checking phase. It also reproduces with --edition=2018 with NLL (after printing solve_nll_region_constraints). Phases before that are comparatively tame, but during borrow checking the compiler somehow starts allocating multiple gigabytes per second.

Even if this will be fixed, I can not recommend compiling this much generated code. A better solution might be to prepare the data in a build script and including it using include_bytes! or include_str!, which shouldn't suffer from this.

[ctrlcctrlv]

@jonas-schievink This should be fixed. Compiling in data like this is routine. I do not want to use a data file or include_str, I want all the data in the binary directly so the program can begin using it right away without deserializing.

Compiling large amounts of data into a program as an optimization is common to do in C and C++. Machine learning, video game graphics, etc. This is essential for a systems programming language to be able to do.

@JuanPotato Are there run-time costs? It's interesting how close the projects are, but I just made crash to demo the problem, my real program is a text mode character map so it will require all the Unicode data to be compiled in.

[JuanPotato]

I'm not sure about runtime costs, you would have to look up more about phf (perfect hash functions) for that.

[ctrlcctrlv]

I tried using PHF but it similarly uses all my memory and crashes. Didn't try using codegen though, just phf::map.

[JuanPotato]

Yeah I had the same issue and ended up with my build.rs. My friend says phf might have better performance than a standard hashmap. I'll try a benchmark sometime today

[hellow554]

I want all the data in the binary directly

That's exactly what include_str! or include_bytes! does.

[ctrlcctrlv]

@hellow554 Without deserializing.

[AlexTMjugador]

I've just got bitten by this issue when building a Rust file with ~26k quickphf set entries, worth 8.2 MiB of Rust source code. Unlike the quickphf benchmarks, I used a struct with strings for the value type instead of primitive integers.