rustc uses insane amounts of memory when compiling a function that loads 1500+ function pointers from a DLL

[fschutt]

For my use-case, I'm trying to load a lot of function pointers from a .dll file using the libloading crate. However, I can't compile a load_my_dll() function if I have more than maybe 1000 function pointers, because rustc uses an insane amount of memory for compilation:

I can pretty much watch my computer lock up in real time because it runs out of memory (see the pie in the lower left corner).

I've built a test-case here: https://github.com/fschutt/libloading-loadtest

Look in the generate.py file, I've currently set it to 2000 function pointers, but that's not even a lot. Larger DLLs will likely need 5000 - 30000 function pointers. Clone the repository and run cargo check. After you wait a while, the memory usage will go up dramatically. This bug also happens on nightly.

I think this may have to do with how verbose the code output of generics are. Is there a way to fix it from my side?

Meta

rustc --version --verbose:

rustc 1.44.1 (c7087fe00 2020-06-17)
binary: rustc
commit-hash: c7087fe00d2ba919df1d813c040a5d47e43b0fe7
commit-date: 2020-06-17
host: x86_64-unknown-linux-gnu
release: 1.44.1
LLVM version: 9.0

[ecstatic-morse]

According to -Ztime-passes, most of the time is spent in type checking (35s for item_bodies_checking/type_check_crate) and borrow checking (19s). This seems to be caused by the thousands of closures passed to map_err, not by libloading itself. After changing map_err to ok and Result to Option, the example takes only 9 seconds to check (down from over a minute).

[the8472]

as a workaround, this reduces peak memory consumption for the provided testcase to 1.5GB:

[profile.dev]
incremental = false

[fschutt]

Well, I just created a "mini" version of libloading that only uses the raw dlopen() / dlsym() and dlclose() and does not use map_err: github / crates.io

I pushed a mini branch that uses the libloading_mini crate, to reduce generics, but I still can't compile it - now it doesn't run out of memory, but I had to stop the process after five minutes, there seems to be something seriously wrong with the typechecking.

I'm don't know what the correct command for showing time-passes is, i.e. how to run cargo with -Ztime-passes. Running cargo -Ztime-passes check only results in error: unknown -Z flag specified: time-passes

[ecstatic-morse]

Your new example calls mem::transmute two thousand times. Each of those calls has two type variables that need to be inferred, just like the closures in your previous example. We should be able to do better here.

[the8472]

The mini branch does not build

error[E0382]: borrow of moved value: `lib`
    --> src/lib.rs:4018:40
     |
4015 |         let lib = Library::new(path)?;
     |             --- move occurs because `lib` has type `libloading_mini::Library`, which does not implement the `Copy` trait
4016 |         let dll = BigDll {
4017 |             lib,
     |             --- value moved here
4018 |             function_0: mem::transmute(lib.get(b"function_0")?),
     |                                        ^^^ value borrowed here after move

[fschutt]

@the8472 Fixed it, try again. However, I didn't even get that far when running cargo check, that's why I didn't notice the error.

[the8472]

With the fix it compiles in 6s.

[fschutt]

It seems that one workaround is to put the function-pointer loading code into separate functions. If I generate hundreds of small functions instead of putting everything in one big function, it compiles fast (well, still 10 seconds on my laptop, but at least I can get it to compile at all):

// compiles fast
unsafe fn load_fn_570(lib: &Library) -> Option<fn(_:  _570) -> _571> { 
    Some(mem::transmute(lib.get(b"function_570")?)) 
}
unsafe fn load_fn_571(lib: &Library) -> Option<fn(_:  _571) -> _572> {
     Some(mem::transmute(lib.get(b"function_571")?)) 
}



// compiles fast
fn load_dll() -> Option<Dll> {
    // <...>
    let function_570 = load_fn_570(&lib)?;
    let function_571 = load_fn_571(&lib)?;
    // <...>
}

instead of:

// compiles slow
fn load_dll() -> Option<Dll> {
    // <...>
    let function_570: fn(_:  _570) -> _571 = mem::transmute(lib.get(b"function_570")?);
    let function_571: fn(_:  _571) -> _572 = mem::transmute(lib.get(b"function_571")?);
    // <...>
}

[fschutt]

Wait, nevermind, the second approach is slower (9s vs 7s).