Remove fork-related code from Rust #7894
illicitonion left a comment
Welcome to the project! This looks great, thanks!
CI is showing that the rust code hasn't been formatted with rustfmt; can you run
I recommend running
This is still failing due to error
I originally thought this was us running out of memory because of V2 using too much of it, but I don't think that's the case anymore. This failure has happened twice now for this PR, and hasn't shown up on any other builds.
@illicitonion any thoughts on what might be causing it? Greg wasn't able to reproduce locally via
Also, has a merge conflict now.
blorente left a comment
Welcome to the project!
I don't know why the out of memory error happens, but to reproduce it locally you might want to try setting this value to something lower: https://github.com/pantsbuild/pants/blob/master/src/rust/engine/fs/src/store.rs#L24
I do have thoughts! So, a while ago, we made the store attempt to allocate a ridiculous amount of virtual memory. This is needed because it mmaps a potentially very large file from disk, and we want to allow the file tot be very large. We make a store per pants process, or per scheduler we create in unit tests. @stuhood ended up making us re-use schedulers across the test-cases in a test-class, because he couldn't find a convenient and reliable way to force python to gc away the schedulers (which have stores) after each test-case, and so we ended up collecting a bunch of these, and they blew up memory usage like crazy.
We currently have two separate LMDB stores, one for files (which may be large), and one for metadata about directories (which should be tiny), but we allocate the same giant amount of virtual memory to both of them. We can probably trivially drop the allocation to the one for directories by a few orders of magnitude, which I will try doing now... This is still kind of hacking around the problem, but... shrug
The reason this PR will be making us more likely to max out memory is that before Stores were only lazily initialized when they were actually needed (because that's how
Also, it's not just this PR; I saw this on another of my PRs today too.
I've just pushed a rebase, because that merge conflict was totally my fault, and totally foreseeable by me! Sorry for the hassle!