Fork leveldb to apply lower mmap limit

[jszwedko]

While investigating #7246 and other issues, we've identified that leveldb will mmap up to 1000 files at 4 MB per file. This commonly results in users running into OOM issues using disk buffers.

Unfortunately, this value is not configurable so, to modify it, we have to fork leveldb-sys to change it here:

https://github.com/skade/leveldb-sys/blob/c5383aaf9e264041c951e9a2aa7381613d0d0dba/deps/leveldb-1.22/util/env_posix.cc#L46

[blt]

Hmm, well. I have forked leveldb and can confirm that only the specified number of files are being mapped into the process, which is good. What's bad is vector's resident mem is still roughly equivalent to the disk buffer size. Looking into this further.

[blt]

Interesting, so check this out:

It looks like, because my system has no memory pressure, linux is being lazy about reclaiming pages but what happens is the unreclaimed space slowly fills up with retries. Allocation point appears to be src/sinks/util/retries.rs:150. Raw massif:

massif.out.778423.zip

I'm going to run vector for a long period of time now, see if these start growing as I expect. Afterward I'll run vector in a memory constrained environment.

[blt]

Here's a longer run with no memory pressure:

massif.out.784090.zip

I had expected the retry memory to continue to grow, so the linear curve here surprises me.

[blt]

Alright, as desired when run like so:

> systemd-run --scope --property MemoryHigh=250M --property MemoryMax=300M ./target/release/vector -qqq --config /tmp/disk-blowup.toml

vector now stays within the 300M limit, which previously was not true. Virtual memory use is still quite high but resident size is within boundary, so this should address #7246. I'll have a PR shortly.

[jszwedko]

Interesting, so check this out:

It looks like, because my system has no memory pressure, linux is being lazy about reclaiming pages but what happens is the unreclaimed space slowly fills up with retries. Allocation point appears to be src/sinks/util/retries.rs:150. Raw massif:

massif.out.778423.zip

I'm going to run vector for a long period of time now, see if these start growing as I expect. Afterward I'll run vector in a memory constrained environment.

Hmmm, interesting. Do you think there is an issue there? Should we be piling up retries like that?

[blt]

It’s a good question. I don’t know. It’s something that users will only notice if their vector doesn’t have memory pressure, which most will have. I’d say it’s more of a curiosity until we learn otherwise, a potential place to optimize away clones if someone has a clever notion.

[portante]

https://github.com/skade/leveldb-sys/blob/c5383aaf9e264041c951e9a2aa7381613d0d0dba/deps/leveldb-1.22/util/env_posix.cc#L46

Perhaps its worth filing an issue against leveldb?

Seems like maintaining a fork can lead to other problems.

[blt]

https://github.com/skade/leveldb-sys/blob/c5383aaf9e264041c951e9a2aa7381613d0d0dba/deps/leveldb-1.22/util/env_posix.cc#L46

Perhaps its worth filing an issue against leveldb?

Seems like maintaining a fork can lead to other problems.

This is a good question. There's been discussion like this in upstream -- see google/leveldb#866 -- and their suggestion is to fork. Realistically leveldb is an awkward fit for what we need in vector. It's a KV store and we just need a disk-backed queue. My expectation is that we'll be reworking the buffer code entirely and removing leveldb within the year.