-
-
Notifications
You must be signed in to change notification settings - Fork 382
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OOM while traversing large database. #1036
Comments
This also causes an OOM kill (before disk is exhausted) if let running long enough. extern crate bincode;
fn main() -> sled::Result<()> {
let db = sled::open("db")?;
let t1 = db.open_tree("t1")?;
let t2 = db.open_tree("t2")?;
let mut bincode = bincode::config();
bincode.big_endian();
// This approximates somewhat my workload...
let infinite_data = (0u64..).flat_map(|i| (0u64..100).map(move |j| (i, j)));
for (i, j) in infinite_data {
let key1 = bincode.serialize(&(i, j)).expect("can serialize");
let key2 = bincode.serialize(&(j, i)).expect("can serialize");
t1.insert(key1, vec![])?;
t2.insert(key2, vec![])?;
}
Ok(())
} Using sled 0.31 (and also latest master) on Ubuntu 18 and Rust 1.45 (nightly 2020-05-12). C'mon, man! I can try to live with sled using tons of space, but this is a deal breaker in my case (and basically any big data usecase). Hope it's easy to fix, though. |
Yo, found a mitigation which might also help solve the problem... I found this comment here:
Which led me to fiddle with the
|
As you'll see in my code, I set the cache capacity as low as I could without resolving my problem. So I wonder if we are hitting different issues. |
Good question... I was reluctant in opening a new issue, though. For context, my issue is to load a big dataset. So it's a write problem, not a read problem. @spacejam has been in contact and told me it's a knwon issue, it seems. |
I'm currently looking into this approach for handling this issue: #1093 |
@spacejam That ticket references many inserts, which wasn't my issue. My issue is simply traversing a large database. It could be related issues for all I know, but wanted to make sure. |
sled 0.31.0
rustc 1.42.0 (b8cedc004 2020-03-09)
Ubuntu 19.10
Code:
Expected outcome: Uses a small amount of memory
Actual outcome: Uses several gigabytes of memory and gets killed.
The database is 4.7gb
Some debug logs: https://pastebin.com/e49teW5m
The text was updated successfully, but these errors were encountered: