vec or hashmap seem to still hold heap memory after lifetime

[Right2Left]

I tried this code:

use std::thread;
use std::time::Duration;
use std::collections::HashMap;

 
fn main() {
    //let value = read_file("./1k.conf");
    let value = "test content".to_string();

    println!("begin");
    thread::sleep(Duration::from_secs(20));

    test_vec_not_clear(&value);
    wait_for_free(10);

    test_map_not_clear(&value);
    wait_for_free(10);

    wait_for_free(40);

    test_map_not_clear(&value);
    wait_for_free(10);

    println!("end");
    loop{
        thread::sleep(Duration::from_secs(20));
    }

}

fn wait_for_free(count: i32){
    for i in 0..count  {
        println!("wait count: {}", i);
        thread::sleep(Duration::from_secs(1));
    }
}

fn test_vec_not_clear(value: &str){
    println!("to test vec not clear");
    thread::sleep(Duration::from_secs(5));
    let mut vec = Vec::new();
    for _i in 0..1000000{
        vec.push(value.to_string());
    }
}

fn test_map_not_clear(value: &str){
    println!("to test map not clear");
    thread::sleep(Duration::from_secs(5));
    let mut map = HashMap::new();
    for i in 0..1000000{
        map.insert(i.to_string(), value.to_string());
    }
}

then use valgrind to start:
valgrind --tool=massif --detailed-freq=1 --massif-out-file=massif.out.test.%p ./test_mem

I expected to see this happen:
memory will down to slow usage after line:24
28M -> 300M -> 28M -> 300M -> 28M -> 300M -> 28M

Instead, this happened:
the memory usage is :
28M -> 300M -> 85M -> 300M -> 137M -> 300M -> 137M
the massif report：

Meta

rustc --version --verbose:

test@test-Standard-PC-i440FX-PIIX-1996:~/hzx/test_mem/target/debug$ rustc --version --verbose
rustc 1.41.1 (f3e1a954d 2020-02-24)
binary: rustc
commit-hash: f3e1a954d2ead4e2fc197c7da7d71e6c61bad196
commit-date: 2020-02-24
host: x86_64-unknown-linux-gnu
release: 1.41.1
LLVM version: 9.0
test@test-Standard-PC-i440FX-PIIX-1996:~/hzx/test_mem/target/debug$ 

rustc 1.46.0-nightly (feb3536eb 2020-06-09)
binary: rustc
commit-hash: feb3536eba10c2e4585d066629598f03d5ddc7c6
commit-date: 2020-06-09
host: x86_64-unknown-linux-gnu
release: 1.46.0-nightly
LLVM version: 10.0

in lastest version is same.

Backtrace

<backtrace>

[Right2Left]

the last snapshot is:

#-----------
snapshot=80
#-----------
time=33575861397
mem_heap_B=104436498
mem_heap_extra_B=2822926
mem_stacks_B=0
heap_tree=detailed
n2: 104436498 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
n2: 104435061 0x1126AB: alloc::alloc::alloc (alloc.rs:84)
n1: 102760464 0x117E9F: hashbrown::raw::RawTable::new_uninitialized (mod.rs:383)
n1: 102760464 0x1182A3: hashbrown::raw::RawTable::try_with_capacity (mod.rs:407)
n1: 102760464 0x118DE5: hashbrown::raw::RawTable::resize (mod.rs:788)
n1: 102760464 0x116E4C: hashbrown::raw::RawTable::reserve_rehash (mod.rs:669)
n1: 102760464 0x119466: hashbrown::raw::RawTable::reserve (mod.rs:626)
n1: 102760464 0x118BEF: hashbrown::raw::RawTable::insert (mod.rs:841)
n1: 102760464 0x11585E: hashbrown::map::HashMap<K,V,S>::insert (map.rs:855)
n1: 102760464 0x119FC8: std::collections::hash::map::HashMap<K,V,S>::insert (map.rs:817)
n2: 102760464 0x119DA9: test_mem::test_map_not_clear (main.rs:52)
n1: 102760464 0x11988E: test_mem::main (main.rs:21)
n1: 102760464 0x11C2BF: std::rt::lang_start::{{closure}} (rt.rs:67)
n1: 102760464 0x125CE2: {{closure}} (rt.rs:52)
n1: 102760464 0x125CE2: std::panicking::try::do_call (panicking.rs:292)
n1: 102760464 0x127AA9: __rust_maybe_catch_panic (lib.rs:78)
n1: 102760464 0x1266DF: try<i32,closure-0> (panicking.rs:270)
n1: 102760464 0x1266DF: catch_unwind<closure-0,i32> (panic.rs:394)
n1: 102760464 0x1266DF: std::rt::lang_start_internal (rt.rs:51)
n1: 102760464 0x11C298: std::rt::lang_start (rt.rs:67)
n0: 102760464 0x119E49: main (in /home/test/hzx/test_mem/target/debug/test_mem)
n0: 0 in 1 place, below massif's threshold (1.00%)
n2: 1674597 0x112541: <alloc::alloc::Global as core::alloc::Alloc>::alloc (alloc.rs:172)
n1: 1123344 0x113BE6: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:98)
n1: 1123344 0x113A54: alloc::raw_vec::RawVec::with_capacity (raw_vec.rs:167)
n1: 1123344 0x112E82: alloc::vec::Vec::with_capacity (vec.rs:362)
n1: 1123344 0x115FFE: alloc::slice::hack::to_vec (slice.rs:158)
n1: 1123344 0x10D253: alloc::slice::<impl [T]>::to_vec (slice.rs:394)
n1: 1123344 0x10D283: alloc::slice::<impl alloc::borrow::ToOwned for [T]>::to_owned (slice.rs:727)
n1: 1123344 0x1196C8: alloc::str::::to_owned (str.rs:207)
n1: 1123344 0x1164C3: <alloc::string::String as core::convert::From<&str>>::from (string.rs:2231)
n2: 1123344 0x116133: ::to_string (string.rs:2191)
n2: 1123332 0x119D7D: test_mem::test_map_not_clear (main.rs:52)
n1: 1123332 0x11988E: test_mem::main (main.rs:21)
n1: 1123332 0x11C2BF: std::rt::lang_start::{{closure}} (rt.rs:67)
n1: 1123332 0x125CE2: {{closure}} (rt.rs:52)
n1: 1123332 0x125CE2: std::panicking::try::do_call (panicking.rs:292)
n1: 1123332 0x127AA9: __rust_maybe_catch_panic (lib.rs:78)
n1: 1123332 0x1266DF: try<i32,closure-0> (panicking.rs:270)
n1: 1123332 0x1266DF: catch_unwind<closure-0,i32> (panic.rs:394)
n1: 1123332 0x1266DF: std::rt::lang_start_internal (rt.rs:51)
n1: 1123332 0x11C298: std::rt::lang_start (rt.rs:67)
n0: 1123332 0x119E49: main (in /home/test/hzx/test_mem/target/debug/test_mem)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 12 in 2 places, all below massif's threshold (1.00%)
n0: 551253 in 2 places, all below massif's threshold (1.00%)
n0: 1437 in 19 places, all below massif's threshold (1.00%)

[crlf0710]

EDIT: Sorry i read the issue wrong.

[Right2Left]

I had try use:
map.clear();
map.shrink_to_fit();
drop(map);

but not effect.

[tesuji]

The allocator doesn't returns memory to the OS but reuse them for later allocations.

[Right2Left]

The allocator doesn't returns memory to the OS but reuse them for later allocations.

i think so, but I have no idea about how much will reuse? What is the relationship with code? Is there any option to returns memory directly?

[CohenArthur]

Constantly allocating and deallocating memory is a costly operation. In order to save on performance I'm guessing Rust chooses to reuse some of it. I don't really see an issue here, this behavior seems completely normal. What are you really trying to do ?

[sfackler]

Rust is not choosing to do anything here - the system allocator is.

[DAigory]

Hi. I'm not sure it is correct behavior.
Try to change HashMap to BtreeMap or vec. Result: BtreeMap and Vec will like you expected (return memory to system)
I want to know...how to control it...and why it happening?
For example I'm going to load bigdata to hashMap and never use it...My process will never back this memory to System?

[willygroup]

Hi I have a similar issue with my project.
Now I'm testing the hashmap performances.

use std::{collections::HashMap, thread, time::Duration};

fn func() {
    let mut map: HashMap<String, Vec<u8>> = HashMap::new();

    for i in 0..100_000 {
        let key = format!("{}", i);
        map.insert(key, vec![0; 50_000]);
    }
}

fn main() {
    func();
    println!("LOOP");
    loop {
        thread::sleep(Duration::from_secs(5));
    }
}

Running this the memory occupied from the map is not released to the system, so it takes about 5 GB of RAM.
Instead if I use an HashMap<i32, Vec> after the executing of func the memory is released and the consumption of RAM drop from 5GB to about zero.
Can you show me a way to understand what this behavior depends on?

Thanks a lot!

[dertin]

I have the same problem I use an HashMap<&'static str, MyEnum>
Is there any news about the cause and how to avoid it?
Thanks!