melnode eats up huge amounts of memory

[ghost]

Details

As the title says, melnode takes up a lot of memory. After reaching a peak in memory, the program exits and displays "Killed" back.

The first machine:

Exit and display "Killed":

The second machine, I couldn't believe that it was taking up a whopping 50 GB of memory...

It's worth noting that after syncing to the latest block, it doesn't seem to be such a heavy memory hog.

Other information

The environment of the first machine is:

• OS: Debian 11.6 amd64
• Kernel: #1 SMP Debian 5.10.162-1 (2023-01-21)
• CPU: Intel i7-8550U
• RAM: 16 GB
• Disk: NVMe SSD 1 TB
• melnode version: 0.20.3
• Commands to run: melnode --index-coins --database /var/local/melnode/

The environment of the second machine is:

• OS: Debian 11.6 amd64
• Kernel: #1 SMP Debian 5.10.162-1 (2023-01-21)
• CPU: Ryzen 5 3600
• RAM: 64 GB
• Disk: Three 500 GB NVMe SSDs in RAID 0
• melnode version: 0.20.3
• Commands to run: melnode --index-coins --database /var/local/melnode/ --advertise I.P.V.4:PORT

[antidmg]

The issue starts happening around block height 1.2M (the blocks start getting pretty big).

Steps to identify the issue:

1. Run melnode under a memory profiler for <1 minute. Make sure to compile it with a profile that has debug symbols turned on (e.g. cargo install --path . --profile release-dbg)
2. Looking at the flamegraph of memory that was never deallocated (leaked only), we see that get_block created large chunks
3. get_block uses a cache that had a VERY high limit of objects (100_000) 🤯 Same as the abbr_block_cache.
4. Lowering these limits to 1000 stabilized the allocation graph as seen below:

The fix is pushed here

Feel free to reopen this issue of the memory issue persists.