Nethermind fills SWAP despite free memory and low vm.swappiness

[499602D2]

Describe the bug
No matter the Init.MemoryHint option, Nethermind always slowly but steadily fills up the swap on my system. The machine has 32 GB of memory, and vm.swappiness set to 5. Currently, 3.11/4 GiB of swap is in use, while only 17.1/31.2 GiB of memory is being used.

Only other programs running are Nimbus and mev-boost, both of which behave nicely.

To Reproduce
Happens always.

Expected behavior

• Nethermind uses available memory instead of swapping onto the disk.
• Increasing init.MemoryHint reduces swap usage. It has no effect: only increases memory usage.
• Reducing init.MemoryHint reduces swap usage. It has no effect: only decreases memory usage.

Screenshots
Grafana dashboard with evolution of memory usage over the last four days. Initial drop in swap usage is due to a restart of Nethermind.

Screenshot from htop. Notice the obscure amount of swap usage.

Desktop (please complete the following information):

• CPU: 4C/4T i5-6500T
• Memory: 32 GB
• OS: Ubuntu 22.04.1 LTS
• Browser: N/A
• Version: Nethermind/v1.14.3-0-707461293-20221022/X64-Linux/6.0.9

Additional context
Startup params:

/usr/share/nethermind/Nethermind.Runner \
                --config=mainnet \
                --Sync.SnapSync=true \
                --Init.BaseDbPath=/var/lib/nethermind \
                --Init.WebSocketsEnabled=true \
                --Network.MaxActivePeers=50 \
                --JsonRpc.Enabled=true \
                --JsonRpc.Port=8545 \
                --JsonRpc.EnginePort=8551 \
                --JsonRpc.JwtSecretFile=/secrets/jwt \
                --Pruning.CacheMb=4096

[asdacap]

@kkokosa Any idea why this is the case?

[499602D2]

This behavior still occurs on v1.16.1, just to add a data point

[OlegJakushkin]

Dear @499602D2: we are currently working on a prototype fixing this issue, will push it as soon as its testing will be completed.

[499602D2]

Thank you for the update @OlegJakushkin, keep up the great work! 🙏

[benaadams]

Resolved in many different changes, but most recent is #6636

[499602D2]

PR looks promising, thank you for your work @benaadams!