[BEND][bug] nearcore Memory Leak

[mbucci]

Describe the bug or test
nearcore appears to have a memory leak.

To Reproduce
Steps to reproduce the behavior:

1. Run a node
2. Observe gradual rise in memory usage, especially during State Sync
3. Highest observed memory usage is 26GiB

Expected behavior OR Outcome of test
Memory usage doesn't exceed a reasonable limit (e.g. 4, 8, 16GiB)

Specs (please complete the following information):

• OS: Amazon Linux 2
• Which cloud / on-premise: Cloud (AWS EC2)
• CPU: 8
• RAM: 32
• Disk space: 500GB

Additional context
Add any other context about the problem here.

Screenshots Optional

[sweetpotatoz]

set -m 2g (edited in nodelib.py) but container still got shutdown
kernel logs as below

Nov 05 10:29:55 kernel: near invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Nov 05 10:29:55 kernel:  oom_kill_process+0x21f/0x420
Nov 05 10:29:55 kernel:  out_of_memory+0x2b6/0x4d0
Nov 05 10:29:55 kernel:  mem_cgroup_out_of_memory+0xbb/0xd0
Nov 05 10:29:55 kernel:  mem_cgroup_oom_synchronize+0x2e8/0x320
Nov 05 10:29:55 kernel:  pagefault_out_of_memory+0x36/0x7b
Nov 05 10:29:55 kernel: memory: usage 2097152kB, limit 2097152kB, failcnt 59
Nov 05 10:29:55 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Nov 05 10:29:55 kernel: Memory cgroup stats for /docker/8d4aa0f4d7ad107e187b5b00377139280ea493e1037edb7d533cca9d0a7d2685: cache:0KB rss:2090264KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:2090240KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 05 10:29:55 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov 05 10:29:55 kernel: Memory cgroup out of memory: Kill process 22239 (near) score 1003 or sacrifice child
Nov 05 10:29:56 kernel: oom_reaper: reaped process 22239 (near), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

[sweetpotatoz]

withOUT setting -m 2g the kernel logs as below

Nov 06 02:30:12   kernel: systemd-udevd invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=-1000
Nov 06 02:30:12   kernel:  oom_kill_process+0x21f/0x420
Nov 06 02:30:12   kernel:  out_of_memory+0x2b6/0x4d0
Nov 06 02:30:12   kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov 06 02:30:12   kernel: Out of memory: Kill process 26464 (near) score 792 or sacrifice child
Nov 06 02:30:12   kernel: oom_reaper: reaped process 26464 (near), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

[bowenwang1996]

This outrageous memory leak should be fixed in the new version.