v1.9.0-rc.0: Memory leak

[twoeths]

Describe the bug

rss keeps going up on beta mainnet node

Expected behavior

expect it to be like stable mainnet node

Steps to reproduce

it also happens on all goerli beta nodes

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.9.0-rc.0

[twoeths]

looking in to /proc/${pid}/maps file and compare to stable mainnet node, 2 candidates I found

13939:7f9958600000-7f9958601000 r-xp 00000000 09:02 27007162                   /usr/src/lodestar/node_modules/utf-8-validate/prebuilds/linux-x64/node.napi.node   =======> 1st candidate
13961:7f9958a00000-7f9958a02000 r-xp 00000000 09:02 26935736                   /usr/src/lodestar/node_modules/bufferutil/prebuilds/linux-x64/node.napi.node =======> 2nd candidate

both point to web3 dev dependencies of prover packages (cc @nazarhussain ), but they should not be loaded in production environment

I reinstalled node modules with yarn --production and add NODE_ENV=production before our node command, proc maps file does not show dev dependencies, will need monitor the issue again

Update: still got the issue

[twoeths]

another direction is to review metrics in the last 30 days

the issue happens since May 23, need to investigate commits around that day

git log --oneline --since="2023-05-22 00:00" --until="2023-05-23 23:59"
186a5e8899 fix(validator): improve error handling in attestation service (#5511)
4fd72b2b61 refactor: change archiving strategy to always store last finalized (#5520)
42c80973a5 feat: more metrics for sync committee message validation (#5516)
504a9d364f test: update the sim tests to simplify assertion logic (#5501)

[twoeths]

42c8097 in the last 24h

4fd72b2b61 in the last 24h

#5520 is the cause for this issue

[dapplion]

• Noting that a user reported a leak on 1.7.2 for mainnet Mainnet crash on 1.7.2 #5349

[twoeths]

@dapplion this is not the same issue, this issue is merely about rss and node was not crashed while #5349 is related to heap memory

[twoeths]

since we revert #5520, this issue does not happen in the final v1.9.0. If we improve any strategies in archiving state, need to refer to this issue