Out of memory (OOM) - Prysm killed

[GavinStein1]

Setup:
CPU - AMD Ryzen™ 5 3600
RAM - 16GiB
Storage - 2 TB
OS - Ubuntu 23.10
Execution client - Geth 1.13.14-stable-2bd6bd01
(4GB swap space partition)

Geth launch command:
geth --mainnet --datadir data/ethereum/ --http --ws --authrpc.addr localhost --authrpc.vhosts="localhost" --authrpc.port 8551 --authrpc.jwtsecret=data/ethereum/geth/jwtsecret --cache 256

I started without setting cache but have incrementally lowered it in order to reduce RAM usage. Geth is also suffering from OOM.

Prysm launch command:
./prysm.sh beacon-chain --execution-endpoint=http://localhost:8551 --mainnet --jwt-secret=~/data/ethereum/geth/jwtsecret --checkpoint-sync-url=https://beaconstate.info --genesis-beacon-api-url=https://beaconstate.info --backfill-batch-size 4 --backfill-worker-count 1

Same here, i started without setting backfill-batch-size and backfill-worker-count but then introduced these parameters and lowered them incrementally but to no avail (still OOM).

I initially started with lighthouse but was also getting OOM, and wanted to shift to a go based client.

Are there things I can change when running Geth/Prysm that can limit its demand on memory? Or is it just a matter of upgrading RAM to 32GB?

Documentation on Prysm and Geth both say 16GB is enough. Is that now outdated and the new recommendation is 32GB?

[nisdas]

@GavinStein1 16gb should be more than enough, do you know what is the current resource consumption by each process ?

[GavinStein1]

Thanks @nisdas for the response. From monitoring usage (by calling top command) Prysm starts around 28% and as syncing progresses it is reaching 50%+. I've only monitored usage while syncing because I've never reached the finish before it gets killed. For Geth, when synced it sits around 30% too. I think it might be a combination of concurrent spikes in memory usage by Prysm and Geth (and likely the same when I was operating Lighthouse).

I think you're right in saying that 16gb should be more than enough but I think 32gb would likely resolve this issue. For now, I'll keep monitoring and share any clues I find here.

[nisdas]

Thanks for the clarification @GavinStein1 , can you try running with --enable-experimental-state ? It should help with memory consumption here

[GavinStein1]

Tried that flag but still the same issue. I am gonna try Nimbus as the client and see if i still have the same issues, might be able to help determine if it is my machine that is the issue. Gonna take a while to sync so I will have an update when that is done.

[GavinStein1]

I've been able to successfully sync and stay connected running geth with nimbus. There are spikes in memory usage usually from geth. My best guess is that simultaneous spikes in geth and prysm caused the crashing, but I do not have the expertise to investigate further. I am going to close this issue, apologies I don't have any better answers for anyone else having this problem.