Container runs out of memory during sync

[tnet-ag]

The 32GB Memory Limit of the Container in docker-compose.yml

    deploy:
      resources:
        limits:
          memory: 32768M

Produces a crash loop during early sync, cause the container runs out of memory

Increased the limit of the running container now manually to 64 GB with

docker update --memory <new_memory_limit> <container_name>

and It works.

The limit should be increased in docker-compose.yml

[AJolly]

Thanks for the fix, I just came here to report the same issues. I tried manually editing the docker-compose.yml file on disk but that did not fix the issue.

[alexpeterson91]

Is anyone still experiencing this issue? @AJolly @tnet-ag ? because i think this was from a bad version, i have erigon clients full archive nodes and all with embedded caplin nodes doing the beacon chain in erigon as well for goerli, mainnet, sepolia, and gnosis chain and ive not had this issue ever. All packages are by default set to a max of 16GB RAM via docker, this is not a hard limit and programs can easily break out of it, (ive had other clients with major memory leaks like older nethermind and older besu versions so much so that they were limited to 16 GB and used all system memory (in some cases on my big servers, several hundred GB of RAM).
We set the 16GB limit for all packages unless specified in the package, like erigon, which we give 32GB of RAM, along with some others known to perform better with more RAM.
However we can never set a package's memory usage to 64GB or more for public publishing, because most of our users dont even have 64GB total, I've tried to explain for years that 64GB is the minimum RAM for Dappnode with a full ethereum stack and any other package, but most people look at minimum requirements of 32GB and go with that (we also have users trying to use 16GB or even as little as 8GB which is just far too little RAM.
The problem with going above 64GB or even using it is that for all Small Form Factor PCs including the most popular ones we use, Intel (now Asus) NUCs have a max limit on even their top of the line models of 64GB, and this is the case with most since they use SO-DIMMS (laptop size memory) and have max 2 slots, some high end latest edition boards claim to handle 42GB modules, but i still have never found a single 42GB SO-DIMM for anything on the market so the physical limit is 64GB. and most users dont even have that.
Further if we increased the max usage of one package on a node with 64GB of RAM to 64GB of RAM, then you're making all the RAM available from the whole machine to one package, none to the system, host, any other packages, etc if it baloons in memory and hits 64 bam your whole system is gonna crash due to OOM.
Erigon uses a lot of reasources, but it doesnt use nearly 32GB of RAM even when syncing in the current and recent versions, your experiences must have been with a bad version with a bad bug in it.

Let me know if either of you are still experiencing it, if not I will close this issue, sorry for the late response, but we handle all support via Discord and Issues on many packages are ignored if they dont fully follow issue templates and if you suggest a fix as you did @tnet-ag open a PR to close your issue with your fix in it, our CI will test it and alert the team to review the automated tests and look deeper into your issue.

[AJolly]

I'm not experiencing this issue, however I've recently had erigon on my dappnode lose state a few times, and have not been able to diagnose. Posted about it in discord a few days ago.

I understand that most users may not have more than 64gb, but wouldn't it make more sense and future proof to set the memory limit based on the ram available to the host?