Beacon Node memory spikes and crashes, nonstop loop ENR address update #2215
Labels
bug
Something isn't working
Comments
|
I believe this is an issue with IPV6. Disabling ipv6 should work as a temporary fix. |
|
A similar problem, I wonder what it happens to different people=/ |
|
flag |
|
|
This was referenced Feb 21, 2021
|
Please see #2216 (comment) for a hotfix to this issue. |
bors bot
pushed a commit
that referenced
this issue
Feb 21, 2021
## Issue Addressed - Resolves #2215 ## Proposed Changes Addresses a potential loop when the majority of peers indicate that we are contactable via an IPv6 address. See sigp/discv5#62 for further rationale. ## Additional Info The alternative to this PR is to use `--disable-enr-auto-update` and then manually supply an `--enr-address` and `--enr-upd-port`. However, that requires the user to know their IP addresses in order for discovery to work properly. This might not be practical/achievable for some users, hence this hotfix.
bors bot
pushed a commit
that referenced
this issue
Feb 21, 2021
## Issue Addressed - Resolves #2215 ## Proposed Changes Addresses a potential loop when the majority of peers indicate that we are contactable via an IPv6 address. See sigp/discv5#62 for further rationale. ## Additional Info The alternative to this PR is to use `--disable-enr-auto-update` and then manually supply an `--enr-address` and `--enr-upd-port`. However, that requires the user to know their IP addresses in order for discovery to work properly. This might not be practical/achievable for some users, hence this hotfix.
|
Resolved in #2216. Thanks for the detailed report @Wazzymandias! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Intermittently the beacon node spikes in memory usage and consumes all memory and swap of the host system.
The spike in memory usage occurs after a large amount of messages related to updating and persisting ENR.
Version
Reproduced in 1.1.0+ using lighthouse docker images
Lighthouse Beacon Client Docker Images
OS
5.8.0-0.bpo.2-amd64 #1 SMP Debian 5.8.10-1~bpo10+1 (2020-09-26) x86_64 GNU/LinuxHardware
Command
Present Behaviour
Around 2021-02-19 14:11:11 UTC the beacon client exited due to OOM when running
sigp/lighthouse:v1.1.1:I was able to reproduce the issue with different lighthouse image versions from v1.1.0+
Prior to exiting due to OOM the beacon client would print a constant stream of messages related to updating and persisting ENR, but for the same IP:
From another run:
Expected Behaviour
Beacon client should not crash nor constantly loop on updating the same ENR with identical address. I'm not sure what the root cause is but I did not experience this issue until most recently. I'm not familiar with the codebase but perhaps there also should be rate limiting for updating the same ENR, and/or checks for persistence of identical address updates for the same ENR. However that's assuming the ENR loop is cause of the issue.
Steps to resolve
For now have been restarting container. But after some time the ENR loop reappears.
I have also tried lowering
target-peersfrom default of 50 to 30 but the issue still occurs.The text was updated successfully, but these errors were encountered: