Address updated loop in beacon leads to OOM

[p4fg]

Description

Lighthouse beacon got killed by the linux OOM killer after consuming 117GB RSS memory.

Nov 18 21:58:05 validator kernel: [1326827.608571] Out of memory: Killed process 304391 (lighthouse) total-vm:181985348kB, anon-rss:117344980kB, file-rss:0kB, shmem-rss:0kB, UID:997 pgtables:307460kB oom_score_adj:0

Looking at the logs the application seems to have got stuck in a Address updated loop where it toggles between
udp_port: 9000 and udp_port: 1046 about 500 times per second according to the logs, until it gets killed and restarts.

The problems started 21:44:21 and it got killed 21:58:05.

Nov 18 21:44:21 validator lighthouse[304391]: Nov 18 21:44:21.747 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:44:21 validator lighthouse[304391]: Nov 18 21:44:21.783 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:44:21 validator lighthouse[304391]: Nov 18 21:44:21.814 INFO  Address updated                               ip: REDACTED, udp_port: 1046

<SIMILAR LINES CUT>

Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.070 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.080 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.080 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.080 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.080 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.094 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.095 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.095 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.095 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.095 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.095 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.095 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.097 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.102 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.102 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.106 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.110 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.111 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.113 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.113 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.117 INFO  Address updated                               ip: REDACTED, udp_port: 9000
Nov 18 21:58:05 validator lighthouse[304391]: Nov 18 21:58:05.118 INFO  Address updated                               ip: REDACTED, udp_port: 1046
Nov 18 21:58:05 validator systemd[1]: lighthousebeacon.service: A process of this unit has been killed by the OOM killer.
Nov 18 21:58:10 validator systemd[1]: lighthousebeacon.service: Main process exited, code=killed, status=9/KILL
Nov 18 21:58:10 validator systemd[1]: lighthousebeacon.service: Failed with result 'oom-kill'.
Nov 18 21:58:10 validator systemd[1]: lighthousebeacon.service: Consumed 4d 4h 52min 37.562s CPU time.
Nov 18 21:58:15 validator systemd[1]: lighthousebeacon.service: Scheduled restart job, restart counter is at 1.
Nov 18 21:58:15 validator systemd[1]: Stopped Lighthouse Consensus Client BN (mainnet).
Nov 18 21:58:15 validator systemd[1]: lighthousebeacon.service: Consumed 4d 4h 52min 37.562s CPU time.
Nov 18 21:58:15 validator systemd[1]: Started Lighthouse Consensus Client BN (mainnet).
Nov 18 21:58:15 validator lighthouse[556287]: Nov 18 21:58:15.959 INFO  Lighthouse started                            version: "Lighthouse/v8.0.0-e3ee7fe"
Nov 18 21:58:15 validator lighthouse[556287]: Nov 18 21:58:15.959 INFO  Configured network                            network_name: "mainnet"
Nov 18 21:58:15 validator lighthouse[556287]: Nov 18 21:58:15.961 INFO  Data directory initialised                    datadir: /var/lib/validator-data/lighthouse

Version

Lighthouse/v8.0.0-e3ee7fe. (binary release from github)

Present Behaviour

After restart it behaves normal.

Expected Behaviour

The application should never get into this state and should not leak memory while doing so.

Steps to resolve

Restart lighthouse beacon

[michaelsproul]

Hi @p4fg, can you please send us some debug logs? You can DM me on Discord @sproul.

Definitely looks like a bug, I'm sorry you're experiencing this.

Do you have port 9000 forwarded through your router? I'd be curious to know if adding a forward fixes this (it should), as a temporary workaround while we develop a fix. It's unlikely we will have time to fix this before the Fusaka fork.

[p4fg]

I would love to but i cannot reproduce the problem. It was once and now (after the restart) everything is well. So it is not anything urgent, but i just thought you might want to know about it. (I know looking for bugs this way is sub-optimal, but its the best i got).

[p4fg]

Did some basic analysis with Claude Code. Take it for what it is, a hint on what might be wrong. I unfortunately know too little to be able to say if this is just hallucinations or not (but it usually is not in my experience).

Root Cause

Critical bug in: beacon_node/lighthouse_network/src/discovery/mod.rs:1087

The SocketUpdated event handler incorrectly updates the TCP port with the UDP socket address, creating a feedback loop between discv5 (discovery/UDP layer) and libp2p (TCP layer).

Buggy Code

discv5::Event::SocketUpdated(socket_addr) => {
    info!(ip = %socket_addr.ip(), udp_port = %socket_addr.port(),"Address updated");
    metrics::inc_counter(&metrics::ADDRESS_UPDATE_COUNT);
    // Discv5 will have updated our local ENR. We save the updated version
    // to disk.

    if (self.update_ports.tcp4 && socket_addr.is_ipv4())
        || (self.update_ports.tcp6 && socket_addr.is_ipv6())
    {
        // BUG: Setting TCP port to UDP socket address!
        self.discv5.update_local_enr_socket(socket_addr, true);  // ← Line 1087
    }
    let enr = self.discv5.local_enr();
    enr::save_enr_to_disk(Path::new(&self.enr_dir), &enr);
    // update network globals
    *self.network_globals.local_enr.write() = enr;
}

The Issue

Line 1087 calls update_local_enr_socket(socket_addr, is_tcp=true) where:

• socket_addr contains the UDP discovery port (9000) from discv5
• The second parameter true means "treat this as a TCP socket"
• This incorrectly sets the ENR TCP port to the UDP port value

Feedback Loop Mechanism

The bug creates an infinite event loop between two components:

1. discv5 detects UDP socket (port 9000)
   ↓
2. Emits SocketUpdated event with UDP port 9000
   ↓
3. Bug: Updates ENR TCP port to 9000 (should be 1046)
   ↓
4. ENR change propagates to libp2p
   ↓
5. libp2p detects TCP port mismatch (expects 1046, sees 9000)
   ↓
6. libp2p fires NewListenAddr event to correct TCP port back to 1046
   ↓
7. update_enr_tcp_port(1046) modifies ENR again
   ↓
8. ENR sequence number increments, triggers discv5 re-announcement
   ↓
9. discv5 emits another SocketUpdated event for UDP port 9000
   ↓
10. GOTO step 3 → INFINITE LOOP

Why Port 9000 and 1046?

From beacon_node/lighthouse_network/src/config.rs:

• Port 9000: DEFAULT_DISC_PORT and DEFAULT_TCP_PORT - the UDP discovery port
• Port 1046: Likely the actual libp2p TCP listening port on this specific node

The alternating pattern occurs because:

• 9000: discv5 SocketUpdated event incorrectly sets TCP port to this value
• 1046: libp2p NewListenAddr event corrects TCP port back to actual listening port

Evidence from Configuration

The bug is triggered when ENR ports are not explicitly configured (auto-discovery mode):

let update_ports = UpdatePorts {
    tcp4: config.enr_tcp4_port.is_none(),  // true = auto-update enabled
    tcp6: config.enr_tcp6_port.is_none(),
    quic4: config.enr_quic4_port.is_none(),
    quic6: config.enr_quic6_port.is_none(),
};

When update_ports.tcp4 == true, the buggy code path at line 1087 executes.

Why This Wasn't Caught Earlier

1. Requires specific timing: The feedback loop needs both discv5 and libp2p to be actively processing events simultaneously
2. Trigger condition: May require specific network conditions (NAT traversal, port detection, peer connections)
3. Fast failure: The OOM happens within 14 minutes, but most testing doesn't run long enough in the exact failure state
4. Rate limiting elsewhere: Other rate limiting or event coalescing might normally prevent this, but failed in this case

Proposed Fix

Remove the incorrect TCP port update from the SocketUpdated handler:

discv5::Event::SocketUpdated(socket_addr) => {
    info!(ip = %socket_addr.ip(), udp_port = %socket_addr.port(),"Address updated");
    metrics::inc_counter(&metrics::ADDRESS_UPDATE_COUNT);

    // SocketUpdated is for UDP changes only.
    // Discv5 has already updated the UDP port in the local ENR.
    // TCP ports should ONLY be updated via libp2p's NewListenAddr events.
    //
    // REMOVED: The buggy conditional that updated TCP port with UDP socket

    let enr = self.discv5.local_enr();
    enr::save_enr_to_disk(Path::new(&self.enr_dir), &enr);
    // update network globals
    *self.network_globals.local_enr.write() = enr;
}

Rationale

1. SocketUpdated is UDP-only: This event is emitted by discv5 when it detects UDP socket changes for discovery
2. Discv5 owns UDP: The comment on line 1080 states "Discv5 will have updated our local ENR" - discv5 already handled the UDP update
3. Libp2p owns TCP: TCP port updates should come exclusively from libp2p's NewListenAddr events (lines 1110-1191)
4. Separation of concerns: Mixing UDP and TCP port updates violates the architectural boundary between discv5 and libp2p

[michaelsproul]

@p4fg You should have debug logs from the incident in $datadir/beacon/logs/ (if they haven't been rotated out yet). Claude's analysis doesn't sound quite right to me, as I imagine port 1046 is not the TCP port you're listening on, although it might be the external port that your router has assigned for you

[michaelsproul]

I'll raise this with our networking experts though

[p4fg]

I have the logs for you. Will DM on discord.

[AgeManning]

I'm at a conference at the moment. But I've seen this before. I need to dig up my old analysis. I'll haven't any time right this moment. But this occurred when running inside docker and having another container also use UDP port 9000. What happens in that case, is that docker does SNAT for 9000 and switches ports for the lighthouse container creating this ENR update loop.

I thought we resolved this in: sigp/discv5@42d6ac5

I just did some quick digging. Lighthouse v8.0.0 in Cargo.lock only includes discv5 10.1.

So the patch isn't in Lighthouse version 8.0.0.

I just had a look at v 8.1 and it looks like we haven't done a cargo update, which will resolve this issue.

Essentially Lighthouse isn't running the fix right now. I'll make a PR to update to the latest discovery which will fix this issue.

[AgeManning]

I think #8443 will resolve this issue

[michaelsproul]

Closed by #8443 which is in unstable now and will be part of v8.1.0