Peer count decays after boot; need 30s keepalive redial loop

[Reiers]

Problem

After boot, peer count quickly decays from MinPeers=50 (and often higher) down to ~10-20 within a few minutes. Observed on live mainnet daemon (f03678816) on commit 525e75a:

boot:         50 peers (DHT seeded rt_size=29, connected=50+)
+~3 min:      33 peers
+~3 min 12s:  30 peers
+~3 min 24s:  20 peers
+~3 min 36s:  19 peers
+~3 min 48s:  14 peers

That's a drop from 50 \u2192 14 in less than a minute of sampling. Way below the configured MinPeers=50 floor.

Why this happens

Two contributing causes:

1. The DHT refresh loop only fires every 5 minutes. dhtRefreshLoop in net/libp2p/dht.go:242 uses opts.RefreshInterval = 5 * time.Minute. So by the time the loop notices peers have dropped, we've been below the target for several minutes. Peer count can decay much faster than that.

2. The refresh loop's trigger is TargetPeers=30, not MinPeers=50. The check is:

   if n < opts.TargetPeers { /* re-seed */ }

   With TargetPeers defaulting to 30 and MinPeers configured at 50, there's a 20-peer band where we're below the floor but the refresh loop does nothing.

3. The connmgr's low watermark is supposed to handle this but only does so when there are 'reserve' peers known to the peerstore. After bootstrap dials settle, the peerstore mostly empties as connections close, so the connmgr has nothing to dial.

Proposed fix

Add a tight redial keepalive loop alongside the existing 5-minute DHT refresh:

keepaliveInterval := 30 * time.Second
go h.keepaliveLoop(ctx, opts)

Where keepaliveLoop:

1. Every 30s, check h.PeerCount().
2. If below MinPeers (NOT TargetPeers):
   • Re-dial all bootstrap peers (cheap, capped at 7 dials)
   • Walk the DHT routing table and dial up to N=10 unconnected peers
   • Push everything successfully connected back into the DHT routing table via TryAddPeer
3. If at or above MinPeers: do nothing (don't churn connections)

The existing dhtRefreshLoop stays for the deeper periodic re-bootstrap (5min cadence). The new keepalive is a tighter, cheaper top-up.

Acceptance test

After deploy on the mainnet daemon, peer count should stabilize at or above MinPeers=50 within 30-60s of boot and stay there. Sample 10 times at 30s intervals over 5 minutes:

for i in {1..10}; do
  curl -sS http://127.0.0.1:9092/api/dashboard/overview | jq .peers
  sleep 30
done

Expected: every sample \u2265 50. Variance \u00b1 5 is fine (connmgr trim during epoch transitions).

Constraints

• Don't be too aggressive: 30s redial is fine; 5s would churn connections needlessly and trigger libp2p backoff.
• Don't bypass the connmgr's grace period (default 20s). Peers we just dialed shouldn't immediately be re-trimmed.
• Stay CGo-free. No new external dependencies.

Files

• net/libp2p/dht.go (add keepaliveLoop)
• net/libp2p/host.go (kick off the loop from EnableDHT)

Related

• Observed during the v1.2.1 mainnet deploy on 2026-05-22.
• The 5-minute refresh loop landed in commit 665a136 (DHT protocol prefix fix); the 30s keepalive is the follow-up that makes it actually maintain the floor in practice.

[Reiers]

Partial fix shipped, peer floor still not held

Commit 68700f0 is live on f03678816. The 30s keepalive loop fires as designed and IS dialing — but peer count still oscillates below MinPeers=50 most of the time.

Live data, 10 samples 30s apart

Time	Peers	Keepalive (cycles/triggered/bootDials/routeDials)
18:30:18	44	1/1/1/7
18:30:48	49	2/1/1/9
18:31:18	47	3/2/2/9
18:31:49	42	4/3/3/11
18:32:19	42	5/4/4/19
18:32:49	41	6/5/5/24
18:33:19	36	7/6/6/27
18:33:50	28	8/7/7/44
18:34:20	26	9/8/8/47
18:34:50	43	10/9/10/64

The loop fires every 30s (cycles 1\u219210), triggers below MinPeers=50 (triggered 1\u21929), and has cumulatively executed 10 bootstrap dials + 64 routing-table dials over 5 minutes. But peer count oscillates 26\u201349.

Honest diagnosis

The 30s redial cadence + 10 routing-dial cap isn't keeping up with the decay rate. Comparing to before the fix:

• Before 68700f0: peer count decayed monotonically from 50 to ~14 over ~3 min with no recovery.
• After 68700f0: peer count oscillates 26\u201349, with the keepalive doing visible work to pull it back up each cycle.

Net: the loop is mitigating the decay but not eliminating it. The acceptance bar from the issue was peers \u2265 50 across all samples; we hit \u22651 only briefly.

Hypothesis on what's actually wrong

Three candidates, ranked by likelihood:

1. Routing-table dials succeed at libp2p layer but peers drop us within seconds. Connect() returns nil when the stream opens, but if the remote peer immediately closes the connection (e.g., they didn't want us, score too low), we count it as a successful dial but lose the peer right back. The 64 routing-dials counter would then overstate actual new-stable-peers.
2. Routing-table is dominated by dead/uninterested peers. The dial-walk loop pulls from the same routing table on every cycle. If most entries are unresponsive Filecoin peers that won't accept us as a libp2p neighbor, we keep retrying them.
3. Outgoing dials are being rate-limited by libp2p backoff. The libp2p dial machinery applies exponential backoff on per-peer dial failure. If we're hitting the same peers each cycle and they're failing, the backoff grows and the effective dial rate drops.

Proposed follow-up work

• Track 'dials that produced a connection still alive at next tick.' Add a Stuck counter alongside RoutingDial: a routing dial "sticks" if the peer is still connected when the next keepalive cycle starts. If RoutingDial - Stuck is consistently high, hypothesis 1 is confirmed.
• Filter out peers we recently tried. Maintain a per-peer lastDialAttempt map in the keepalive loop. Skip peers we tried in the last 5 minutes (defer to libp2p backoff).
• Walk the routing table differently each cycle. Right now ListPeers() returns peers in routing-table order. Randomize each call so we don't keep retrying the same dead set.
• Aggressive bootstrap-only-mode option. As a safety net: when peers < MinPeers/2, fire a GetClosestPeers(random_key) walk to force the routing table to absorb new peers from across the network. Currently this only happens every 5min via dhtClosestWalkLoop.

Reopening vs new issue?

Two options. Leaving #9 open and tracking the follow-up work in this comment, OR closing #9 (the 30s loop is shipped) and filing #12 for the deeper fix. I'll leave it to whoever picks it up; no strong preference.

Keepalive observability via the dashboard works \u2014 the data above came straight from /api/dashboard/sync and shows the loop's activity in real time, which makes diagnosing follow-up regressions much faster than grepping logs.

[Reiers]

Reopening: the 30s keepalive loop shipped in 68700f0 didn't meet the acceptance bar. Live evidence on f03678816 shows peer count oscillating 26-49, never sustaining the MinPeers=50 floor. See the prior comment in this thread for the data + diagnosis. Keepalive is mitigating the decay but not eliminating it. The follow-up work outlined above is what's left.

[Reiers]

Acceptance test results, 2026-05-22 21:10-21:15 CEST (commit f1d3b98)

Deployed the four follow-up improvements (stuck-tracking, dial cooldown, randomized walk, aggressive closest-walk) to f03678816 and ran the acceptance sweep. The new instrumentation is the most valuable output of this iteration; the acceptance bar still isn't met.

sample   peers    triggered   stuck   routing_dials   bootstrap_dials   closest_walks
   1       39         1          0         0                0                  0
   2       85         1          1         9                1                  0
   3       33         2         10         9                1                  0
   4       18         3         11        19                2                  0
   5       24         4         13        29                3                  0
   6       38         5         14        44                5                  0
   7       26         6         18        49                6                  0
   8       30         7         19        60                7                  1   <- aggressive walk
   9       25         8         21        64                8                  1
  10       22         9         22        70                9                  2   <- second aggressive walk

1/10 samples \u2265 MinPeers=50. Bar not met. But: this is the first run with real signal on why instead of pure black-box.

The smoking gun

22 of 70 routing dials (31%) didn't survive to the next cycle. That's a huge stuck rate. The two aggressive closest-walks fired (samples 8, 10) didn't measurably help. The dial budget isn't the bottleneck \u2014 the keepalive is making dials, they're connecting, then peers are dropping us within 30 seconds.

That confirms hypothesis 1 from the prior diagnosis: we're accepting the libp2p stream then getting dropped, almost certainly because we don't speak the Filecoin sub-protocols those peers expect (no graphsync, no F3 voting, no Filecoin Bitswap quota, low score on whatever they run). Real Filecoin nodes are scoring us low and trimming us during their own connmgr passes.

What this means

The keepalive layer can't fix this. We can dial faster, walk wider, randomize harder \u2014 the peers still drop us. The fix is at a different layer:

1. Engage more libp2p protocols. Right now we run gossipsub + Bitswap + DHT + cert-exchange. We don't run graphsync, F3 voting, or the data-transfer protocols. Adding even passive participation in graphsync (just answering 'I don't have it') would make us look more like a real Filecoin node and could raise our score.

2. Use the Filecoin Hello protocol. Lotus / Forest start every connection with /fil/hello/1.0.0 exchanging chain head info. We don't. Peers without a hello response within a few seconds may be downgraded in their connmgr.

3. Apply for a positive score with our own connmgr decorations. When we successfully fetch a Bitswap block, mark the source peer as 'useful' so our own connmgr doesn't trim them either. Right now we have nothing pinning successful peers.

4. Accept that we're a light client. The keepalive could simply target a lower MinPeers (e.g., 20-25) that we can actually hold. The marketing claim 'connects to MinPeers Filecoin nodes' would degrade to 'connects to ~25 nodes,' which is fine for client-mode functionality (we only need 3-5 peers to fetch reliably; the rest is redundancy).

Recommendation

Close this iteration of the keepalive fix. Open a new issue, P1, 'Improve Lantern's peer-score on remote Filecoin nodes', scoped to the four protocol gaps above. The keepalive instrumentation we built (Stuck, ClosestWalks, etc.) is the diagnosis surface we'll need to know if that work is helping.

The honest result of this comment is: the 30s keepalive loop + cooldown + randomized walk + aggressive closest-walk is as far as we can take peer count from the dialing side. The next round is on the protocol-participation side.

Leaving #9 open for now to track the deeper work, but the commit f1d3b98 ships everything the issue title called for: a 30s keepalive redial loop. The acceptance bar that peers stay >= 50 isn't met but that target may itself be wrong for a light client.