Node peer recovery after network outage: seeds, discovery, and sync selection

[adamantmm]

ADAMANT nodes maintain peer connectivity through three separate mechanisms that are easy to confuse when reading console logs after a network outage. This note explains how they interact, why sync can stall even while seed peers are still being contacted, and what operators should expect during recovery.

Background

A node keeps an in-memory peer table. Entries come from:

• Seed peers listed in peers.list (config)
• Persisted peers loaded from the database on startup
• Discovered peers returned by other nodes via GET /peer/list

Seed peers are "frozen": they are never removed from the table, even when requests fail.

Peer states

Each peer has a state:

State	Meaning
BANNED (0)	Excluded from normal use
DISCONNECTED (1)	Known peer, not currently usable for sync/broadcast
CONNECTED (2)	Recently responded successfully; eligible for sync

On request failures, a peer's success rate drops. When a previously CONNECTED peer falls below 80% success, its state degrades to DISCONNECTED. Network timeouts (ECONNABORTED) do not remove the peer; they only reduce the success rate.

Three parallel mechanisms

1. Seed ping (quiet)

On startup and every ~5 seconds, the node pings every seed peer from config (GET /peer/height). Failures are logged at trace level and are usually invisible in default console output.

A successful ping promotes the peer back to CONNECTED.

2. Peer discovery (loud)

Every ~5 seconds, the node picks one random peer from memory (states DISCONNECTED or CONNECTED) and requests GET /peer/list to learn new addresses.

If that single random pick times out, the console shows:

Discovering new peers failed. ECONNABORTED Request failed GET http://<peer>/peer/list

This error names only the randomly selected peer, not the full peer table. During recovery it often surfaces little-known cloud-hosted nodes that were discovered earlier and saved to the database. That does not mean the node ignores seed peers.

3. Blockchain sync (strict)

The loader sync path uses peers.list() with the default filter: CONNECTED peers only.

If no peer is currently CONNECTED with a usable height, sync finishes with:

Failed to find enough good peers

In that situation the node is not "disconnected from the network" in the sense of having no peer records. It simply has zero active peers suitable for block download.

Typical outage timeline

1. Network loss — HTTP requests to all peers start failing.
2. State degradation — previously CONNECTED peers become DISCONNECTED.
3. Sync stall — loader cannot select good peers; height stops advancing.
4. Misleading console output — discovery errors continue against random stale entries, while seed pings run quietly in the background.
5. Recovery — once at least one seed (or other known peer) answers a ping again, it returns to CONNECTED and sync resumes.

The gap between "internet is back" and "node is syncing again" can be several minutes (or longer if remote peers are still unreachable), because recovery depends on a successful round-trip to a peer that becomes CONNECTED, not merely on local connectivity.

Operator expectations

• Seeing discovery errors against unfamiliar addresses after an outage is normal and does not by itself indicate misconfiguration.
• Seed peers from config are still contacted; their ping failures are just not prominent in default logs.
• Failed to find enough good peers means no currently active peers, not that the peer table was wiped.
• Restarting the node reloads seeds and database peers, but recovery still requires at least one remote peer to respond.

Possible improvements (for discussion)

• Log seed ping failures at warn when no CONNECTED peers remain for longer than N minutes.
• Prefer seed peers (or recently working peers) in getFromRandomPeer instead of a uniform random pick.
• Retry all seed peers in parallel when sync reports Failed to find enough good peers.
• Reduce duplicate warn lines when async.retry exhausts all sync attempts.

Tags

nodes peer-discovery sync reliability operators network-outage