Ephemeral zaino/zebra topology: dedicated zebras contend on per-IP peer cap — invest in readstate mode

[nachog00]

Context / history

Ephemeral zaino test instances need a backing zebra, and we've gone back and forth on topology:

• Started with a single shared golden zebra backing all ephemeral zainos.
• Got (correct) pushback: a shared backing isn't isolated enough for experiments — one zaino's
  test load/behavior can perturb the shared zebra. So we moved to a dedicated zebra per ephemeral
  zaino, which is the right isolation model.
• But dedicated-per-zaino collides with a Zcash P2P mechanism (below).

The problem: Zcash caps peer connections to 1 per source IP

Every ephemeral zebra runs behind the same NAT egress IP (the k8s node, tekau). Zcash nodes
cap peer connections to 1 per source IP (max_connections_per_ip, default 1) as an
anti-eclipse / DoS measure — and it's enforced by the remote peers, so it can't be configured
away on our side.

So once golden (or a first ephemeral) zebra holds the slot on the popular seeder peers from our IP,
a second zebra behind the same IP gets its handshakes dropped. Symptom: handshake_success_total=1,
can't sustain near-tip sync — while golden, on the same node, syncs fine on its established peers.

Confirmed root cause (code + docs + spec):

• zebra DEFAULT_MAX_CONNS_PER_IP = 1 (zebra-network/src/constants.rs), enforced in two places —
  pre-handshake RecentByIp rate limit (peer_set/initialize.rs) and post-handshake peer-set drop
  num_peers_with_ip(ip) >= max_conns_per_ip (peer_set/set.rs). Present/identical in all versions.
• zcashd matches (1/IP). The v2 P2P ZIP draft (Add draft ZIP: Version 2 Zcash P2P Network Protocol zcash/zips#1344) documents the eclipse-resistance
  rationale: "Bitcoin Core and zcashd make at most one outbound connection per group", address-book
  segmentation by IP range.

What still works

A dedicated zebra syncing from genesis has worked fine — full checkpoint sync completes even with
degraded peer diversity, as long as we don't need continuous tip-tracking. The contention bites
hardest for near-tip / snapshot-restore-then-follow, where golden is actively holding the good peers.
So "dedicated zebra, sync from genesis, don't care about staying at tip" remains a viable, if slow,
fallback.

Options

1. Distinct egress IP per ephemeral zebra (per-pod SNAT / dedicated tailscale exit node). Removes
   the collision, but fragile: zebra is moving toward per-subnet caps (IPv4 /24, IPv6 /64 —
   feat(network): add per-subnet connection caps for IPv6 /64 and IPv4 /24 ZcashFoundation/zebra#11256), so even adjacent IPs would contend.
2. Shared golden zebra (zaino tryout mode) — solves contention but is the isolation model we
   already rejected.
3. Invest in zaino readstate/state mode — preferred direction (below).

Preferred direction: readstate/state mode

Ephemerals currently run zaino in fetch mode (JSON-RPC to a dedicated zebrad). Readstate mode
puts ~no load on a separate zebra process — zaino reads the chain state directly. If readstate
zainos can back off a shared zebra state (one syncing/peering zebra, many isolated zaino indexes
reading it), we get per-experiment isolation and stop running N peering zebras behind one IP —
dissolving the per-IP problem entirely.

Recommendation: properly support and default ephemeral zainos to readstate/state mode.

Open question to resolve first: does zaino's readstate path spin up its own p2p chain syncer
(which would re-introduce the same per-IP contention), or can it read a shared / snapshot-restored
zebra state without peering? Determine the exact architecture, then design the ephemeral topology
around it.

Refs

• Session devlog devlog/2026-08-18-zaino-getblockchaininfo-boot-fix.md; zaino#1455 / zaino#1456.
• max_connections_per_ip research: ZcashFoundation/zebra #6980, #6993, #7013, #11256; Add draft ZIP: Version 2 Zcash P2P Network Protocol zcash/zips#1344.