Skip to content

atenet: two replicas + PDBs for router and dns - #704

Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:issue-645-dataplane-pdb
Open

atenet: two replicas + PDBs for router and dns#704
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:issue-645-dataplane-pdb

Conversation

@orangeCatDeveloper

@orangeCatDeveloper NekoPunch (orangeCatDeveloper) commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

The atenet-router and dns Deployments are single-replica SPOFs: every deploy, node drain, or eviction takes the whole data plane down (measured ~5.5 s of hard connection failures per router pod kill; 6,880 dial errors during one GKE node-pool event — see #645). This bumps both to 2 replicas with a surge-only rollout (maxUnavailable: 0), an Envoy readiness probe (admin /ready, which passes only after the initial xDS load from the router container), preStop drain sleeps on all three containers, soft hostname spread, and PodDisruptionBudgets.

preStop uses the native K8s ≥1.30 sleep action because the ko and coredns images are distroless (no sleep binary). The router container needs its own drain: it hosts ext_proc, and if it dies while the draining Envoy still receives traffic, requests fail with 500.

The multi-replica xDS question flagged in #645 is resolved by #699 (restart- and replica-unique version epochs); in standalone sidecar mode each Envoy only talks to its pod-local router, so replicas never share an xDS stream.

Out of scope: the managed-Envoy path (envoyrunner.go) stays single-replica — #635 removes that mode entirely.

Fixes #645

Test plan

On a kind cluster, with a 5 req/s in-cluster probe against a running actor:

  • rolling restart under traffic: 278/278 requests OK (single-replica baseline: ~5.5 s outage)
  • force-delete one router replica under traffic: 183/183 OK
  • dns rolling restart: actor domain still resolves to the router Service; no FailedPreStopHook events
  • kustomize strict render of both manifests passes

Both were single-replica SPOFs: every deploy or node drain took the
data plane down. Surge-only rollout, Envoy readiness gating, preStop
drain, soft hostname spread, and PDBs make disruptions non-events.

The managed-Envoy path (envoyrunner.go) stays single-replica; agent-substrate#635
removes that mode.
@maxsmythe

Copy link
Copy Markdown
Collaborator

This looks reasonable, but sending to Bowei Du (@bowei) as atenet expert

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/network kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1] atenet-router and dns run as single replicas with no PDB — every restart is a guaranteed full data-plane outage

2 participants