feat(atenet): graceful termination of atenet-router

[shrutiyam-glitch]

Which issue(s) this PR is related to:

Fixes #721
Required for System Upgrade flow (#473)

What this PR does / why we need it:

This PR implements graceful termination and zero-outage rolling updates for atenet-router , building on the readiness probe and graceful drain patterns established in #719 (atelet).

Draining a two-container networking pod (atenet-router Go control-plane + envoy C++ proxy dataplane) introduces complex lifecycle interdependencies. This PR resolves the dual-container SIGTERM race, respects Envoy's failClosed ext_proc filter dependency, preserves parked requests riding out worker pool saturation, and accelerates idle deployments via an event-driven file handshake.

1. Event-Driven Dataplane Synchronization (emptyDir Marker File)

Envoy fast-exits by default on SIGTERM. To keep Envoy alive while the Go control-plane orchestrates the drain, we configured an IPC handshake between containers via a pod-shared emptyDir volume mounted at /var/run/atenet:

• Go Router Container: Removes any stale marker at startup, then writes /var/run/atenet/drain-complete when its shutdown sequence finishes.
• Envoy Container preStop Hook: Runs while [ ! -f /var/run/atenet/drain-complete ]; do sleep 0.5; done.

Outcome: Envoy exits as soon as — the drain is done, rather than wasting time in a fixed worst-case sleep. If the router crashes, Kubelet terminates the preStop hook at terminationGracePeriodSeconds: 60—slower cleanup, never a wedge.

2. The Multi-Container Shutdown Sequence (drain.go)

Because Kubernetes issues SIGTERM to both containers at once, a coordination state machine ensures Envoy never drops a connection and ext_proc is never stopped prematurely:

Phase / (best-case ex.) Timeline	atenet-router	envoy	K8s / Service Status
SIGTERM Sent ($t=0\text{s}$)	Catches SIGTERM • Flips /readyz $\rightarrow$ 503. • Starts 13s drain-delay.	Enters lifecycle.preStop hook: while [ ! -f .../drain-complete ]; do sleep 0.5; done • Kubelet holds SIGTERM back.	Deployment will recreate the pod. EndpointSlice controller begins dropping Old Pod IP.
Propagation ($t=0\text{s}$ – $t=13\text{s}$)	Keeps serving normally. • ext_proc continues unparking/routing requests.	Runs normally inside preStop loop. • Serves active TCP connections.	Service endpoint removal completes. No new connections arrive at Old Pod.
Envoy drain ($t=13\text{s}$)	Issues Envoy admin API calls: • /healthcheck/fail • /drain_listeners?graceful&skip_exit • Polls /stats for active downstream connections.	Begins graceful listener drain: • GOAWAY / Connection: close on established connections. • In-flight requests keep running.	In-flight HTTP requests finish executing through Envoy.
ext_proc drain ($t\approx18\text{s}$)	Active Envoy connections hit 0 (the poll exits early). • Calls extproc.GracefulStop(). • Parked request streams finish.	All downstream connections closed. • Still waiting in the preStop loop.	All client HTTP responses delivered.
Handshake & Exit ($t=18.5\text{s}$)	ext_proc drain finishes. • Writes drain-complete marker. • Hard-stops xDS (Stop()) & exits.	preStop loop detects marker file! • preStop exits 0. • Kubelet sends SIGTERM to Envoy. • Envoy exits immediately.	Old Pod deleted cleanly at ~18.5s (well under the 60s budget).

Envoy ref: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining

Testing & Verification

1. Unit Tests (drain_test.go): Added comprehensive unit tests covering:

• Graceful completion of in-flight ext_proc streams within deadline.
• Force-stopping straggler streams past --drain-timeout.
• Envoy admin API interaction and connection polling.
• Stale marker cleanup and marker file creation.

2. Local testing on kind cluster (Observations):
   Test script: hack/verify-atenet-drain.sh

• Steady state: /readyz=200, /healthz=200; the instant the pod turned Terminating: /readyz=503 while /healthz=200 — NotReady but alive, for the whole drain.
• The parked request survived the shutdown: fired at a busy 1-worker pool → parked → pod deleted while parked → worker freed → HTTP=200, total=1.89s, body hello from: 169.254.17.2 | preserved memory count: 1 |
  preserved file counter: 1 — park → resume → route, served by the Terminating pod during its drain-delay window.
• Pod terminated 14s after deletion — the idle floor, confirmed across three runs: >13s drain-delay (sequence ran), ≪60s grace (marker handshake released Envoy's preStop; no SIGKILL).
• Log sequence captured verbatim, drain-delay honored to the millisecond (17:34:41.706 → 17:34:54.707):

Shutdown signal received; draining
(+13.000s) Draining Envoy {window: 15s}
Envoy drained
Starting ext_proc drain
ext_proc drain completed within deadline
Drain-complete marker written {path: /var/run/atenet/drain-complete}
Shutdown complete

• Tests pass
• Appropriate changes to documentation are included in the PR

[shrutiyam-glitch]

/retest

[Lior Lieberman (LiorLieberman)]

Thanks shrutiyam-glitch !

[Lior Lieberman (LiorLieberman)]

lets add a cmoment and open an issue to add metrics here. Would be good to have a sense of how many unclean shutdowns we had

[Lior Lieberman (LiorLieberman)]

what is "none to drive"

suggest rewording to be more clear.

Also a function as in the parmas is not great. can we have something like "withDrainFunc" that decorates it or something?

[Lior Lieberman (LiorLieberman)]

Can we remove Envoy and agentgateway from the names here and everywhere in the PR? You can have drainDataplaneFunc (or drainDataplane) or something and just say that when set to nil, it simply means the dataplane doesn't require active graceful draining (or manages its own lifecycle). This keeps the generic router orchestrator cleanly decoupled from the underlying proxy implementation.