[CI-1951] fix(podiprecovery): add Pod watch so recovery is level-triggered - #4987
Merged
Conversation
coutinhop
force-pushed
the
pedro-CI-1951-2
branch
from
July 20, 2026 23:18
183d36c to
8d2a6a3
Compare
…gered Recovery was edge-triggered on the Node watch alone: the reconcile fired only when a node's host IPs changed. On a KubeVirt VM reboot the node's new IP is reported promptly, but the node's host-networked pods are still restarting at that instant with empty status.podIPs, so the reconcile skips them. Seconds later a surviving pod comes back reporting its old, now-stale IP (Kubernetes never refreshes status.podIPs for a surviving hostNetwork pod) — but the node's host IP has already settled, no further Node event fires, and the stale pod is never re-evaluated. The earlier autoscaler-tick approach did not have this gap because it re-checked on every tick. Add a second watch on operator-managed host-networked Pods that re-enqueues a pod's node when the pod settles into a state where its status.podIPs can be judged — its IPs appear/change, or it becomes Ready. Both watches funnel into the same node-keyed, idempotent Reconcile, so recovery is now level-triggered on both inputs to its decision (node addresses and pod IPs) while staying event-driven — no return to polling. The predicate is gated on the host-networked marker label plus spec.hostNetwork so event volume stays to the handful of such pods cluster-wide. Adds unit tests for the pod-settle predicate (create/update/delete, label and hostNetwork gating, IPs-appear and became-Ready transitions, steady-state no-op) and the podToNode mapping.)
The Pod watch and the per-node pod List previously went through the manager's shared cache, and tigera#4784 registered a spec.nodeName field index on Pod in cmd/main.go. Either of those forces controller-runtime to start a cluster-wide Pod informer eagerly at startup, holding every pod object in memory — a sizable regression at scale, since the operator's other (pre-existing) pod readers are all conditional and otherwise leave the pod informer lazy/unstarted. Give podiprecovery its own cache.New scoped server-side to the operator.tigera.io/host-networked label, register the spec.nodeName index on that cache, and run it via mgr.Add. The Pod watch and the per-node List now read only the handful of host-networked pods cluster-wide; the shared cache is no longer forced to watch all pods. Node reads, the Installation gate, and pod deletes still use the shared client. Guard against the scoped-cache footgun (silently reading a filtered subset while expecting all pods): the scoped reader is wrapped in a hostNetworkedPodLister that exposes only onNode(node) — there is no way to Get/List arbitrary pods through it. - ctrlruntime: add WatchObjectInCache to watch a type via a supplied cache instead of the manager's shared cache. - cmd/main.go: drop the shared-cache Pod spec.nodeName field index (and its now-unused corev1 import); the index lives on the scoped cache now. - update mockController in utils/egressgateway tests for the new interface method. Tests: 33/33 podiprecovery specs pass; go build/go vet ./... clean.
coutinhop
force-pushed
the
pedro-CI-1951-2
branch
from
July 24, 2026 17:51
c15cf82 to
0ebca71
Compare
caseydavenport
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #4784. Two fixes to the podiprecovery controller.
1. Recovery was edge-triggered (add a Pod watch)
The Node watch alone fired only when a node's host IPs changed. On a KubeVirt
reboot, the node's new IP is reported while its host-networked pods are still
restarting (empty
status.podIPs), so the reconcile skips them; when asurviving pod later comes back with its old, now-stale IP, the node IP has
already settled and no further Node event fires — so the stale pod is never
re-checked.
Add a second watch on operator-managed host-networked Pods that re-enqueues a
pod's node when it settles (IPs appear/change, or it becomes Ready). Both
watches feed the same idempotent, node-keyed Reconcile, making recovery
level-triggered on both inputs (node addresses + pod IPs) while staying
event-driven. The predicate is gated on the host-networked label +
spec.hostNetwork.2. Don't force a cluster-wide Pod cache (use a scoped cache)
As written, #4784 registered a
Pod.spec.nodeNamefield index incmd/main.goand read pods via the shared cache — either forces acluster-wide Pod informer holding every pod in memory, always-on from startup.
That's a sizable memory cost at scale and a regression (the operator's other
pod readers are conditional, so the shared Pod informer was previously
lazy/absent).
Give the controller its own
cache.Newscoped server-side to theoperator.tigera.io/host-networkedlabel; the watch, the per-node List, andthe
spec.nodeNameindex all live on it, so only the handful ofhost-networked pods are cached and the shared cache is untouched. The
cmd/main.gofield index is removed (supersedes that part of #4784). Thescoped reader is wrapped in a
hostNetworkedPodListerexposing only"list host-networked pods on a node", so it can't be misused as a full-pod
reader (which would silently drop non-host-networked pods).
Changes
podiprecovery: Pod watch +podToNode+hostNetPodSettledPredicate;dedicated label-scoped
cache.New(withspec.nodeNameindex), wrapped inhostNetworkedPodLister.ctrlruntime: addWatchObjectInCache(watch via a supplied cache).cmd/main.go: remove the shared-cachePod.spec.nodeNameindex from [CORE-12452] feat: auto-recover host-networked pods when node IP changes #4784.mockControllerin utils/egressgateway tests for the new method.Testing
Unit tests for the pod-settle predicate and
podToNodemapping;go build/go vet ./...clean; podiprecovery/utils/egressgateway tests pass.Release Note