Skip to content

[CI-1951] fix(podiprecovery): add Pod watch so recovery is level-triggered - #4987

Merged
coutinhop merged 6 commits into
tigera:masterfrom
coutinhop:pedro-CI-1951-2
Jul 27, 2026
Merged

[CI-1951] fix(podiprecovery): add Pod watch so recovery is level-triggered#4987
coutinhop merged 6 commits into
tigera:masterfrom
coutinhop:pedro-CI-1951-2

Conversation

@coutinhop

@coutinhop coutinhop commented Jul 3, 2026

Copy link
Copy Markdown
Member

Follow-up to #4784. Two fixes to the podiprecovery controller.

1. Recovery was edge-triggered (add a Pod watch)

The Node watch alone fired only when a node's host IPs changed. On a KubeVirt
reboot, the node's new IP is reported while its host-networked pods are still
restarting (empty status.podIPs), so the reconcile skips them; when a
surviving pod later comes back with its old, now-stale IP, the node IP has
already settled and no further Node event fires — so the stale pod is never
re-checked.

Add a second watch on operator-managed host-networked Pods that re-enqueues a
pod's node when it settles (IPs appear/change, or it becomes Ready). Both
watches feed the same idempotent, node-keyed Reconcile, making recovery
level-triggered on both inputs (node addresses + pod IPs) while staying
event-driven. The predicate is gated on the host-networked label +
spec.hostNetwork.

2. Don't force a cluster-wide Pod cache (use a scoped cache)

As written, #4784 registered a Pod.spec.nodeName field index in
cmd/main.go and read pods via the shared cache — either forces a
cluster-wide Pod informer holding every pod in memory, always-on from startup.
That's a sizable memory cost at scale and a regression (the operator's other
pod readers are conditional, so the shared Pod informer was previously
lazy/absent).

Give the controller its own cache.New scoped server-side to the
operator.tigera.io/host-networked label; the watch, the per-node List, and
the spec.nodeName index all live on it, so only the handful of
host-networked pods are cached and the shared cache is untouched. The
cmd/main.go field index is removed (supersedes that part of #4784). The
scoped reader is wrapped in a hostNetworkedPodLister exposing only
"list host-networked pods on a node", so it can't be misused as a full-pod
reader (which would silently drop non-host-networked pods).

Changes

  • podiprecovery: Pod watch + podToNode + hostNetPodSettledPredicate;
    dedicated label-scoped cache.New (with spec.nodeName index), wrapped in
    hostNetworkedPodLister.
  • ctrlruntime: add WatchObjectInCache (watch via a supplied cache).
  • cmd/main.go: remove the shared-cache Pod.spec.nodeName index from [CORE-12452] feat: auto-recover host-networked pods when node IP changes #4784.
  • update mockController in utils/egressgateway tests for the new method.

Testing

Unit tests for the pod-settle predicate and podToNode mapping;
go build/go vet ./... clean; podiprecovery/utils/egressgateway tests pass.

Release Note

None

For PR author

- [x] Tests for change.
- [ ] If changing pkg/apis/, run make gen-files
- [ ] If changing versions, run make gen-versions

For PR reviewers

- [ ] Milestone set according to targeted release.
- [ ] Appropriate labels:
  - [x] kind/bug
  - [ ] kind/enhancement
  - [ ] enterprise

Comment thread pkg/controller/podiprecovery/podiprecovery_controller.go Outdated
Comment thread pkg/controller/podiprecovery/podiprecovery_controller.go
…gered

Recovery was edge-triggered on the Node watch alone: the reconcile
fired only when a node's host IPs changed. On a KubeVirt VM reboot
the node's new IP is reported promptly, but the node's host-networked
pods are still restarting at that instant with empty status.podIPs,
so the reconcile skips them. Seconds later a surviving pod comes back
reporting its old, now-stale IP (Kubernetes never refreshes
status.podIPs for a surviving hostNetwork pod) — but the node's host
IP has already settled, no further Node event fires, and the stale
pod is never re-evaluated. The earlier autoscaler-tick approach did
not have this gap because it re-checked on every tick.

Add a second watch on operator-managed host-networked Pods that
re-enqueues a pod's node when the pod settles into a state where its
status.podIPs can be judged — its IPs appear/change, or it becomes
Ready. Both watches funnel into the same node-keyed, idempotent
Reconcile, so recovery is now level-triggered on both inputs to its
decision (node addresses and pod IPs) while staying event-driven —
no return to polling.

The predicate is gated on the host-networked marker label plus
spec.hostNetwork so event volume stays to the handful of such pods
cluster-wide.

Adds unit tests for the pod-settle predicate (create/update/delete,
label and hostNetwork gating, IPs-appear and became-Ready
transitions, steady-state no-op) and the podToNode mapping.)
The Pod watch and the per-node pod List previously went through the
manager's shared cache, and tigera#4784 registered a spec.nodeName field
index on Pod in cmd/main.go. Either of those forces controller-runtime
to start a cluster-wide Pod informer eagerly at startup, holding every
pod object in memory — a sizable regression at scale, since the
operator's other (pre-existing) pod readers are all conditional and
otherwise leave the pod informer lazy/unstarted.

Give podiprecovery its own cache.New scoped server-side to the
operator.tigera.io/host-networked label, register the spec.nodeName
index on that cache, and run it via mgr.Add. The Pod watch and the
per-node List now read only the handful of host-networked pods
cluster-wide; the shared cache is no longer forced to watch all pods.
Node reads, the Installation gate, and pod deletes still use the shared
client.

Guard against the scoped-cache footgun (silently reading a filtered
subset while expecting all pods): the scoped reader is wrapped in a
hostNetworkedPodLister that exposes only onNode(node) — there is no way
to Get/List arbitrary pods through it.

- ctrlruntime: add WatchObjectInCache to watch a type via a supplied
  cache instead of the manager's shared cache.
- cmd/main.go: drop the shared-cache Pod spec.nodeName field index
  (and its now-unused corev1 import); the index lives on the scoped
  cache now.
- update mockController in utils/egressgateway tests for the new
  interface method.

Tests: 33/33 podiprecovery specs pass; go build/go vet ./... clean.
@coutinhop coutinhop removed the hold merge Do not merge label Jul 27, 2026
@coutinhop
coutinhop merged commit b661e5e into tigera:master Jul 27, 2026
7 checks passed
@coutinhop
coutinhop deleted the pedro-CI-1951-2 branch July 27, 2026 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants