Defaulting/Mutating webhooks may have stale configurations when draining.

[dprotaso]

We have a test in serving that's flaky. knative/serving#16421

The test toggles a config map setting and then creates a service assuming the configuration changes have propagated to all components (webhooks and controllers)

The test started failing recently and the root cause is when a webhook is draining we are no longer receiving config map changes. Then when the service is created the the draining webhook handles the defaulting. Since it has the outdated config it fails creating the service.

Although the draining webhook shouldn't be receiving the request (we wait 30 seconds for the propagation) there's no guarantee so we should ensure our config map informers are still active during draining.

/assign @dprotaso

[dprotaso]

One thing that is confusing is the timing of the request in relation to the readiness probe in all the failure cases

We have about 36 seconds after the draining starts and readiness probes failing

2026-03-15T18:59:38.938Z - Readiness probes start failing
2026-03-15T19:00:14.402Z - CREATE Service - fails

But based on our configuration

        readinessProbe:
          periodSeconds: 1
          httpGet:
            scheme: HTTPS
            port: 8443

Pod readiness should fail after 3 seconds. (periodSeconds (1) * failureThreshold (default 3) = 3). So that bit is confusing

[dprotaso]

Seems like it's an API Server issue: kubernetes/kubernetes#127335 (comment)