-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net-istio should guard the prober against resync flooding #262
Comments
/assign |
Yeah exactly. see: knative-extensions/net-contour#231 |
/assign @arturenault |
@JRBANCEL: GitHub didn't allow me to assign the following users: arturenault. Note that only knative-sandbox members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/assign |
Right now the prober is unconditionally called during reconciliation to determine readiness: https://github.com/knative-sandbox/net-istio/blob/81a9b95df6ba5516bb3240b53caaa1686fc2dd75/pkg/reconciler/ingress/ingress.go#L203
This is ok under normal circumstances because internally the prober caches the probe result and so on global resyncs we hit the cache and things finish quickly. However, when we resync due to failing over (perhaps due to rollout) this cache is empty.
If I have 1000 ksvc, they are exposed on both gateways, and those gateways have 5 pods each, then the prober must perform
5 * 2 * 1,000 = 10,000
probes before normal work may resume.In
net-contour
, I changed things to have the main Reconcile loop rely on the recorded readiness of thekingress
to elideIsReady
checks, effectively recording the probers internal cache in the CRD's durable status.cc @JRBANCEL @tcnghia @nak3 @ZhiminXiang
The text was updated successfully, but these errors were encountered: