net-istio should guard the prober against resync flooding

[mattmoor]

Right now the prober is unconditionally called during reconciliation to determine readiness: https://github.com/knative-sandbox/net-istio/blob/81a9b95df6ba5516bb3240b53caaa1686fc2dd75/pkg/reconciler/ingress/ingress.go#L203

This is ok under normal circumstances because internally the prober caches the probe result and so on global resyncs we hit the cache and things finish quickly. However, when we resync due to failing over (perhaps due to rollout) this cache is empty.

If I have 1000 ksvc, they are exposed on both gateways, and those gateways have 5 pods each, then the prober must perform 5 * 2 * 1,000 = 10,000 probes before normal work may resume.

In net-contour, I changed things to have the main Reconcile loop rely on the recorded readiness of the kingress to elide IsReady checks, effectively recording the probers internal cache in the CRD's durable status.

cc @JRBANCEL @tcnghia @nak3 @ZhiminXiang

[JRBANCEL]

have the main Reconcile loop rely on the recorded readiness of the kingress to elide IsReady checks.
Basically, calling IsReady only if KIngress.Status is not Ready?
That should work, since for KIngress to be Ready (and ObservedGeneration == Generation) probing must have succeeded at some point int the past.

/assign

[mattmoor]

Yeah exactly. see: knative-extensions/net-contour#231

[JRBANCEL]

/assign @arturenault

[knative-prow-robot]

@JRBANCEL: GitHub didn't allow me to assign the following users: arturenault.

Note that only knative-sandbox members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @arturenault

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[arturenault]

/assign