Add retry checker for DNS failure from Ingress

[nak3]

This patch fixes the recent flake in service_to_service_test.go on Kind/Istio.
istio-ingressgateway returns 502 with no such host error when it fails to resolve the cluster domain.

/cc @markusthoemmes

[nak3]

knative/serving#12294 runs this patch and seems error could be avoided.

[codecov]

Codecov Report

Merging #2351 (6bd2b4b) into main (0b0c339) will decrease coverage by 0.84%.
The diff coverage is 52.63%.

@@            Coverage Diff             @@
##             main    #2351      +/-   ##
==========================================
- Coverage   64.52%   63.67%   -0.85%     
==========================================
  Files         226      227       +1     
  Lines        9668     9814     +146     
==========================================
+ Hits         6238     6249      +11     
- Misses       3154     3288     +134     
- Partials      276      277       +1     

Impacted Files	Coverage Δ
injection/sharedmain/main.go	5.63% <50.00%> (ø)
test/spoof/spoof.go	54.54% <50.00%> (+1.15%)	⬆️
test/spoof/error_checks.go	85.71% <100.00%> (+1.09%)	⬆️
controller/controller.go	86.13% <0.00%> (-0.37%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d82be48...6bd2b4b. Read the comment docs.

[markusthoemmes]

/lgtm
/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: markusthoemmes, nak3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/OWNERS [markusthoemmes,nak3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dprotaso]

This patch fixes the recent flake in service_to_service_test.go on Kind/Istio.

I guess to clarify this isn't a flake as it consistently failed in Kind running on GitHub actions. Thus I wonder if the retry is masking some other issue.

The other concerns I have, and let me know if this is just paranoia, is that by including this error checker by default we could mask issues in other ingresses and wouldn't know about it

[dprotaso]

To be more specific this only failed on GitHub actions with kind 1.22 & net-istio

[nak3]

I was also concerned but it retries only for the no DNS record in the internal DNS temporary.
I think it is difficult for ingress/coredns to guarantee that *.svc.cluster.local record is registered as soon as k8s svc is created so we need to retry it 🤔

[dprotaso]

Then maybe as part of probing we should ensure that DNS works as expected.

K8s 1.22 introduces a newer CoreDNS version - ie. with endpoint slice support thus maybe istio isn't playing well with it

[markusthoemmes]

Sounds fair enough! Shall we track this and make sure we do our due diligence here?

[nak3]

Sure, opened knative/networking#575