Fix for health check stability issues

[markmandel]

Because of race conditions, on non-warm CI builds, health checks can occasionally fail.

This should solve this problem 🤞

[markmandel]

@dzlier-gcp I think this will solve the issue with the healthchecks we've been seeing.

Also figured you might be interested - it's also a nice little util lib backed into K8s's client-go.

[dzlier-gcp]

LGTM, though any time I see duplicated code like this (which is partially my fault since I did 1.5 of the health checks) it makes me itch. I wonder if, when, and in what way we could consolidate any of this.

[markmandel]

I tend to work on a 'if you do something 3 times, then consolidate' - but if you feel strongly, happy to look at PRs ;)