Router and Registry should have readiness checks and health checks

[smarterclayton]

Both should have those set up by default.

@ramr @pweil- @miminar

If we have them, then we should be using them in the registry / router generators. Both should not return true until the members are ready to be in rotation.

[miminar]

#5892 partially addresses this by adding liveness probe to docker-registry. I'll build on top of it to add readiness checks to both registry and route pods.

[miminar]

Blocked by #4339. No longer blocked. #5404 re-enabled liveness probe for the registry.

[ramr]

@smarterclayton we can do it by checking the router itself - aka check port 80/443. Currently, we have a liveness probe for haproxy and can add a readiness one which is same/similar.

However, the waiting for members to be ready in rotation will probably not work for the router case ... won't be correct to wait for multiple routes + multiple backends to all be available as some routes might be slow getting available aka the drools case and shouldn't affect the availability of other routes.
Note to self: need to check if there's something we can use for readiness/liveness checks for F5.

[ramr]

Ignore previous comment, didn't realize @miminar already had a PR for this - thanks @miminar

[miminar]

#6439 is a last piece providing readiness probe for registry.