application controller needs a liveness probe

[jessesuen]

We hit a situation where application controller could not talk to the K8s API server (which it was running on), failing with network unreachable. A application controller pod deletion had fixed the issue.

We currently have a readiness probe which simply gets the kubernetes version which the controller is running on, but we do not use this as a liveness probe. If we used this same check for liveness, then kubernetes would have killed our container, and we might have automatically recovered from the failure to talk to the K8s api server.

I say might because I'm not sure if container restart would have fixed it, or if a pod deletion was necessary to fix the problem.