feat(kubernetes): add livenessProbe

[maggieneterval]

Per discussion on spinnaker/spinnaker#4023, reuses readinessProbe as a livenessProbe so that containers that fail health checks are not only removed from the load balancer, but also restarted automatically.

[maggieneterval]

Just discussed with @ezimanyi / @dibyom and one potential issue here is that a service's readinessProbe may depend on downstream services being ready. This means we would want a different livenessProbe. Wondering if a simple TCP health check on the container's port would work for this?

cc @davidxia, let us know if this kind of livenessProbe would have prevented the issue you described!

[dibyom]

I like the idea of adding a TCP health check for the liveness probe!

[davidxia]

@maggieneterval This PR's current changes looks like it'll make every spinnaker component's readinessProbe a livenessProbe as well?

The clouddriver readinessProbe is this right now.

    readinessProbe:
      exec:
        command:
        - wget
        - --no-check-certificate
        - --spider
        - -q
        - http://localhost:7002/health

I think a wget for a livenessProbe would've restarted clouddriver in my case of spinnaker/spinnaker#4023. But I don't know if a TCP probe would've restarted it.

[dibyom]

I think a TCP health check should have worked in the spinnaker/spinnaker#4023 case as well since wget was failing to establish a connection to clouddriver:
Connecting to localhost|::1|:7002...

[maggieneterval]

Thanks for looking into that @dibyom! It sounds like at least starting with a TCP health check for the livenessProbe is a good idea; I will update this PR later today.

[maggieneterval]

@dibyom @ezimanyi should be ready for another look!

[ezimanyi]

Looks good!