Liveness probe is not working

[alewitt2]

our razeedeploy resources are getting into a bad state, and basically stalling out. What we expect to happen, is the liveness custom script will fail, and kubernetes will see that and restart the pod. we have seen the script works as expected, but kube isnt restarting the pod.

We either need to investigate if our script is not returning the correct error response, or look into setting up a new liveness check that kube will respect.

example liveness probe definition https://github.com/razee-io/RemoteResource/blob/master/kubernetes/RemoteResource/resource.yaml#L55-L58

example liveness script that should force kube to restart us on script failing https://github.com/razee-io/RemoteResource/blob/master/sh/liveness.sh#L18-L31

[alewitt2]

1. setup minikube or kubernetes on docker desktop setup.
2. build docker image
   1. runs a shell script that touches file once
   2. sleeps 2hr
3. put that docker image into a deployment yaml, with a liveness script defined
4. run that deployment on your local test cluster and see if it works as we think it should