Need way to notify Top-Level Load Balancer of shutdown during a upgrade on /healthz endpoint

[bsteve99]

The ingress needs to notify our load-balancer that sits above our k8 cluster on the /healthz endpoint that we are shutting down or upgrading for seamless/graceful upgrades.

Our Load Balancer sends to our configured pool of k8 worker nodes on a NodePort.

I was thinking about configuring a monitor-fail to do this, but somethings not right about it.

This was said on slack.

"but I see we can make some improvements on how haproxy itself notifies a listener, by e.g. removing the monitor uri during a grace period, between the removing of a pod and the pod being effectively removed."

[bsteve99]

I think if the controller itself had a /health endpoint; that would work also, that could return a 504 during graceful shutdown....

[gdziwoki]

We have decided to go with this preStop workaround:

      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - sed -i 's/healthz/terminating/g' /etc/haproxy/haproxy.cfg && ./haproxy-reload.sh reusesocket /etc/haproxy

sed changes the path for the healthcheck endpoint so it starts failing and the node with terminating pod is marked as Unhealthy on the TargetGroup