add graceful shutdown (#3435)

[kdorosh]

backport of #3435

Description

Add a set of helm options to support performing a "graceful shutdown" of envoy when gloo is sitting behind external loadbalancers that are outside of the Kubernetes context.

The main functionality is provided via a k8s preStop hook which will be called prior to the envoy pod being terminated. This hook will first call the envoy admin interface to explicitly begin failing healthchecks (that have been configured via the health check filter), and then sleep for a certain, customized amount of time.

This sleep period will allow for envoy to fail the healthchecks that the external loadbalancers are relying on, causing the envoy instance to be removed as a healthy endpoint, while still able to handle existing requests, etc.

Context

For example, when using an AWS ALB with target groups of envoy proxies running inside an EKS cluster, there is no way to remove/drain an envoy instance for the ALB without abruptly killing the instance and eventually letting the ALB figure this out via failing healthchecks.

This feature allows users to do the following:

• define a health check via the healthcheck filter
• associate the health check with the ALB target groups
• on termination request of the envoy pod, begin failing health checks
• sleep for the specified amount of time
• allow the ALB to see enough failing health checks to remove this envoy instance as a healthy endpoint (which prevents new requests/connections being sent)
• allow currently processed requests to finish

Checklist:

• I included a concise, user-facing changelog (for details, see https://github.com/solo-io/go-utils/tree/master/changelogutils) which references the issue that is resolved.
• If I updated APIs (our protos) or helm values, I ran make install-go-tools generated-code to ensure there will be no code diff
• I followed guidelines laid out in the Gloo contribution guide
• I opened a draft PR or added the work in progress label if my PR is not ready for review
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
  BOT NOTES:
  resolves Provide option for graceful shutdown #3308

[solo-changelog-bot]

Issues linked to changelog:
#3308

[Sodman]

👍

[kdorosh]

/kick mix of valid/invalid vs flake https://console.cloud.google.com/cloud-build/builds/fe50281c-beb6-456b-b2e3-34a0220754d0?project=907026730415