Feature: Add --term-delay option to CLI

[donna-156]

What this PR does / why we need it:
This PR adds a --term-delay option to the CLI for the ingress-controller.

Context:
There are cases where due to network load balancer (NLB) configuration that a delay needs to be introduced to the containers, this delay is configurable for the Kong Proxy container through the helm chart, however it is not possible to configure this delay for the ingress-controller. This can result in stale upstreams during NLB draining and downtime for services. Making this delay configurable seems the easiest approach to addressing this limitation.

With the ingress-controller gone, any changes to the underlying services that the proxy is routing to would be missed. Due to limitations with AWS NLB deregistration delays, we've had to configure a sleep of up to 4 minutes for the proxy container before shutting down. This currently means when we are scaling down, or rolling Kong pods there is a 3-4 minute window where the proxy container is running with no ingress-controller to monitor for updates with our underlying services.

Impact: This can result in stale upstreams, and errors for clients during this window. 😞

This PR makes multiple changes:

1. Adds a new --term-delay argument to the CLI
   This makes it possible for users to configure the SIGTERM delay value by passing a CLI argument, or an environment variable. The default is set to zero so it won't impact existing users.

2. Adjusts where the SignalHandler is called:
   Originally this was in the entry point to the controller manager, however in order to pass through the CLI argument and make the term-delay configurable, this needed moved down into the rootcmd package.

3. Supports adding a delay to the SignalHandler
   This takes the original SetupSignalHandler, and modifies the logic to support setting a time delay. It should still support overriding that time-delay by sending a second signal. This involved copying and altering the existing k8s controller-runtime signal package.

Special notes for your reviewer:

Alternative option considered:

• Add a lifecycle value for the ingress-controller container to kong/charts so that we can pass a preStop command similar to the proxy container. Naming this would be hard because the existing value is kong.lifecycle rather than kong.proxy.lifecycle. It would also require a sleep binary in the proxy container which doesn’t currently exist from the distroless/static base image - we could mount one from an initContainer but it would be fiddly.

Testing
I've tested this locally, running with the following commands:

go run -tags gcp ./internal/cmd/main.go \
--kubeconfig ~/.kube/config \
--publish-service=kong/kong-proxy \
--apiserver-host=http://localhost:8002 \
--kong-admin-url https://localhost:8444 \
--kong-admin-tls-skip-verify blah \
--term-delay 10s

From this I've been able to exercise trying to CTRL + C to kill the process, and validated that the delay is working as expected. I've also validated that by sending a second signal it does exit with code 1.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[CLAassistant]

All committers have signed the CLA.

[shaneutt]

Thank you for your contribution, the PR is looking good!

I have a couple of comments for your considerations, let me know what you think.

[chazdnato]

Thank you for your contribution, the PR is looking good!

I have a couple of comments for your considerations, let me know what you think.

This is great stuff! Instead of doing it via GH, I'll add all these to my local branch and push a change. I'll also make sure the linter runs successfully locally first (I'm using gofmt, not sure why it's sad). Appreciate the quick and helpful feedback!

[shaneutt]

Just checking in?

[donna-156]

Hi @shaneutt, thanks for following up! We'll have the remaining updates available shortly, our team has had some vacation time which has resulted in a longer turnaround from us. 🙂

[chazdnato]

Just checking in?

Apologies for the delay! Wanted to ensure I had the necessary helm chart changes tested (a PR in that repo forthcoming). We can rebase / squash these commit as well if you'd like, just let me know your preference.

We believe we have this in the state that we'd like!

[shaneutt]

I like it, thank you for the contribution 👍

If you haven't done so before, please do feel free to grab yourself a free Kong contributor's t-shirt (each of you), we give these out once per contributor: https://github.com/Kong/kong/blob/master/CONTRIBUTING.md#contributor-t-shirt

Thank you for your contribution, we appreciate it 🖖

[donna-156]

Thank you so much!

Two more questions @shaneutt 🙂

1. Do we also need to modify the Kong docs repo? i.e. here: https://github.com/Kong/docs.konghq.com
2. What is the timeframe for this being available in a release? (both ingress-controller and Kong)

[shaneutt]

Thank you so much!

Two more questions @shaneutt slightly_smiling_face

1. Do we also need to modify the Kong docs repo? i.e. here: https://github.com/Kong/docs.konghq.com

Yes please.

2. What is the timeframe for this being available in a release? (both ingress-controller and Kong)

Most likely this is going into v2.4.0 which is expected to release in the next week or two.