Inject pod readiness gate

[solmonk]

In addition to supporting pod readiness gate conditions, ideally we need to "inject" readinessGates field via webhook.

We have to make sure only the desired resources are impacted. aws-load-balancer-controller defines a label to address this problem - when a pod is created under the labelled namespace, the controller webhook will update the pod definition with readinessGates field. We can refer to the documentation and take the similar approach.

If the user needs more fine-grained control than namespaces, they may still define readinessGates field manually on pods / pod templates instead.

Requires #560 - but can be developed independently.

[erikfuller]

Adding some notes here as automatically injecting the readiness gate will require creation of a kubernetes secret if installying with deploy.yaml. Helm has functionality to provision these certificates and secrets, so that should all be automatically configured.

Plan is to use the following for manually provisioning the certificate and configuring the webhook:

CERT_FILE=tls.crt
KEY_FILE=tls.key

WEBHOOK_SVC_NAME=webhook-service
WEBHOOK_NAME=aws-appnet-gwc-mutating-webhook
WEBHOOK_NAMESPACE=aws-application-networking-system
WEBHOOK_SECRET_NAME=webhook-cert

# Step 1: generate a certificate if needed, can also be provisioned through orgnanizational PKI, etc
# This cert includes a 100 year expiry
HOST=${WEBHOOK_SVC_NAME}.${WEBHOOK_NAMESPACE}.svc
openssl req -x509 -nodes -days 36500 -newkey rsa:2048 -keyout ${KEY_FILE} -out ${CERT_FILE} -subj "/CN=${HOST}/O=${HOST}" \
   -addext "subjectAltName = DNS:${HOST}, DNS:${HOST}.cluster.local"
   
# Step 2: add the secret - can be done so long as the namespace exists
# note that running "kubectl delete -f deploy.yaml" will remove the controller namespace AND this secret.
kubectl create secret tls $WEBHOOK_SECRET_NAME --namespace $WEBHOOK_NAMESPACE --cert=${CERT_FILE} --key=${KEY_FILE}

# Step 3: after applying deploy.yaml, patch the webhook CA bundle to exactly the cert being used
# this will ensure Kubernetes API server is able to trust the certificate presented by the webhook
CERT_B64=$(cat tls.crt | base64)
kubectl patch mutatingwebhookconfigurations.admissionregistration.k8s.io $WEBHOOK_NAME \
    --namespace $WEBHOOK_NAMESPACE --type='json' \
    -p="[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value': '${CERT_B64}'}]"

[erikfuller]

This functionality is now available when deploying via deploy.yaml. I still need to follow up with

1. Helm chart update to support the webhook and configure the TLS certificate
2. Update to build scripts to automatically run webhook e2e tests

[zijun726911]

Done by #605 #608 #612