feat: add Admin API TLS client verification support

[czeslavo]

What this PR does / why we need it:

To make the connection between the controller and the gateways secure, we should allow configuring client verification.

Which issue this PR fixes

Part of Kong/kubernetes-ingress-controller#3833.

Special notes for your reviewer:

I tried to follow a similar pattern that we use for webhook certs.

The modified examples were tested locally. I also verified that providing the caBundle as a blob works as well.

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

• PR is based off the current tip of the main branch.
• Changes are documented under the "Unreleased" header in CHANGELOG.md
• New or modified sections of values.yaml are documented in the README.md
• Commits follow the Kong commit message guidelines

[rainest]

Do you have logs from the test available? Those are useful for review if it's something like this that requires manual testing.

Some minor changes to the docs , mostly to cover the default behavior.

[czeslavo]

@rainest I ran two tests scenarios, please find the logs/procedure below.

Defaults - no certs provided by the users

# Gateway's values.yaml

admin:
  enabled: true
  type: ClusterIP
  clusterIP: None
  tls:
    client:
      secretName: "admin-api-ca-cert"

ingressController:
  enabled: false

replicaCount: 3

# Controller's values.yaml

deployment:
  kong:
    enabled: false

proxy:
  ## This must match the gateway release's proxy Service name.
  ## The Service name uses the pattern "<release name>-kong-proxy".
  ## In this example, the companion gateway release is named "gw"
  nameOverride: gw-kong-proxy

ingressController:
  enabled: true

  gatewayDiscovery:
    enabled: true
    adminApiService:
      ## This must match the gateway release's admin Service name.
      ## The Service name uses the pattern "<release name>-kong-admin".
      ## In this example, the companion gateway release is named "gw"
      name: gw-kong-admin

  adminApi:
    tls:
      client:
        # Enable TLS client authentication for the Admin API.
        enabled: true
        # We're specifying the name of the secret to have a static name that we
        # will use in the gateway release.
        caSecretName: "admin-api-ca-cert"

  env:
    # This must match the gateway release's proxy Service HTTPs port name.
    kong_admin_svc_port_names: "kong-admin-tls"

# Controller release installation 
$ helm upgrade --install controller . -n kong --version 2.17.0 --values ./example-values/minimal-kong-gd-controller.yaml

# Secrets get created
$ k get secrets -n kong 
 NAME                                                         TYPE                 DATA   AGE
admin-api-ca-cert                                            kubernetes.io/tls    2      34s
controller-kong-admin-api-keypair                            kubernetes.io/tls    2      34s

# Controller deployment waits for Gateways to be discovered
$ k get deployments -n kong     
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
controller-kong   0/1     1            0           2m3s

# Gateway release installation
$ helm upgrade --install gw . -n kong --version 2.17.0 --values ./example-values/minimal-kong-gd-gateway.yaml

# Gateways and Controller become ready
$ k get deployments -n kong                                                                                  
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
controller-kong   1/1     1            1           3m22s
gw-kong           3/3     3            3           33s

# Controller synced config with all Gateways
$ k logs controller-kong-5c5d6f697d-8vpbc -n kong 
time="2023-04-07T12:31:11Z" level=info msg="successfully synced configuration to kong" kong_url="https://10.244.0.14:8444"
time="2023-04-07T12:31:11Z" level=info msg="successfully synced configuration to kong" kong_url="https://10.244.0.13:8444"
time="2023-04-07T12:31:11Z" level=info msg="successfully synced configuration to kong" kong_url="https://10.244.0.15:8444"

# Port forward the same port that KIC us
$ kubectl port-forward -n kong deploy/gw-kong 8444:8444

# Calling the endpoint without certs gives 400
$ http --verify=no https://localhost:8444   
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 202
Content-Type: text/html; charset=UTF-8
Date: Fri, 07 Apr 2023 12:35:58 GMT

<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
</body>
</html>

# Calling the endpoint with certs gives 200
$ k get secret -n kong controller-kong-admin-api-keypair -o json | jq -r '.data."tls.crt"' | base64 -d > tls.crt
$ k get secret -n kong controller-kong-admin-api-keypair -o json | jq -r '.data."tls.key"' | base64 -d > tls.key
$ http --verify=no https://localhost:8444 --cert tls.crt --cert-key tls.key
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 13061
Content-Type: application/json; charset=utf-8
Date: Fri, 07 Apr 2023 12:40:53 GMT
Server: kong/3.2.2
X-Kong-Admin-Latency: 322

<body>

Certs provided by the user

# Create a self-signed CA
$ openssl req -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout rootCA.key -out rootCA.crt

# Create client's key
$ openssl genrsa -out tls.key 4096

# Generate CSR 
$ openssl req -new -key tls.key -out csr.pem 

# Sign the CSR with CA
$ openssl x509 -req -days 9999 -in csr.pem -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out tls.crt 

# Verify client cert with CA
$ openssl verify -CAfile rootCA.crt tls.crt        
tls.crt: OK 

# Create a secret for Controller
$ k create secret tls --key tls.key --cert tls.crt -n kong admin-api-client-tls-keypair
secret/admin-api-client-tls-keypair created 

# Controller's values.yaml

deployment:
  kong:
    enabled: false

proxy:
  ## This must match the gateway release's proxy Service name.
  ## The Service name uses the pattern "<release name>-kong-proxy".
  ## In this example, the companion gateway release is named "gw"
  nameOverride: gw-kong-proxy

ingressController:
  enabled: true

  gatewayDiscovery:
    enabled: true
    adminApiService:
      ## This must match the gateway release's admin Service name.
      ## The Service name uses the pattern "<release name>-kong-admin".
      ## In this example, the companion gateway release is named "gw"
      name: gw-kong-admin

  adminApi:
    tls:
      client:
        # Enable TLS client authentication for the Admin API.
        enabled: true

        # We're providing client certificates in the secret below.
        certProvided: true
        secretName: "admin-api-client-tls-keypair"

  env:
    # This must match the gateway release's proxy Service HTTPs port name.
    kong_admin_svc_port_names: "kong-admin-tls"

# Install Controller
$ helm upgrade --install gw . -n kong --version 2.17.0 --values ./example-values/minimal-kong-gd-gateway.yaml

# Gateway's values.yaml
admin:
  enabled: true
  type: ClusterIP
  clusterIP: None
  tls:
    client:
      # Copy-pasted rootCA.crt contents
      caBundle: |
        -----BEGIN CERTIFICATE-----
        MIICpjCCAY4CCQDDyWXN1NMsSzANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDDApr
        b25naHEuY29tMB4XDTIzMDQwNzEyNDUxM1oXDTI4MDQwNTEyNDUxM1owFTETMBEG
        A1UEAwwKa29uZ2hxLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
        AN1Vbedqbq0v4BVPeuKKnk1hDYNpar3ApoLALm3acO71WdZStDcvDTzd+zvRJeqo
        4YFZYFptdfuDaP2PKMeVmWVeKTFntV7n8zkMfx0heQb6kdO3QwtPMwk+krDRTSxI
        3dR6OIitqUYamsqnlu/1vBpNg2xeJR5yLwPpnEoqemJMi42VpOmoz7E3u1slrPGz
        IJsiEizq1dhP8k9VKU6V3QUSkOi9zEfjGVKryjjTP1CdJFHsbopgZEsIDdURvig6
        uC9DTrxMIRVii4k1Fe1BX224nKNPy+XSUO9ORVP/hPwrQBwr8UTxZgxZ6/Hd0Zf1
        RQJRuD+Q35Wk6ivySaWrickCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAmG3vrY2H
        iGhoK0xZBJORECWQ8zdFSn7JV/bbfC+Dc99HW7VMSNkGwUTY5czz7iEfwuJn+rvX
        2mr/hqBROlor+DsA+V1Qcxt9jpWPlFnloRsayMnarVAzEPXut7nUjvH+p7D+3g7p
        Tl6KRkycYSGSE/dPE2V+/AHLtFQTVhhKzR4nd1ktKFa6hf2iZwfpJnVi/lOr2YFk
        Fm20qz93sMVmB/vVquGsbUAPYixc8bHLENnDznYweSUvg9CFIz9cPW9FbOI80WtI
        Uu1xxtJAQjAB2GyD6QlvM6gLAMu7LdP9ht94xi/0UtkOghumrrdqyEw6qoiOojAJ
        7xxqjJgL8WOvGg==
        -----END CERTIFICATE-----

ingressController:
  enabled: false

replicaCount: 3

# Install Gateway
$ helm upgrade --install gw . -n kong --version 2.17.0 --values ./example-values/minimal-kong-gd-gateway.yaml

# Both deployments become ready
$ k get deployments -n kong                            
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
controller-kong   1/1     1            1           6m5s
gw-kong           3/3     3            3           2m59s

# Controller synced config with all Gateways
$ k logs -n kong controller-kong-544f5bcb75-bsgp5
time="2023-04-07T13:08:39Z" level=info msg="successfully synced configuration to kong" kong_url="https://10.244.0.19:8444"
time="2023-04-07T13:08:39Z" level=info msg="successfully synced configuration to kong" kong_url="https://10.244.0.18:8444"
time="2023-04-07T13:08:39Z" level=info msg="successfully synced configuration to kong" kong_url="https://10.244.0.17:8444"

# Port forward the same port that KIC uses
$ kubectl port-forward -n kong deploy/gw-kong 8444:8444

# Calling the endpoint without certs gives 400
$ http --verify=no https://localhost:8444                                  
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 202
Content-Type: text/html; charset=UTF-8
Date: Fri, 07 Apr 2023 13:09:54 GMT

<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
</body>
</html>

# Calling the endpoint with certs gives 200
$ http --verify=no https://localhost:8444 --cert tls.crt --cert-key tls.key
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 13061
Content-Type: application/json; charset=utf-8
Date: Fri, 07 Apr 2023 13:10:21 GMT
Server: kong/3.2.2
X-Kong-Admin-Latency: 15

<body>