GKE HTTP01 infinite failure loop if creating dual IPv4 and IPv6 ingresses for same host at same time

[abevoelker]

Describe the bug:

When creating dual IPv4 and IPv6 Ingresses for the same host with the following manifest:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-ingress-ipv4
  annotations:
    kubernetes.io/ingress.global-static-ip-name: webapp-ipv4
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
spec:
  rules:
  - host: captioned-images.duckdns.org
    http:
      paths:
      - path: /*
        backend:
          serviceName: webapp
          servicePort: 3000
  tls:
  - secretName: webapp-tls
    hosts:
    - captioned-images.duckdns.org
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-ingress-ipv6
  annotations:
    kubernetes.io/ingress.global-static-ip-name: webapp-ipv6
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
spec:
  rules:
  - host: captioned-images.duckdns.org
    http:
      paths:
      - path: /*
        backend:
          serviceName: webapp
          servicePort: 3000
  tls:
  - secretName: webapp-tls
    hosts:
    - captioned-images.duckdns.org

cert-manager seems to first create an Ingress rule / UrlMap just for the IPv4 Ingress:

$ kubectl describe ing
Name:             webapp-ingress-ipv4
Namespace:        default
Address:          35.244.211.136
Default backend:  default-http-backend:80 (10.4.2.8:8080)
TLS:
  webapp-tls terminates captioned-images.duckdns.org
Rules:
  Host                          Path  Backends
  ----                          ----  --------
  captioned-images.duckdns.org  
                                /*                                                                        webapp:3000 (<none>)
                                /.well-known/acme-challenge/W50w7IcM1NSrDUlXNnF0c6_h86_e4GXHLT-uhSzvIkk   cm-acme-http-solver-r5x22:8089 (<none>)
Annotations:
  kubernetes.io/ingress.global-static-ip-name:       webapp-ipv4
  certmanager.k8s.io/acme-http01-edit-in-place:      true
  certmanager.k8s.io/cluster-issuer:                 letsencrypt-prod
  ingress.kubernetes.io/backends:                    {"k8s-be-30660--4285d79a798c4100":"HEALTHY","k8s-be-30760--4285d79a798c4100":"HEALTHY","k8s-be-31522--4285d79a798c4100":"Unknown"}
  ingress.kubernetes.io/forwarding-rule:             k8s-fw-default-webapp-ingress-ipv4--4285d79a798c4100
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-webapp-ingress-ipv4--4285d79a798c4100
  ingress.kubernetes.io/url-map:                     k8s-um-default-webapp-ingress-ipv4--4285d79a798c4100
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/acme-http01-edit-in-place":"true","certmanager.k8s.io/cluster-issuer":"letsencrypt-prod","kubernetes.io/ingress.global-static-ip-name":"webapp-ipv4"},"name":"webapp-ingress-ipv4","namespace":"default"},"spec":{"rules":[{"host":"captioned-images.duckdns.org","http":{"paths":[{"backend":{"serviceName":"webapp","servicePort":3000},"path":"/*"}]}}],"tls":[{"hosts":["captioned-images.duckdns.org"],"secretName":"webapp-tls"}]}}

Events:
  Type     Reason  Age               From                     Message
  ----     ------  ----              ----                     -------
  Warning  Sync    2m (x84 over 3h)  loadbalancer-controller  Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce


Name:             webapp-ingress-ipv6
Namespace:        default
Address:          2600:1901:0:ed32::
Default backend:  default-http-backend:80 (10.4.2.8:8080)
TLS:
  webapp-tls terminates captioned-images.duckdns.org
Rules:
  Host                          Path  Backends
  ----                          ----  --------
  captioned-images.duckdns.org  
                                /*   webapp:3000 (<none>)
Annotations:
  ingress.kubernetes.io/target-proxy:                k8s-tp-default-webapp-ingress-ipv6--4285d79a798c4100
  ingress.kubernetes.io/url-map:                     k8s-um-default-webapp-ingress-ipv6--4285d79a798c4100
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/acme-http01-edit-in-place":"true","certmanager.k8s.io/cluster-issuer":"letsencrypt-prod","kubernetes.io/ingress.global-static-ip-name":"webapp-ipv6"},"name":"webapp-ingress-ipv6","namespace":"default"},"spec":{"rules":[{"host":"captioned-images.duckdns.org","http":{"paths":[{"backend":{"serviceName":"webapp","servicePort":3000},"path":"/*"}]}}],"tls":[{"hosts":["captioned-images.duckdns.org"],"secretName":"webapp-tls"}]}}

  kubernetes.io/ingress.global-static-ip-name:   webapp-ipv6
  certmanager.k8s.io/acme-http01-edit-in-place:  true
  certmanager.k8s.io/cluster-issuer:             letsencrypt-prod
  ingress.kubernetes.io/backends:                {"k8s-be-30660--4285d79a798c4100":"HEALTHY","k8s-be-30760--4285d79a798c4100":"HEALTHY"}
  ingress.kubernetes.io/forwarding-rule:         k8s-fw-default-webapp-ingress-ipv6--4285d79a798c4100
Events:
  Type     Reason  Age               From                     Message
  ----     ------  ----              ----                     -------
  Warning  Sync    6m (x22 over 3h)  loadbalancer-controller  Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce

After the GCLB path / UrlMap is online enough for the 5 successful requests happen:

The challenge gets passed to Let's Encrypt; but unfortunately the Let's Encrypt request comes in on IPv6, which doesn't have the Ingress path / cm-acme-http-solver running, so it gets passed to my internal application and 404s (note request IP 2600:1901:0:ed32::):

Then the challenge process is failed (acme: authorization for identifier captioned-images.duckdns.org is invalid), the process repeats with a new challenge/path, then fails again, until Let's Encrypt rate limits me, ad infinitum.

Expected behaviour:
I expect cert-manager to be amazing and successfully create certificates for both Ingresses 😄

Steps to reproduce the bug:
I'm following the instructions at https://github.com/ahmetb/gke-letsencrypt with a fresh cluster, only changing Kubernetes version to 1.11.5-gke.4 and enabling private networking by checking "Enable VPC-native (using alias IP)" when creating the GKE cluster, and using the above sample manifest for the Ingresses.

Anything else we need to know?:

If I delete the IPv6 Ingress and wait, the process still seems to fail. I'm guessing because I still have an AAAA record out there and Let's Encrypt must default to IPv6 if it finds one.

Environment details::

• Kubernetes version (e.g. v1.10.2):

v1.11.5-gke.4

• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc):

GKE

• cert-manager version (e.g. v0.4.0):

v0.5.2

• Install method (e.g. helm or static manifests):

cert-manager installed via helm; my application installed via static manifests

/kind bug

[abevoelker]

FWIW, I just tried re-ordering the manifest to put the IPv6 Ingress on top:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-ingress-ipv6
  annotations:
    kubernetes.io/ingress.global-static-ip-name: webapp-ipv6
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
spec:
  rules:
  - host: captioned-images.duckdns.org
    http:
      paths:
      - path: /*
        backend:
          serviceName: webapp
          servicePort: 3000
  tls:
  - secretName: webapp-tls
    hosts:
    - captioned-images.duckdns.org
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-ingress-ipv4
  annotations:
    kubernetes.io/ingress.global-static-ip-name: webapp-ipv4
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
spec:
  rules:
  - host: captioned-images.duckdns.org
    http:
      paths:
      - path: /*
        backend:
          serviceName: webapp
          servicePort: 3000
  tls:
  - secretName: webapp-tls
    hosts:
    - captioned-images.duckdns.org

And after kubectl apply -f'ing, kubectl describe ing still outputs the IPv4 Ingress first, however cert-manager does seem to pick the IPv6 Ingress to edit first:

Name:             webapp-ingress-ipv4
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (10.4.2.8:8080)
TLS:
  webapp-tls terminates captioned-images.duckdns.org
Rules:
  Host                          Path  Backends
  ----                          ----  --------
  captioned-images.duckdns.org  
                                /*   webapp:3000 (<none>)
Annotations:
  certmanager.k8s.io/cluster-issuer:                 letsencrypt-prod
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/acme-http01-edit-in-place":"true","certmanager.k8s.io/cluster-issuer":"letsencrypt-prod","kubernetes.io/ingress.global-static-ip-name":"webapp-ipv4"},"name":"webapp-ingress-ipv4","namespace":"default"},"spec":{"rules":[{"host":"captioned-images.duckdns.org","http":{"paths":[{"backend":{"serviceName":"webapp","servicePort":3000},"path":"/*"}]}}],"tls":[{"hosts":["captioned-images.duckdns.org"],"secretName":"webapp-tls"}]}}

  kubernetes.io/ingress.global-static-ip-name:   webapp-ipv4
  certmanager.k8s.io/acme-http01-edit-in-place:  true
Events:
  Type     Reason             Age   From                     Message
  ----     ------             ----  ----                     -------
  Normal   ADD                50s   loadbalancer-controller  default/webapp-ingress-ipv4
  Normal   UpdateCertificate  45s   cert-manager             Successfully updated Certificate "webapp-tls"
  Warning  Sync               2s    loadbalancer-controller  Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce


Name:             webapp-ingress-ipv6
Namespace:        default
Address:          2600:1901:0:ed32::
Default backend:  default-http-backend:80 (10.4.2.8:8080)
TLS:
  webapp-tls terminates captioned-images.duckdns.org
Rules:
  Host                          Path  Backends
  ----                          ----  --------
  captioned-images.duckdns.org  
                                /*                                                                        webapp:3000 (<none>)
                                /.well-known/acme-challenge/F78kcBiu65aDgIwdhS9vI3bBNNkG7QgtIOuXxkds26w   cm-acme-http-solver-94djg:8089 (<none>)
Annotations:
  ingress.kubernetes.io/url-map:                     k8s-um-default-webapp-ingress-ipv6--4285d79a798c4100
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/acme-http01-edit-in-place":"true","certmanager.k8s.io/cluster-issuer":"letsencrypt-prod","kubernetes.io/ingress.global-static-ip-name":"webapp-ipv6"},"name":"webapp-ingress-ipv6","namespace":"default"},"spec":{"rules":[{"host":"captioned-images.duckdns.org","http":{"paths":[{"backend":{"serviceName":"webapp","servicePort":3000},"path":"/*"}]}}],"tls":[{"hosts":["captioned-images.duckdns.org"],"secretName":"webapp-tls"}]}}

  kubernetes.io/ingress.global-static-ip-name:   webapp-ipv6
  certmanager.k8s.io/acme-http01-edit-in-place:  true
  certmanager.k8s.io/cluster-issuer:             letsencrypt-prod
  ingress.kubernetes.io/backends:                {"k8s-be-30660--4285d79a798c4100":"Unknown","k8s-be-30760--4285d79a798c4100":"Unknown"}
  ingress.kubernetes.io/forwarding-rule:         k8s-fw-default-webapp-ingress-ipv6--4285d79a798c4100
  ingress.kubernetes.io/target-proxy:            k8s-tp-default-webapp-ingress-ipv6--4285d79a798c4100
Events:
  Type     Reason             Age   From                     Message
  ----     ------             ----  ----                     -------
  Normal   ADD                50s   loadbalancer-controller  default/webapp-ingress-ipv6
  Normal   CreateCertificate  45s   cert-manager             Successfully created Certificate "webapp-tls"
  Normal   UpdateCertificate  45s   cert-manager             Successfully updated Certificate "webapp-tls"
  Warning  Sync               20s   loadbalancer-controller  Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce
  Normal   CREATE             7s    loadbalancer-controller  ip: 2600:1901:0:ed32::

However, cert-manager's self checks continually fail on 404s even though the http-solver has been up and responding to health checks consistently for the last 10 minutes (i.e. it's fully up):

$ kubectl logs cert-manager-7d4bfc44ff-tp9g6 -n kube-system -f
I1218 21:10:43.972331       1 controller.go:171] certificates controller: syncing item 'default/webapp-tls'
I1218 21:10:43.972467       1 sync.go:274] Preparing certificate default/webapp-tls with issuer
I1218 21:10:43.972691       1 logger.go:43] Calling GetOrder
I1218 21:10:44.092836       1 logger.go:73] Calling GetAuthorization
I1218 21:10:44.152894       1 logger.go:93] Calling HTTP01ChallengeResponse
I1218 21:10:44.152932       1 prepare.go:279] Cleaning up old/expired challenges for Certificate default/webapp-tls
I1218 21:10:44.152970       1 logger.go:68] Calling GetChallenge
I1218 21:10:44.286374       1 http.go:134] wrong status code '404'
I1218 21:10:44.286547       1 helpers.go:201] Found status change for Certificate "webapp-tls" condition "Ready": "False" -> "False"; setting lastTransitionTime to 2018-12-18 21:10:44.286540651 +0000 UTC m=+14634.324425237
I1218 21:10:44.286570       1 sync.go:276] Error preparing issuer for certificate default/webapp-tls: http-01 self check failed for domain "captioned-images.duckdns.org"
E1218 21:10:44.286602       1 sync.go:197] [default/webapp-tls] Error getting certificate 'webapp-tls': secret "webapp-tls" not found
I1218 21:10:44.291987       1 controller.go:168] ingress-shim controller: syncing item 'default/webapp-ingress-ipv6'
I1218 21:10:44.292056       1 sync.go:140] Certificate "webapp-tls" for ingress "webapp-ingress-ipv6" already exists
I1218 21:10:44.292071       1 sync.go:143] Certificate "webapp-tls" for ingress "webapp-ingress-ipv6" is up to date
I1218 21:10:44.292079       1 controller.go:182] ingress-shim controller: Finished processing work item "default/webapp-ingress-ipv6"
E1218 21:10:44.293122       1 controller.go:180] certificates controller: Re-queuing item "default/webapp-tls" due to error processing: http-01 self check failed for domain "captioned-images.duckdns.org"

$ kubectl logs cm-acme-http-solver-xwxkd -f   
2018/12/18 20:59:38 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 20:59:38 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 20:59:38 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 20:59:38 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 20:59:39 [captioned-images.duckdns.org] Responding OK to health check '/'
...
2018/12/18 21:10:39 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 21:10:39 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 21:10:39 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 21:10:39 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 21:10:39 [captioned-images.duckdns.org] Responding OK to health check '/'
2018/12/18 21:10:39 [captioned-images.duckdns.org] Responding OK to health check '/'

And of course, the reason for this is because cert-manager is doing self checks over IPv4, which is being passed to my internal application that doesn't have the http-solver path/UrlMap (note 35.244.211.136 request IP address):

I, [2018-12-18T21:10:44.285337 #1]  INFO -- : [eb0fd65e-9ba4-4b86-9230-dcb2325f7064] Started GET "/.well-known/acme-challenge/F78kcBiu65aDgIwdhS9vI3bBNNkG7QgtIOuXxkds26w" for 35.244.211.136 at 2018-12-18 21:10:44 
F, [2018-12-18T21:10:44.285965 #1] FATAL -- : [eb0fd65e-9ba4-4b86-9230-dcb2325f7064]   
F, [2018-12-18T21:10:44.286006 #1] FATAL -- : [eb0fd65e-9ba4-4b86-9230-dcb2325f7064] ActionController::RoutingError (No route matches [GET] "/.well-known/acme-challenge/F78kcBiu65aDgIwdhS9vI3bBNNkG7QgtIOuXxkds26w"
F, [2018-12-18T21:10:44.286023 #1] FATAL -- : [eb0fd65e-9ba4-4b86-9230-dcb2325f7064]   
F, [2018-12-18T21:10:44.286046 #1] FATAL -- : [eb0fd65e-9ba4-4b86-9230-dcb2325f7064] actionpack (5.2.1) lib/action_dispatch/middleware/debug_exceptions.rb:65:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] actionpack (5.2.1) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] railties (5.2.1) lib/rails/rack/logger.rb:38:in `call_app'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] railties (5.2.1) lib/rails/rack/logger.rb:26:in `block in call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] activesupport (5.2.1) lib/active_support/tagged_logging.rb:71:in `block in tagged'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] activesupport (5.2.1) lib/active_support/tagged_logging.rb:28:in `tagged'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] activesupport (5.2.1) lib/active_support/tagged_logging.rb:71:in `tagged'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] railties (5.2.1) lib/rails/rack/logger.rb:26:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] actionpack (5.2.1) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] actionpack (5.2.1) lib/action_dispatch/middleware/request_id.rb:27:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] rack (2.0.6) lib/rack/method_override.rb:22:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] rack (2.0.6) lib/rack/runtime.rb:22:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] activesupport (5.2.1) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] actionpack (5.2.1) lib/action_dispatch/middleware/executor.rb:14:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] actionpack (5.2.1) lib/action_dispatch/middleware/static.rb:127:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] rack (2.0.6) lib/rack/sendfile.rb:111:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] railties (5.2.1) lib/rails/engine.rb:524:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] puma (3.12.0) lib/puma/configuration.rb:225:in `call'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] puma (3.12.0) lib/puma/server.rb:658:in `handle_request'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] puma (3.12.0) lib/puma/server.rb:472:in `process_client'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] puma (3.12.0) lib/puma/server.rb:332:in `block in run'
[eb0fd65e-9ba4-4b86-9230-dcb2325f7064] puma (3.12.0) lib/puma/thread_pool.rb:133:in `block in spawn_thread'

Seems #291 is related to this. I will research filing a feature request with Google to support IPv4 + IPv6 on a single Ingress, but until then us GKE'ers are forced to make two separate Ingresses if we want to support IPv4 and IPv6 on our applications, so it would be awesome if cert-manager worked for us.

[munnerz]

You should not specify the ingress-shim annotations on both ingresses if they specify the same secret name resource.

Instead, you should only include the annotations on one of those, and reference the same secret in both.

If you'd prefer to not use the annotation at all, you can manually create a Certificae resource which creates/manages a secret that is used by both ingress resources 😄

I'm going to close this, as I don't think there's actually a bug here? If I'm wrong though then please re-open the issue!

[skomma]

@abevoelker
If you can install nginx-ingress-controller on the same cluster, you can issue/auto-reissue a certificate with IPv4 + IPv6 Ingresses.
This works as expected in my cluster.
Hope this helps.

1. helm install nginx-ingress-controller with the following values.yaml

controller:
  config:
    # hide Server header
    server-tokens: "false"
    use-forwarded-headers: "true"
  # use default-http-backend service installed on GKE workers
  defaultBackendService: kube-system/default-http-backend
  kind: DaemonSet
  readinessProbe:
    # for GCLB health checking
    port: 80
  service:
    type: NodePort
    # https terminated at GCLB
    enableHttps: false

defaultBackend:
  enabled: false

serviceAccount:
  create: true

2. kubectl apply the following Ingresses(controlled by ingress-gce)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-ingress-external-ipv4
  annotations:
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.global-static-ip-name: webapp-ipv4
spec:
  rules:
  - host: captioned-images.duckdns.org
    http:
      paths:
      - path: /*
        backend:
          serviceName: nginx-ingress-controller
          servicePort: 80
  tls:
  - secretName: webapp-tls
    hosts:
    - captioned-images.duckdns.org
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-ingress-external-ipv6
  annotations:
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.global-static-ip-name: webapp-ipv6
spec:
  rules:
  - host: captioned-images.duckdns.org
    http:
      paths:
      - path: /*
        backend:
          serviceName: nginx-ingress-controller
          servicePort: 80
  tls:
  - secretName: webapp-tls
    hosts:
    - captioned-images.duckdns.org

3. kubectl apply the following Ingress(controlled by nginx-ingress-controller)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapp-ingress-internal
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: captioned-images.duckdns.org
    http:
      paths:
      - path: /
        backend:
          serviceName: webapp
          servicePort: 3000

4. kubectl apply the following Certificate

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: webapp-cert
spec:
  acme:
    config:
    - domains:
      - captioned-images.duckdns.org
      http01:
        ingress: webapp-ingress-internal
  dnsNames:
  - captioned-images.duckdns.org
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: webapp-tls