Ingress returns 301 on ACME challenge

[pimjansen]

Describe the bug
The ACME challenge for Cert-Manager keeps returning a 301 on which the cert cant be approved. I guess it is not really a bug though however i can not find anything related in docs nor online. The ingress should not redirect on an acme challenge if im correct.

To Reproduce
Hard to say the exact steps. Below is at least the ingress that i have deployed

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-http
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-http
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - argocd.<domain>.nl
      secretName: argocd.<domain>.nl-tls
  rules:
    - host: argocd.<domain>.nl
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port: 
                  name: https

The actual challenge which keeps pending:

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://argocd.<domain>.nl/.well-known/acme-challenge/KwUul2vb_qbb-f-': Get "https://argocd.<domain>.nl:443/.well-known/acme-challenge/KwUul2vb_qbb-f-": remote error: tls: unrecognized name
  State:       pending
Events:        <none>

Expected behavior
The challenge to be handled properly

Your environment

• Version of the Ingress Controller - release version or a specific commit: 2.1.1
• Version of Kubernetes: 1.21.9
• Kubernetes platform (e.g. Mini-kube or GCP): AKS
• Using NGINX or NGINX Plus: nginx

Additional context
Add any other context about the problem here. Any log files you want to share.

    spec:
      containers:
        - name: nginx-ingress
          image: nginx/nginx-ingress:2.1.1
          args:
            - '-enable-tls-passthrough'
            - '-enable-custom-resources'
            - '-nginx-configmaps=$(POD_NAMESPACE)/nginx-config'
            - '-default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret'
            - '-enable-snippets'
            - '-report-ingress-status'
            - '-external-service=nginx-ingress'

[github-actions]

Hi @pimjansen thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this 🙂

Cheers!

[ciarams87]

Hi @pimjansen sorry for the delay in replying!

You are likely seeing 301 redirect errors because the acme challenge serves on HTTP, not HTTPS, and SSLRedirect defaults to "true". This can be disabled by setting the annotation ingress.kubernetes.io/ssl-redirect: "false" (see the docs here).

This can be circumvented by doing one of two options:

1. You can set the acme.cert-manager.io/http01-edit-in-place: "true" annotation (see here) which will edit the existing Ingress to fulfil the HTTP01 challenge instead of creating a new temporary challenge Ingress. This annotation defaults to false so cert-manager will always attempt to create a new temporary Ingress instead of editing the existing resource.

2. To use cert-manager with Ingress resources and the NGINX Ingress controller without the acme.cert-manager.io/http01-edit-in-place: "true" annotation in place, it is necessary to use mergeable ingress types to avoid host collisions (see the examples here and the documentation on host collisions here).

To do so, configure your HTTP01 ClusterIssuer to use a minion ingress resource, create a master Ingress resource for the host, and use a minion resource for your application Ingress, e.g.:

http01 issuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: http01-issuer
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: myemail@example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: http01-issuer-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
          ingressTemplate:
            metadata:
              annotations:
                "nginx.org/mergeable-ingress-type": "minion"

master Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-http-master
  namespace: argocd
  annotations:
    nginx.org/mergeable-ingress-type: "master"
    cert-manager.io/cluster-issuer: letsencrypt-http
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - argocd.<domain>.nl
    secretName: argocd.<domain>.nl-tls
  rules:
  - host: argocd.<domain>.nl

minion Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-http-minion
  namespace: argocd
  annotations:
    nginx.org/mergeable-ingress-type: "minion"
spec:
  ingressClassName: nginx
  rules:
  - host: argocd.<domain>.nl
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port: 
              name: https

Hopefully this configuration works for you, please let me know if you run into any further difficulty!

[brianehlert]

Since this is related to cert-manager - could this also be achieved using the VirtualServer resource?
The ACME challenge portion of the cert-manager integration was completed with v2.3

https://docs.nginx.com/nginx-ingress-controller/releases/#nginx-ingress-controller-230
https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/custom-resources/certmanager

[pimjansen]

I solved it by using acme.cert-manager.io/http01-edit-in-place: "true" on my ingess

[FaKod]

adding ingress.kubernetes.io/ssl-redirect: "false" to argocd-http-master does not seem to be a solution, right?
I do not want to disable the ssl redirect for all minions.
it would be better i can set there only for the minion that the cert manager creates

[centromere]

Note that this solution will not work with VirtualServer resources.