cert-manager v1.5.4 don't recognize the AGIC ingressClassName as valid

[carlosbustillordguez]

When we configure the Application Gateway Ingress Controller (AGIC) with cert-manager integration, the AGIC ingressClassName is not recognized as valid (Ingress.extensions "cm-acme-http-solver-ftmpn" is invalid: spec.ingressClassName: Invalid value: "azure/application-gateway").

Error details:

E1019 16:15:11.634170       1 controller.go:163] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="Ingress.extensions \"cm-acme-http-solver-ftmpn\" is invalid: spec.ingressClassName: Invalid value: \"azure/application-gateway\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')" "key"="default/nginx-secret-name-zxb9l-723394921-2840615747"

Environment details:

• AKS version: 1.20.9
• cert-manager version: 1.5.4
• AGIC version: 1.4.0

I tested cert-manager version 1.4.3 and version 1.5.3 and works as expected.

My ClusterIssuer is defined as:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:

    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: certs@domain.com

    # ACME server URL for Let’s Encrypt’s staging environment.
    # The staging environment will not issue trusted certificates but is
    # used to ensure that the verification process is working properly
    # before moving to production
    server: https://acme-staging-v02.api.letsencrypt.org/directory

    privateKeySecretRef:
      # Name of a secret used to store the ACME account private key.
      name: letsencrypt-staging

    # Enable the HTTP-01 challenge provider
    # you prove ownership of a domain by ensuring that a particular
    # file is present at the domain
    solvers:
    # An empty 'selector' means that this solver matches all domains
    - selector: {}
      http01:
        ingress:
          class: azure/application-gateway

I noted for cert-manager version 1.5.4 if I modify http01.ingress.class to http01.ingress.name in the ClusterIssuer definition works, e.g.:

http01:
  ingress:
    # class: azure/application-gateway
    name: <ingress_name>

For now, I am using cert-manager v1.5.3 to select the ingress definitions by their class name.

[irbekrm]

Hi @carlosbustillordguez ,

Thanks for opening the issue and pointing this out.

In cert-manager v1.5.4 we made some changes to the ingress-shim component in response to changes to Ingress upstream (to support the v1 Ingress API and the new ingressClassName field).

When cert-manager's ingress-shim component creates a temporary Ingress resource to solve the HTTP-01 challenge it now sets the spec.ingressClassName field instead of the deprecated class name annotation. Since this field is expected to point at an IngressClass resource, azure/application-gateway would be an invalid value (not a valid Kubernetes resource name).

You might be able to work around the invalid class name issue by setting a different class name, but it looks like AGIC does not yet respect the ingressClassName field and because we no longer set the deprecated annotation on our Ingresses, this will not work.

It looks like that the support for the ingressClassName field might be added in the next release of AGIC Azure/application-gateway-kubernetes-ingress#1122.

[irbekrm]

From our side, we might need to add some more documentation around the change in v1.5.4

[carlosbustillordguez]

Hi @carlosbustillordguez ,

Thanks for opening the issue and pointing this out.

In cert-manager v1.5.4 we made some changes to the ingress-shim component in response to changes to Ingress upstream (to support the v1 Ingress API and the new ingressClassName field).

When cert-manager's ingress-shim component creates a temporary Ingress resource to solve the HTTP-01 challenge it now sets the spec.ingressClassName field instead of the deprecated class name annotation. Since this field is expected to point at an IngressClass resource, azure/application-gateway would be an invalid value (not a valid Kubernetes resource name).

You might be able to work around the invalid class name issue by setting a different class name, but it looks like AGIC does not yet respect the ingressClassName field and because we no longer set the deprecated annotation on our Ingresses, this will not work.

It looks like that the support for the ingressClassName field might be added in the next release of AGIC Azure/application-gateway-kubernetes-ingress#1122.

Hi @irbekrm:

Thanks for your quick reply and explanations. Now I can understand what is happening, I think we can use cert-manger v1.5.3 until AGIC supports the v1 Ingress API.

[irbekrm]

Thanks for your quick reply and explanations. Now I can understand what is happening, I think we can use cert-manger v1.5.3 until AGIC supports the v1 Ingress API.

You can either do this or use the acme.cert-manager.io/http01-edit-in-place: "true" annotation (see our docs) on your existing AGIC Ingress so cert-manager modifies that Ingress rather than creating a new one that AGIC doesn't know how to parse. I think this should work too although I haven't tried it with AGIC.

[carlosbustillordguez]

Hi @irbekrm:

You can either do this or use the acme.cert-manager.io/http01-edit-in-place: "true" annotation (see our docs) on your existing AGIC Ingress so cert-manager modifies that Ingress rather than creating a new one that AGIC doesn't know how to parse. I think this should work too although I haven't tried it with AGIC.

I tried the use of acme.cert-manager.io/http01-edit-in-place: "true" in my Ingress and the certificate has been issued correctly. Thanks for the tip!! So, using the above annotation we can use the cert-manager v1.5.4 with AGIC v1.4.0.

[jgregorcmh]

Thank you to everyone else in this thread as it helped me find the issue with my own AGIC / cert-manager / Kubernetes version issue.

I am running Kubernetes v1.21.2 with AGIC v1.4.0 with cert-manager v1.6.1. I reached this version combo for a series of reasons. I too ran afoul of the issue with cert-manager implementing API networking.k8s.io/v1 and AGIC not fully supporting that spec. I had to set my ingress apiVersion to networking.k8s.io/v1 to match updated cert-manager but still use the annotation for kubernetes.io/ingress.class to satisfy AGIC. I then used the acme.cert-manager.io/http01-edit-in-place: true to force cert-manager to rewrite this ingress rather than spin up it's own. To then pass the challenge I had to add appgw.ingress.kubernetes.io/ssl-redirect: true to push the http challenge to https as the app gateway creates only the https listener on a simple ingress. Then it was necessary to add cert-manager.io/issue-temporary-certificate: true so that the https connection would work until the cert is issued. The HTTP-01 challenge respected the redirect and Let's Encrypt issued a certificate.

[carlosbustillordguez]

Thank you to everyone else in this thread as it helped me find the issue with my own AGIC / cert-manager / Kubernetes version issue.

Hello, @jgregorcmh I am glad this thread has helped you to solve your issues with AGIC and cert-manager.

I am running Kubernetes v1.21.2 with AGIC v1.4.0 with cert-manager v1.6.1. I reached this version combo for a series of reasons. I too ran afoul of the issue with cert-manager implementing API networking.k8s.io/v1 and AGIC not fully supporting that spec. I had to set my ingress apiVersion to networking.k8s.io/v1 to match updated cert-manager but still use the annotation for kubernetes.io/ingress.class to satisfy AGIC. I then used the acme.cert-manager.io/http01-edit-in-place: true to force cert-manager to rewrite this ingress rather than spin up it's own. To then pass the challenge I had to add appgw.ingress.kubernetes.io/ssl-redirect: true to push the http challenge to https as the app gateway creates only the https listener on a simple ingress. Then it was necessary to add cert-manager.io/issue-temporary-certificate: true so that the https connection would work until the cert is issued. The HTTP-01 challenge respected the redirect and Let's Encrypt issued a certificate.

Good to know your comments about cert-manager v1.6.1. I don't upgrade to that version because AGIC v1.4.0 doesn't have full support for the networking.k8s.io/v1 API version. The following is a working configuration for AGIC v1.4.0, cert-manager v1.5.4, and Kubernetes v1.20.9:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-agic
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-production
    cert-manager.io/acme-challenge-type: http01
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    # cert-manager modifies that Ingress rather than creating a new one, this is required with cert-manager v1.5.4 and AGIC v1.4.0
    # see: https://github.com/jetstack/cert-manager/issues/4547
    acme.cert-manager.io/http01-edit-in-place: "true"

[rlaveycal]

Also running Kubernetes v1.21.2 with AGIC v1.4.0 with cert-manager v1.6.1

I changed my ClusterIssuer to be

    solvers:
      - http01:
          ingress:
          #   class: azure/application-gateway
            ingressTemplate:
              metadata:
                annotations:
                  kubernetes.io/ingress.class: azure/application-gateway

This avoids the need to put annotations on the ingress