Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ACME Challenge failed in OTC due forbidden resource creation otcdns #1

Open
hcv-adaumann opened this issue Aug 24, 2023 · 2 comments
Open

ACME Challenge failed in OTC due forbidden resource creation otcdns #1

hcv-adaumann opened this issue Aug 24, 2023 · 2 comments

Comments

@hcv-adaumann
Copy link

I followed your instructions but the ACME challenge failed: Error presenting challenge: otcdns.xxx-development.otc-cert-manager-webhook is forbidden: User "system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller" cannot create resource "otcdns" in API group "xxx-development.otc-cert-manager-webhook" at the cluster scope

I have no idea, that what I have done for troubleshooting:

(a) Setup: values.yaml

infra-otc-cert-manager-webhook:
  groupName: xxx-development.otc-cert-manager-webhook
  cert-manager:
    namespace: xxx-certmanager
    serviceAccountName: certmanager-cert-manager-webhook 
  image:
    repository: swr.eu-de.otc.t-systems.com/xxxxx-development/infra-otc-cert-manager-webhook
    tag: latest
    pullSecret: secretregistryotc

(b) Cluster issuer (Credentials secrets are existing too)

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory

    # Email address used for ACME registration
    email: info@xxxx.de # REPLACE THIS WITH YOUR EMAIL!!!

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging-otcdms

    solvers:
      - dns01:
          webhook:
            groupName: xxx-development.otc-cert-manager-webhook
            solverName: otcdns
            config:
              authURL: "https://iam.eu-de.otc.t-systems.com:443/v3"
              region: "eu-de"
              
              # Only for local testing, if no secrets are available.
              # accessKey: ACCESSKEY
              # secretKey: SECRETKEY

              accessKeySecretRef:
                name: otcdns-credentials
                key: accessKey
              secretKeySecretRef:
                name: otcdns-credentials
                key: secretKey

(c) Ingress Configuration (Helm extracted)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  namespace: "xxx-keycloak"
  labels:
    app.kubernetes.io/name: keycloak
    helm.sh/chart: keycloak-15.1.7
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: keycloak
  annotations:
    cert-manager.io/cluster-issuer: xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
    kubernetes.io/elb.class: performance
    kubernetes.io/elb.id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    kubernetes.io/elb.port: "443"
spec:
  ingressClassName: "cce"
  rules:
    - host: "xxx-cloud.de"
      http:
        paths:
          - path: /iam/
            pathType: ImplementationSpecific
            backend:
              service:
                name: keycloak
                port:
                  name: http
  tls:
    - hosts:
        - "xxxxxxxxxxxxxxxx.de"
      secretName: xxx.de-tls

(d) Certificates are created in differnt namepsace (not certmanager)

Name:         xxx-cloud.de-tls
Namespace:    xxx-keycloak
Labels:       app.kubernetes.io/component=keycloak
              app.kubernetes.io/instance=keycloak
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keycloak
              helm.sh/chart=keycloak-15.1.7
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-08-24T10:39:49Z
  Generation:          1
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  keycloak
    UID:                   a0087ade-a18c-4988-aab8-21c638c04e08
  Resource Version:        4650231
  UID:                     fdd70cf8-c6ca-4b94-a420-0e0580c3cb5a
Spec:
  Dns Names:
    xxx-cloud.de
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Secret Name:  xxx-cloud.de-tls
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:        2023-08-24T10:39:49Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2023-08-24T10:39:49Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  xxx.de-tls-sj845
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    18m   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  18m   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "xxx-cloud.de-tls-sj845"
  Normal  Requested  18m   cert-manager-certificates-request-manager  Created new CertificateRequest resource "xxx-cloud.de-tls-7qdrq"

(e) Certificate request


Name:         xxx.de-tls-7qdrq
Namespace:    xxx-keycloak
Labels:       app.kubernetes.io/component=keycloak
              app.kubernetes.io/instance=keycloak
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keycloak
              helm.sh/chart=keycloak-15.1.7
Annotations:  cert-manager.io/certificate-name: xxx-cloud.de-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: xxx-cloud.de-tls-sj845
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2023-08-24T10:39:49Z
  Generate Name:       xxx-cloud.de-tls-
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  xxx-cloud.de-tls
    UID:                   fdd70cf8-c6ca-4b94-a420-0e0580c3cb5a
  Resource Version:        4650252
  UID:                     5d016903-5319-4753-bfaf-9c5756121533
Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      certmanager-cert-manager-controller-5489f79646-7w4zj
    authentication.kubernetes.io/pod-uid:
      10ebd0e2-77fc-4ce1-ac98-69479264467a
  Groups:
    system:serviceaccounts
    system:serviceaccounts:xxx-certmanager
    system:authenticated
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2dqQ0NBV29DQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxsawp2bDd6dTlvdkozUU5URXoyRmxqS2NFZkswRFhqWmRYU3dMV3B2NXpQTGZnelRZZjkwOWdDNjYyWithWXVyWDE5CmhBYVJEemRnMW1SeENrdkxWT0JETGt5ZFBZWTZIZE1wY2VZMGEvWHVlOVE2RmlSYnpwaFBrbGdXVzFmMHlOQU4KLzZNcWwvM2ptSjl0WDJFaU9RbkVKRFF3RkZ0SzAxOGdzYlV1Y21VNXVXNDVDSEgzTFJ4VWNmRmFDYWVXSzBpcQpreVZ1VTdVdm9hRWdEanV2TVFuN3lkZ1RsV3ZEcll3QWVoVUVlZnp2eTM1QjdoVnlUQzVMVnYzMVV0Mys3RzNtCkFhSGdRVnVEcUdJeGZGM1A3UjN3b3MyTzJQVU1PMldFVGJFeHRWVFFEWXo1OHB5empwdFl6TVc2Um5JSm9rOWUKVXZDS1dLdGFkRVRzZ1JzUmFCVUNBd0VBQWFBOU1Ec0dDU3FHU0liM0RRRUpEakV1TUN3d0hRWURWUjBSQkJZdwpGSUlTYUdNdGRtbHphVzl1TFdOc2IzVmtMbVJsTUFzR0ExVWREd1FFQXdJRm9EQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBc3hqOHV4RVNQYjVaQ0ZxcW9nYlEwbjY1S2U5NmhnTytWeXlBNDNvZGxDTXcvekFRSEN5ZmtYR1AKZGNaSTVsNEdXSk5DeCtXVGprTmNNZUlSRFVyWnV1dVFBTWVVSi9nY1QxY2Roc0RKTFJFYzhUM3NrVHVYMENaSQpFdW1vaXAzK0VOcVR1Qjk3S3c4N1hpcmtpTTgvV2lqb3J5ejF4ay91d1RvM2xQNXJ0OU11NTJkc08zTHRBdUppCm1DYWZZNlFWekVjYWRkaWVIdklsL3VNaDVxNmJKYnlmVlFyRHl4RW5jc0ZJRmJXc3I2L1hjcjJQUzVCdUhXTTEKVVVuSGpPNWR0eUZBVFpIUjU3RGRRUENOSlVwRUcxdTg2K3JMeWxzVFNqQzVOTlprTVN2dWRpTGVRVkU3VVZjWApva21YQXkxNmJ6L0tyU1RWSjNYRUZ5aXU5R2sza0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  UID:      b08953a3-459a-48e1-a43b-8e964fb5a6b1
  Usages:
    digital signature
    key encipherment
  Username:  system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller
Status:
  Conditions:
    Last Transition Time:  2023-08-24T10:39:49Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2023-08-24T10:39:49Z
    Message:               Waiting on certificate issuance from order xxx-keycloak/xxx-cloud.de-tls-7qdrq-3502903507: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason              Age   From                                                Message
  ----    ------              ----  ----                                                -------
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-ca          Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-venafi      Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-vault       Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-selfsigned  Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-acme        Not signing CertificateRequest until it is Approved
  Normal  cert-manager.io     24m   cert-manager-certificaterequests-approver           Certificate request has been approved by cert-manager.io
  Normal  OrderCreated        24m   cert-manager-certificaterequests-issuer-acme        Created Order resource xxx-keycloak/xxx-cloud.de-tls-7qdrq-3502903507
  Normal  OrderPending        24m   cert-manager-certificaterequests-issuer-acme        Waiting on certificate issuance from order xxx-keycloak/xxx-cloud.de-tls-7qdrq-3502903507: ""

(f) Order

Name:         xxx-cloud.de-tls-7qdrq-3502903507
Namespace:    xxx-keycloak
Labels:       app.kubernetes.io/component=keycloak
              app.kubernetes.io/instance=keycloak
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keycloak
              helm.sh/chart=keycloak-15.1.7
Annotations:  cert-manager.io/certificate-name: xxx-cloud.de-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: xxx-cloud.de-tls-sj845
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2023-08-24T10:39:49Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  xxx-cloud.de-tls-7qdrq
    UID:                   5d016903-5319-4753-bfaf-9c5756121533
  Resource Version:        4650254
  UID:                     7ce873d9-ad09-45e5-8b5d-4063b31bfcae
Spec:
  Dns Names:
    xxx-cloud.de
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2dqQ0NBV29DQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxsawp2bDd6dTlvdkozUU5URXoyRmxqS2NFZkswRFhqWmRYU3dMV3B2NXpQTGZnelRZZjkwOWdDNjYyWithWXVyWDE5CmhBYVJEemRnMW1SeENrdkxWT0JETGt5ZFBZWTZIZE1wY2VZMGEvWHVlOVE2RmlSYnpwaFBrbGdXVzFmMHlOQU4KLzZNcWwvM2ptSjl0WDJFaU9RbkVKRFF3RkZ0SzAxOGdzYlV1Y21VNXVXNDVDSEgzTFJ4VWNmRmFDYWVXSzBpcQpreVZ1VTdVdm9hRWdEanV2TVFuN3lkZ1RsV3ZEcll3QWVoVUVlZnp2eTM1QjdoVnlUQzVMVnYzMVV0Mys3RzNtCkFhSGdRVnVEcUdJeGZGM1A3UjN3b3MyTzJQVU1PMldFVGJFeHRWVFFEWXo1OHB5empwdFl6TVc2Um5JSm9rOWUKVXZDS1dLdGFkRVRzZ1JzUmFCVUNBd0VBQWFBOU1Ec0dDU3FHU0liM0RRRUpEakV1TUN3d0hRWURWUjBSQkJZdwpGSUlTYUdNdGRtbHphVzl1TFdOc2IzVmtMbVJsTUFzR0ExVWREd1FFQXdJRm9EQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBc3hqOHV4RVNQYjVaQ0ZxcW9nYlEwbjY1S2U5NmhnTytWeXlBNDNvZGxDTXcvekFRSEN5ZmtYR1AKZGNaSTVsNEdXSk5DeCtXVGprTmNNZUlSRFVyWnV1dVFBTWVVSi9nY1QxY2Roc0RKTFJFYzhUM3NrVHVYMENaSQpFdW1vaXAzK0VOcVR1Qjk3S3c4N1hpcmtpTTgvV2lqb3J5ejF4ay91d1RvM2xQNXJ0OU11NTJkc08zTHRBdUppCm1DYWZZNlFWekVjYWRkaWVIdklsL3VNaDVxNmJKYnlmVlFyRHl4RW5jc0ZJRmJXc3I2L1hjcjJQUzVCdUhXTTEKVVVuSGpPNWR0eUZBVFpIUjU3RGRRUENOSlVwRUcxdTg2K3JMeWxzVFNqQzVOTlprTVN2dWRpTGVRVkU3VVZjWApva21YQXkxNmJ6L0tyU1RWSjNYRUZ5aXU5R2sza0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
Status:
  Authorizations:
    Challenges:
      Token:        r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/ShWR4A
      Token:        r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/3dlcFw
      Token:        r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/i4oMiA
    Identifier:     xxx-cloud.de
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7898030084
    Wildcard:       false
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/115937054/10473000664
  State:            pending
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/115937054/10473000664
Events:
  Type    Reason   Age   From                 Message
  ----    ------   ----  ----                 -------
  Normal  Created  25m   cert-manager-orders  Created Challenge resource "xxx-cloud.de-tls-7qdrq-3502903507-2917238827" for domain "xxx-cloud.de"

(g) ACME Challenge

Name:         xxx-cloud.de-tls-7qdrq-3502903507-2917238827
Namespace:    xxx-keycloak
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2023-08-24T10:39:51Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  xxx-cloud.de-tls-7qdrq-3502903507
    UID:                   7ce873d9-ad09-45e5-8b5d-4063b31bfcae
  Resource Version:        4650268
  UID:                     780aefbc-edff-48cd-bbd4-1c69c707562a
Spec:
  Authorization URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7898030084
  Dns Name:           xxx-cloud.de
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Key:      cGlSqs15z01PWk_PhWLi5WS4zm1QgQ4LnMs5vHmsenI
  Solver:
    dns01:
      Webhook:
        Config:
          Access Key Secret Ref:
            Key:     accessKey
            Name:    otcdns-credentials
          Auth URL:  https://iam.eu-de.otc.t-systems.com:443/v3
          Region:    eu-de
          Secret Key Secret Ref:
            Key:      secretKey
            Name:     otcdns-credentials
        Group Name:   xxx-development.otc-cert-manager-webhook
        Solver Name:  otcdns
  Token:              r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
  Type:               DNS-01
  URL:                https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/3dlcFw
  Wildcard:           false
Status:
  Presented:   false
  Processing:  true
  Reason:      otcdns.xxx-development.otc-cert-manager-webhook is forbidden: User "system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller" cannot create resource "otcdns" in API group "xxx-development.otc-cert-manager-webhook" at the cluster scope
  State:       pending
Events:
  Type     Reason        Age                  From                     Message
  ----     ------        ----                 ----                     -------
  Normal   Started       27m                  cert-manager-challenges  Challenge scheduled for processing
  Warning  PresentError  6m20s (x9 over 27m)  cert-manager-challenges  Error presenting challenge: otcdns.xxx-development.otc-cert-manager-webhook is forbidden: User "system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller" cannot create resource "otcdns" in API group "xxx-development.otc-cert-manager-webhook" at the cluster scope
@hcv-adaumann
Copy link
Author

Same problem if using a issuer instead of a clusterissuer.

@Crenshinibon
Copy link

Likely a problem with access rights in OTC itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Crenshinibon @hcv-adaumann