Route53 DNS01 cannot process due to refused error #1627

nmiculinic · 2019-05-02T13:57:37Z

Describe the bug:
In kubernetes I have cert-manager providing TLS certificated. It correctly created the TXT entry on the route53, but it cannot fetch it for some reason.

Expected behaviour:
Correctly fetch the _acme-challenge.ascalia.io. TXT entry which has successfuly setup in an earlier step

A concise description of what you expected to happen.

Steps to reproduce the bug:

install cert-manager via helm
create needed secrets for accessing route53
attempt to create cerficate

Anything else we need to know?:

The route53 credentials have full route53 admin permissions

helm setup:

fullnameOverride: cert-manager
securityContext:
  enabled: true
ingressShim:
  defaultIssuerName: kraken
  defaultIssuerKind: ClusterIssuer
  defaultACMEChallengeType:	dns01
  defaultACMEDNS01ChallengeProvider: route53
webhook:
  enabled: false

Certificate in question:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: wild-ascalia-io
  namespace: ingress
spec:
  acme:
    config:
      - dns01:
          provider: route53
        domains:
          - '*.ascalia.io'
  dnsNames:
    - '*.ascalia.io'
  issuerRef:
    kind: ClusterIssuer
    name: kraken
  secretName: 'wild-ascalia-io'

Logs:

Alias tip: kl cert-manager-656476c4dd-28sl5
I0502 13:15:31.789219       1 start.go:81] starting cert-manager v0.6.2 (revision f5e1477bd7ced69e53a233484905fea16bf4102f)
I0502 13:15:31.790877       1 controller.go:141] Using the following nameservers for DNS01 checks: [10.233.0.3:53]
I0502 13:15:31.791953       1 leaderelection.go:193] attempting to acquire leader lease  ingress/cert-manager-controller...
I0502 13:16:41.249629       1 leaderelection.go:202] successfully acquired lease ingress/cert-manager-controller
I0502 13:16:41.250088       1 controller.go:82] Starting certificates controller
I0502 13:16:41.250208       1 controller.go:82] Starting clusterissuers controller
I0502 13:16:41.254078       1 metrics.go:145] Listening on http://0.0.0.0:9402
I0502 13:16:41.254239       1 controller.go:82] Starting issuers controller
I0502 13:16:41.254212       1 controller.go:82] Starting ingress-shim controller
I0502 13:16:41.257175       1 controller.go:82] Starting orders controller
I0502 13:16:41.257534       1 controller.go:82] Starting challenges controller
I0502 13:16:41.353258       1 controller.go:145] certificates controller: syncing item 'ingress/wild-ascalia-io'
I0502 13:16:41.353410       1 controller.go:141] clusterissuers controller: syncing item 'kraken'
I0502 13:16:41.353793       1 setup.go:149] Skipping re-verifying ACME account as cached registration details look sufficient.
I0502 13:16:41.353819       1 controller.go:147] clusterissuers controller: Finished processing work item "kraken"
I0502 13:16:41.353902       1 issue.go:154] Order ingress/wild-ascalia-io-2419620338 is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate.
I0502 13:16:41.353931       1 controller.go:151] certificates controller: Finished processing work item "ingress/wild-ascalia-io"
I0502 13:16:41.357468       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/prom-grafana'
I0502 13:16:41.357487       1 sync.go:64] Not syncing ingress kube-system/prom-grafana as it does not contain necessary annotations
I0502 13:16:41.357491       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/prom-grafana"
I0502 13:16:41.357495       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/prom-prometheus-operator-alertmanager'
I0502 13:16:41.357504       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/prom-prometheus-operator-prometheus'
I0502 13:16:41.357512       1 sync.go:64] Not syncing ingress kube-system/prom-prometheus-operator-prometheus as it does not contain necessary annotations
I0502 13:16:41.357517       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/prom-prometheus-operator-prometheus"
I0502 13:16:41.357528       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/kubernetes-dashboard'
I0502 13:16:41.357545       1 sync.go:64] Not syncing ingress kube-system/kubernetes-dashboard as it does not contain necessary annotations
I0502 13:16:41.357550       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/kubernetes-dashboard"
I0502 13:16:41.357512       1 sync.go:64] Not syncing ingress kube-system/prom-prometheus-operator-alertmanager as it does not contain necessary annotations
I0502 13:16:41.357584       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/prom-prometheus-operator-alertmanager"
I0502 13:16:41.357530       1 controller.go:173] ingress-shim controller: syncing item 'kube-system/elk-kibana'
I0502 13:16:41.357622       1 sync.go:64] Not syncing ingress kube-system/elk-kibana as it does not contain necessary annotations
I0502 13:16:41.357628       1 controller.go:179] ingress-shim controller: Finished processing work item "kube-system/elk-kibana"
I0502 13:16:41.357728       1 controller.go:183] orders controller: syncing item 'ingress/wild-ascalia-io-2419620338'
I0502 13:16:41.357746       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:16:41.357906       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
I0502 13:16:41.357928       1 sync.go:274] Need to create 0 challenges
I0502 13:16:41.357939       1 sync.go:323] Waiting for all challenges for order "wild-ascalia-io-2419620338" to enter 'valid' state
I0502 13:16:41.357974       1 controller.go:189] orders controller: Finished processing work item "ingress/wild-ascalia-io-2419620338"
E0502 13:16:41.389396       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:16:46.318806       1 controller.go:141] clusterissuers controller: syncing item 'kraken'
I0502 13:16:46.319024       1 setup.go:149] Skipping re-verifying ACME account as cached registration details look sufficient.
I0502 13:16:46.319048       1 controller.go:147] clusterissuers controller: Finished processing work item "kraken"
I0502 13:16:46.389597       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:16:46.389758       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:16:46.405789       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:16:56.406051       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:16:56.406239       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:16:56.432658       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:17:16.432842       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:17:16.433023       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:17:16.450047       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:17:56.450226       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:17:56.450477       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:17:56.467009       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:19:16.467194       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:19:16.467374       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:19:16.487313       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1536.awsdns-00.co.uk.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:21:56.487496       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:21:56.487668       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:21:56.505097       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:27:16.505337       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:27:16.505516       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:27:16.523518       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.
I0502 13:37:56.523716       1 controller.go:205] challenges controller: syncing item 'ingress/wild-ascalia-io-2419620338-0'
I0502 13:37:56.523901       1 dns.go:110] Checking DNS propagation for "ascalia.io" using name servers: [10.233.0.3:53]
E0502 13:37:56.543110       1 controller.go:207] challenges controller: Re-queuing item "ingress/wild-ascalia-io-2419620338-0" due to error processing: NS ns-1024.awsdns-00.org.:53 returned REFUSED for _acme-challenge.ascalia.io.

There exists a private zone ascalia.io, though I see cert-manager create the DNS TXT entry in the proper public zone.

Environment details::

Kubernetes version (e.g. v1.10.2): v1.13.5
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AWS
cert-manager version (e.g. v0.4.0): v0.6.2
Install method (e.g. helm or static manifests): helm

/kind bug

The text was updated successfully, but these errors were encountered:

wiltonfelix · 2019-05-17T12:41:04Z

hi @nmiculinic , this is a problem in k8s internal cluster, to solve just add to your cert-manager config yaml this:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: cert-manager
  namespace: "cert-manager"
  labels:
    app: cert-manager
    chart: cert-manager-v0.7.2
    release: cert-manager
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cert-manager
      release: cert-manager
  template:
    metadata:
      labels:
        app: cert-manager
        release: cert-manager
      annotations:
        prometheus.io/path: "/metrics"
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9402'
    spec:
      serviceAccountName: cert-manager
      containers:
        - name: cert-manager
          image: "quay.io/jetstack/cert-manager-controller:v0.7.2"
          imagePullPolicy: IfNotPresent
          args:
          - --cluster-resource-namespace=$(POD_NAMESPACE)
          - --leader-election-namespace=$(POD_NAMESPACE)
          - --dns01-recursive-nameservers="8.8.8.8:53"
          - --dns01-recursive-nameservers-only
          ports:
          - containerPort: 9402
          env:
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          resources:
            requests:
              cpu: 10m
              memory: 32Mi

or in helm values config:

extraArg:
--dns01-recursive-nameservers-only
--dns01-self-check-nameservers=8.8.8.8:53

for me this is an information problem in the cert-manager documentation, but I will open a PR for this.

PabloCastellano · 2019-05-24T10:11:32Z

@wiltonfelix Thanks! I've noticed that you have used --dns01-self-check-nameservers=8.8.8.8:53 in the helm example and --dns01-recursive-nameservers="8.8.8.8:53" in the manifest. Is it a typo?

Also, could you edit your comment and add markdown format to it by using triple backticks? (check this)

TheKangaroo · 2019-06-11T09:21:11Z

@PabloCastellano, --dns01-self-check-nameservers is deprecated in favor of --dns01-recursive-nameservers, but as of 0.8.0 both versions still work.

retest-bot · 2019-09-09T10:01:28Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

retest-bot · 2019-10-09T10:45:53Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

retest-bot · 2019-11-08T11:13:29Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

jetstack-bot · 2019-11-08T11:13:38Z

@retest-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

chrislovecnm · 2022-04-26T14:50:19Z

/reopen

Do we have documentation around this problem?

jetstack-bot · 2022-04-26T14:50:21Z

@chrislovecnm: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Do we have documentation around this problem?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

chrislovecnm · 2022-04-26T15:31:18Z

#1627 (comment) fixed the problem on EKS

jetstack-bot added the kind/bug Categorizes issue or PR as related to a bug. label May 2, 2019

jetstack-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 9, 2019

jetstack-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 9, 2019

jetstack-bot closed this as completed Nov 8, 2019

kobethuwis mentioned this issue Nov 18, 2022

"propagation check failed" for DNS-01 ACME challenge on internal EKS-cluster #5586

Closed

maelvls mentioned this issue Nov 24, 2023

DNS-01 challenge propagation | NS ns-512.awsdns-00.net.:53 returned REFUSED for _acme-challenge .... #6510

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Route53 DNS01 cannot process due to refused error #1627

Route53 DNS01 cannot process due to refused error #1627

nmiculinic commented May 2, 2019

wiltonfelix commented May 17, 2019 •

edited

PabloCastellano commented May 24, 2019 •

edited

TheKangaroo commented Jun 11, 2019

retest-bot commented Sep 9, 2019

retest-bot commented Oct 9, 2019

retest-bot commented Nov 8, 2019

jetstack-bot commented Nov 8, 2019

chrislovecnm commented Apr 26, 2022

jetstack-bot commented Apr 26, 2022

chrislovecnm commented Apr 26, 2022

Route53 DNS01 cannot process due to refused error #1627

Route53 DNS01 cannot process due to refused error #1627

Comments

nmiculinic commented May 2, 2019

wiltonfelix commented May 17, 2019 • edited

PabloCastellano commented May 24, 2019 • edited

TheKangaroo commented Jun 11, 2019

retest-bot commented Sep 9, 2019

retest-bot commented Oct 9, 2019

retest-bot commented Nov 8, 2019

jetstack-bot commented Nov 8, 2019

chrislovecnm commented Apr 26, 2022

jetstack-bot commented Apr 26, 2022

chrislovecnm commented Apr 26, 2022

wiltonfelix commented May 17, 2019 •

edited

PabloCastellano commented May 24, 2019 •

edited