Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wildcard certificate generation with digital ocean dns: unexpected response code NOTIMPL #4088

Closed
dpsarrou opened this issue Oct 22, 2018 · 4 comments
Closed

Wildcard certificate generation with digital ocean dns: unexpected response code NOTIMPL #4088

dpsarrou opened this issue Oct 22, 2018 · 4 comments
Labels
area/acme kind/bug/possible a possible bug that needs analysis before it is confirmed or fixed. priority/P3 maybe status/5-frozen-due-to-age

Comments

@dpsarrou
Copy link

dpsarrou commented Oct 22, 2018 •
edited by ldez

Do you want to request a feature or report a bug?

Bug

What did you do?

I tried to configure traefik in order to setup LetsEncrypt with Docker for wildcard certificates with DNS of digital ocean. The project is fairly simple: a single node at monitor.local.example.com hosting a Prometheus stack where each component is a docker container and listens to different subdomain. Eg:

alertmanager.monitor.local.example.com
prometheus.monitor.local.example.com
grafana.monitor.local.example.com

I kept receiving error messages (as you can see in the next sections) and couldn't make it work.
In order to verify if something is wrong with the letsencrypt side I used certbot for the same wildcard domain I have configured in traefik.toml and I was able to successfully generate the certificates.
At this point it is not clear to me from the existing documentation if I'm doing something wrong or if this is indeed a bug in traefik.

The code I used to verify that certificates can be generated with certbot:

docker run -it --rm --name certbot \
            -v "/app/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            certbot/dns-digitalocean certonly \
            --server https://acme-staging-v02.api.letsencrypt.org/directory \
            --dns-digitalocean \
            --dns-digitalocean-credentials /etc/letsencrypt/digitalocean.ini \
            -d *.monitor.local.example.com -d monitor.local.example.com

What did you expect to see?

The certificates being generated.

What did you see instead?

Error obtaining certificate: acme: Error -> One or more domains had a problem:\n[monitor.local.example.com] error presenting token: digitalocean: could not determine zone for domain: 'monitor.local.example.com'. unexpected response code 'NOTIMPL' for monitor.local.example.com

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.7.3
Codename:     maroilles
Go version:   go1.11.1
Built:        2018-10-15_10:13:00AM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

[entryPoints]

  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"

  [entryPoints.https]
  address = ":443"

    [entryPoints.https.tls]
    [entryPoints.https.auth.basic]
    usersFile = "/app/traefik/entrypoints/https.htpasswd"

  [entryPoints.metrics]
  address = ":8484"

    [entryPoints.metrics.tls]
    [entryPoints.metrics.auth.basic]
    usersFile = "/app/traefik/entrypoints/metrics.htpasswd"

    [entryPoints.metrics.whiteList]
    sourceRange = ["127.0.0.1/32", "172.0.0.0/8", "192.168.68.6"]


[metrics]
  [metrics.prometheus]
    entryPoint = "metrics"
    buckets = [0.1, 0.3, 1.2, 5.0]

[docker]
domain = "monitor.local.example.com"
watch = true
exposedByDefault = false

[acme]
email = "email@example.com"
storage = "acme.json"
entryPoint = "https"
acmeLogging = true
onHostRule = true
caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

[acme.dnsChallenge]
  provider = "digitalocean"

[[acme.domains]]
  main = "*.monitor.local.example.com"
  sans = ["monitor.local.example.com"]

If applicable, please paste the log output in DEBUG level (--logLevel=DEBUG switch)

logs 
time="2018-10-22T07:38:57Z" level=info msg="Using TOML configuration file /etc/traefik/traefik.toml"
time="2018-10-22T07:38:57Z" level=info msg="Traefik version v1.7.3 built on 2018-10-15_10:13:00AM"
time="2018-10-22T07:38:57Z" level=debug msg="Global configuration loaded {\"LifeCycle\":{\"RequestAcceptGraceTimeout\":0,\"GraceTimeOut\":10000000000},\"GraceTimeOut\":0,\"Debug\":true,\"CheckNewVersion\":true,\"SendAnonymousUsage\":false,\"AccessLogsFile\":\"\",\"AccessLog\":null,\"TraefikLogsFile\":\"\",\"TraefikLog\":null,\"Tracing\":null,\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Address\":\":80\",\"TLS\":null,\"Redirect\":{\"entryPoint\":\"https\"},\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}},\"https\":{\"Address\":\":443\",\"TLS\":{\"MinVersion\":\"\",\"CipherSuites\":null,\"Certificates\":null,\"ClientCAFiles\":null,\"ClientCA\":{\"Files\":null,\"Optional\":false},\"DefaultCertificate\":null,\"SniStrict\":false},\"Redirect\":null,\"Auth\":{\"basic\":{\"usersFile\":\"/app/traefik/entrypoints/https.htpasswd\"}},\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}},\"metrics\":{\"Address\":\":8484\",\"TLS\":{\"MinVersion\":\"\",\"CipherSuites\":null,\"Certificates\":null,\"ClientCAFiles\":null,\"ClientCA\":{\"Files\":null,\"Optional\":false},\"DefaultCertificate\":null,\"SniStrict\":false},\"Redirect\":null,\"Auth\":{\"basic\":{\"usersFile\":\"/app/traefik/entrypoints/metrics.htpasswd\"}},\"WhitelistSourceRange\":null,\"WhiteList\":{\"sourceRange\":[\"127.0.0.1/32\",\"172.0.0.0/8\",\"192.168.68.6\"]},\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}},\"traefik\":{\"Address\":\":8080\",\"TLS\":null,\"Redirect\":null,\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}}},\"Cluster\":null,\"Constraints\":[],\"ACME\":{\"Email\":\"email@example.com\",\"Domains\":[{\"Main\":\"*.monitor.local.example.com\",\"SANs\":[\"monitor.local.example.com\"]}],\"Storage\":\"acme.json\",\"StorageFile\":\"\",\"OnDemand\":false,\"OnHostRule\":true,\"CAServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"EntryPoint\":\"https\",\"KeyType\":\"\",\"DNSChallenge\":{\"Provider\":\"digitalocean\",\"DelayBeforeCheck\":0},\"HTTPChallenge\":null,\"TLSChallenge\":null,\"DNSProvider\":\"\",\"DelayDontCheckDNS\":0,\"ACMELogging\":true,\"OverrideCertificates\":false,\"TLSConfig\":null},\"DefaultEntryPoints\":[\"http\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"IdleTimeout\":0,\"InsecureSkipVerify\":false,\"RootCAs\":null,\"Retry\":null,\"HealthCheck\":{\"Interval\":30000000000},\"RespondingTimeouts\":null,\"ForwardingTimeouts\":null,\"AllowMinWeightZero\":false,\"Web\":null,\"Docker\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Trace\":false,\"TemplateVersion\":2,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\"monitor.local.example.com\",\"TLS\":null,\"ExposedByDefault\":true,\"UseBindPortIP\":false,\"SwarmMode\":false,\"Network\":\"\"},\"File\":null,\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":null,\"Mesos\":null,\"Eureka\":null,\"ECS\":null,\"Rancher\":null,\"DynamoDB\":null,\"ServiceFabric\":null,\"Rest\":null,\"API\":{\"EntryPoint\":\"traefik\",\"Dashboard\":true,\"Debug\":true,\"CurrentConfigurations\":null,\"Statistics\":null},\"Metrics\":{\"Prometheus\":{\"Buckets\":[0.1,0.3,1.2,5],\"EntryPoint\":\"metrics\"},\"Datadog\":null,\"StatsD\":null,\"InfluxDB\":null},\"Ping\":null,\"HostResolver\":null}"
time="2018-10-22T07:38:57Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"
time="2018-10-22T07:38:57Z" level=debug msg="Setting Acme Certificate store from Entrypoint: https"
time="2018-10-22T07:38:57Z" level=debug msg="configured IP white list: [127.0.0.1/32 172.0.0.0/8 192.168.68.6]"
time="2018-10-22T07:38:57Z" level=debug msg="Configured Prometheus metrics"
time="2018-10-22T07:38:58Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:0xc000918e00 Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0000274c0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-10-22T07:38:58Z" level=info msg="Preparing server https &{Address::443 TLS:0xc0002ce6c0 Redirect:<nil>Auth:0xc000578a80 WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000027520} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-10-22T07:38:58Z" level=info msg="Starting server on :80"
time="2018-10-22T07:38:59Z" level=debug msg="configured IP white list: [127.0.0.1/32 172.0.0.0/8 192.168.68.6]"
time="2018-10-22T07:38:59Z" level=info msg="Preparing server metrics &{Address::8484 TLS:0xc0002ce7e0 Redirect:<nil> Auth:0xc000578ba0 WhitelistSourceRange:[] WhiteList:0xc00018f8e0 Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000027560} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-10-22T07:38:59Z" level=info msg="Starting server on :443"
time="2018-10-22T07:38:59Z" level=info msg="Preparing server traefik &{Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000027600} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-10-22T07:38:59Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
time="2018-10-22T07:38:59Z" level=info msg="Starting server on :8484"
time="2018-10-22T07:38:59Z" level=info msg="Starting server on :8080"
time="2018-10-22T07:38:59Z" level=info msg="Starting provider *docker.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Trace\":false,\"TemplateVersion\":2,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\"monitor.local.example.com\",\"TLS\":null,\"ExposedByDefault\":true,\"UseBindPortIP\":false,\"SwarmMode\":false,\"Network\":\"\"}"
time="2018-10-22T07:38:59Z" level=info msg="Starting provider *acme.Provider {\"Email\":\"email@example.com\",\"ACMELogging\":true,\"CAServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"Storage\":\"acme.json\",\"EntryPoint\":\"https\",\"KeyType\":\"\",\"OnHostRule\":true,\"OnDemand\":false,\"DNSChallenge\":{\"Provider\":\"digitalocean\",\"DelayBeforeCheck\":0},\"HTTPChallenge\":null,\"TLSChallenge\":null,\"Domains\":[{\"Main\":\"*.monitor.local.example.com\",\"SANs\":[\"monitor.local.example.com\"]}],\"Store\":{}}"
time="2018-10-22T07:38:59Z" level=info msg="Testing certificate renew..."
time="2018-10-22T07:38:59Z" level=debug msg="Configuration received from provider ACME: {}"
time="2018-10-22T07:38:59Z" level=debug msg="Looking for provided certificate(s) to validate [\"*.monitor.local.example.com\" \"monitor.local.example.com\"]..."
time="2018-10-22T07:38:59Z" level=debug msg="Domains [\"*.monitor.local.example.com\" \"monitor.local.example.com\"] need ACME certificates generation for domains \"*.monitor.local.example.com,monitor.local.example.com\"."
time="2018-10-22T07:38:59Z" level=debug msg="Loading ACME certificates [*.monitor.local.example.com monitor.local.example.com]..."
time="2018-10-22T07:38:59Z" level=info msg="The key type is empty. Use default key type 4096."
time="2018-10-22T07:38:59Z" level=debug msg="Creating entry point redirect http -> https"
time="2018-10-22T07:38:59Z" level=debug msg="Provider connection established with docker 18.06.1-ce (API 1.38)"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[com.docker.compose.service:traefik com.docker.compose.version:1.22.0 org.label-schema.description:A modern reverse-proxy org.label-schema.name:Traefik org.label-schema.vendor:Containous org.label-schema.version:v1.7.3 com.docker.compose.config-hash:68f9de8de0ff407954ef1ed00e646fff47a6ea2327da42b6813f970924b8c1bd com.docker.compose.oneoff:False com.docker.compose.project:traefik org.label-schema.docker.schema-version:1.0 org.label-schema.url:https://traefik.io com.docker.compose.container-number:1]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[traefik.frontend.entryPoints:metrics traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/node-exporter;ReplacePath:/metrics com.docker.compose.config-hash:5937ece18126e625fbe50ad5c4b1e5f9d08cf6d5995f45bb521e0f246720588e monitoring.service.type:prometheus-exporter prometheus.exporter:node-exporter traefik.docker.network:monitoring traefik.enable:true com.docker.compose.oneoff:False com.docker.compose.project:node-exporter com.docker.compose.service:node-exporter com.docker.compose.version:1.22.0 com.docker.compose.container-number:1 monitoring.node.hostname:monitor.local.example.com traefik.port:9100]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.port:9100 traefik.docker.network:monitoring traefik.frontend.entryPoints:metrics traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/node-exporter;ReplacePath:/metrics traefik.enable:true]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[traefik.port:3000 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:prometheus com.docker.compose.version:1.22.0 traefik.frontend.rule:Host:monitor.local.example.com com.docker.compose.config-hash:fba6e4f60e1955527fed43dff45af1f962b62a8243154178cda2384c25260598 com.docker.compose.service:grafana traefik.docker.network:web traefik.enable:true traefik.frontend.entryPoints:https]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.frontend.rule:Host:monitor.local.example.com traefik.port:3000 traefik.docker.network:web traefik.enable:true traefik.frontend.entryPoints:https]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[com.docker.compose.config-hash:7e5b9be19620bb1e23f253353840725a68398deeb998c841b0611753c0ae186c com.docker.compose.oneoff:False com.docker.compose.project:prometheus com.docker.compose.service:alertmanager traefik.enable:true com.docker.compose.container-number:1 com.docker.compose.version:1.22.0 traefik.docker.network:web traefik.frontend.entryPoints:https traefik.frontend.rule:Host:alertmanager.monitor.local.example.com traefik.port:9093]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.docker.network:web traefik.frontend.entryPoints:https traefik.frontend.rule:Host:alertmanager.monitor.local.example.com traefik.port:9093 traefik.enable:true]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[traefik.docker.network:monitoring com.docker.compose.project:cadvisor com.docker.compose.version:1.22.0 monitoring.service.type:prometheus-exporter prometheus.exporter:cadvisor traefik.enable:true traefik.port:8080 com.docker.compose.service:cadvisor monitoring.node.hostname:monitor.local.example.com traefik.frontend.entryPoints:metrics traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/cadvisor;ReplacePath:/metrics com.docker.compose.config-hash:1e5b3a7de297cb2cbd15913ab928d6af2abc19c1fdc5af5592729546a0125eb7 com.docker.compose.container-number:1 com.docker.compose.oneoff:False]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.enable:true traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/cadvisor;ReplacePath:/metrics traefik.port:8080 traefik.frontend.entryPoints:metrics traefik.docker.network:monitoring]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[org.label-schema.name:Traefik org.label-schema.vendor:Containous org.label-schema.version:v1.7.3 com.docker.compose.config-hash:68f9de8de0ff407954ef1ed00e646fff47a6ea2327da42b6813f970924b8c1bd com.docker.compose.service:traefik com.docker.compose.version:1.22.0 org.label-schema.description:A modern reverse-proxy org.label-schema.url:https://traefik.io com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:traefik org.label-schema.docker.schema-version:1.0]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[com.docker.compose.container-number:1 monitoring.node.hostname:monitor.local.example.com traefik.port:9100 traefik.frontend.entryPoints:metrics traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/node-exporter;ReplacePath:/metrics com.docker.compose.config-hash:5937ece18126e625fbe50ad5c4b1e5f9d08cf6d5995f45bb521e0f246720588e monitoring.service.type:prometheus-exporter prometheus.exporter:node-exporter traefik.docker.network:monitoring traefik.enable:true com.docker.compose.oneoff:False com.docker.compose.project:node-exporter com.docker.compose.service:node-exporter com.docker.compose.version:1.22.0]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.frontend.entryPoints:metrics traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/node-exporter;ReplacePath:/metrics traefik.enable:truetraefik.port:9100 traefik.docker.network:monitoring]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[traefik.frontend.entryPoints:https com.docker.compose.config-hash:fba6e4f60e1955527fed43dff45af1f962b62a8243154178cda2384c25260598 com.docker.compose.service:grafana traefik.docker.network:web traefik.enable:true traefik.frontend.rule:Host:monitor.local.example.com traefik.port:3000 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:prometheus com.docker.compose.version:1.22.0]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.docker.network:web traefik.enable:true traefik.frontend.entryPoints:https traefik.frontend.rule:Host:monitor.local.example.com traefik.port:3000]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[com.docker.compose.config-hash:7e5b9be19620bb1e23f253353840725a68398deeb998c841b0611753c0ae186c com.docker.compose.oneoff:False com.docker.compose.project:prometheus com.docker.compose.service:alertmanager traefik.enable:true com.docker.compose.container-number:1 com.docker.compose.version:1.22.0 traefik.docker.network:web traefik.frontend.entryPoints:https traefik.frontend.rule:Host:alertmanager.monitor.local.example.com traefik.port:9093]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.frontend.entryPoints:https traefik.frontend.rule:Host:alertmanager.monitor.local.example.com traefik.port:9093 traefik.enable:true traefik.docker.network:web]]"
time="2018-10-22T07:38:59Z" level=debug msg="originLabelsmap[monitoring.service.type:prometheus-exporter prometheus.exporter:cadvisor traefik.docker.network:monitoring com.docker.compose.project:cadvisor com.docker.compose.version:1.22.0 traefik.enable:true traefik.frontend.entryPoints:metrics traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/cadvisor;ReplacePath:/metrics traefik.port:8080 com.docker.compose.service:cadvisor monitoring.node.hostname:monitor.local.example.com com.docker.compose.oneoff:False com.docker.compose.config-hash:1e5b3a7de297cb2cbd15913ab928d6af2abc19c1fdc5af5592729546a0125eb7 com.docker.compose.container-number:1]"
time="2018-10-22T07:38:59Z" level=debug msg="allLabelsmap[:map[traefik.frontend.rule:Host:metrics.monitor.local.example.com;Path:/cadvisor;ReplacePath:/metrics traefik.docker.network:monitoring traefik.enable:true traefik.port:8080 traefik.frontend.entryPoints:metrics]]"
time="2018-10-22T07:38:59Z" level=debug msg="Backend backend-traefik-traefik: no load-balancer defined, fallback to 'wrr' method"
time="2018-10-22T07:38:59Z" level=debug msg="Backend backend-alertmanager-prometheus: no load-balancer defined, fallback to 'wrr' method"
time="2018-10-22T07:38:59Z" level=debug msg="Backend backend-cadvisor-cadvisor: no load-balancer defined, fallback to 'wrr' method"
time="2018-10-22T07:38:59Z" level=debug msg="Backend backend-grafana-prometheus: no load-balancer defined, fallback to 'wrr' method"
time="2018-10-22T07:38:59Z" level=debug msg="Backend backend-node-exporter-node-exporter: no load-balancer defined, fallback to 'wrr' method"
time="2018-10-22T07:38:59Z" level=debug msg="Configuration received from provider docker: {\"backends\":{\"backend-alertmanager-prometheus\":{\"servers\":{\"server-alertmanager-4cb3f945666077858f35d4018dd76eaa\":{\"url\":\"http://172.18.0.3:9093\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-cadvisor-cadvisor\":{\"servers\":{\"server-cadvisor-e42dd141b28258fda36b39d92119a422\":{\"url\":\"http://172.19.0.2:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-grafana-prometheus\":{\"servers\":{\"server-grafana-9733510c04552af6363b3428626e564f\":{\"url\":\"http://172.18.0.5:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-node-exporter-node-exporter\":{\"servers\":{\"server-node-exporter-0fee9f2051d33a490d9f32e9bacbc888\":{\"url\":\"http://172.19.0.3:9100\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"backend-traefik-traefik\":{\"servers\":{\"server-traefik-780f230448df16d66397c0c29cebc062\":{\"url\":\"http://172.18.0.2:80\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"frontend-Host-alertmanager-monitor-local-example-com-3\":{\"entryPoints\":[\"https\"],\"backend\":\"backend-alertmanager-prometheus\",\"routes\":{\"route-frontend-Host-alertmanager-monitor-local-example-com-3\":{\"rule\":\"Host:alertmanager.monitor.local.example.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-metrics-monitor-local-example-com-Path-cadvisor-ReplacePath-metrics-4\":{\"entryPoints\":[\"metrics\"],\"backend\":\"backend-cadvisor-cadvisor\",\"routes\":{\"route-frontend-Host-metrics-monitor-local-example-com-Path-cadvisor-ReplacePath-metrics-4\":{\"rule\":\"Host:metrics.monitor.local.example.com;Path:/cadvisor;ReplacePath:/metrics\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-metrics-monitor-local-example-com-Path-node-exporter-ReplacePath-metrics-1\":{\"entryPoints\":[\"metrics\"],\"backend\":\"backend-node-exporter-node-exporter\",\"routes\":{\"route-frontend-Host-metrics-monitor-local-example-com-Path-node-exporter-ReplacePath-metrics-1\":{\"rule\":\"Host:metrics.monitor.local.example.com;Path:/node-exporter;ReplacePath:/metrics\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-monitor-local-example-com-2\":{\"entryPoints\":[\"https\"],\"backend\":\"backend-grafana-prometheus\",\"routes\":{\"route-frontend-Host-monitor-local-example-com-2\":{\"rule\":\"Host:monitor.local.example.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"frontend-Host-traefik-traefik-monitor-local-example-com-0\":{\"entryPoints\":[\"http\"],\"backend\":\"backend-traefik-traefik\",\"routes\":{\"route-frontend-Host-traefik-traefik-monitor-local-example-com-0\":{\"rule\":\"Host:traefik.traefik.monitor.local.example.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2018-10-22T07:39:00Z" level=info msg="Server configuration reloaded on :80"
time="2018-10-22T07:39:00Z" level=info msg="Server configuration reloaded on :443"
time="2018-10-22T07:39:00Z" level=info msg="Server configuration reloaded on :8484"
time="2018-10-22T07:39:00Z" level=info msg="Server configuration reloaded on :8080"
time="2018-10-22T07:39:00Z" level=debug msg="Creating entry point redirect http -> https"
time="2018-10-22T07:39:02Z" level=debug msg="Wiring frontend frontend-Host-alertmanager-monitor-local-example-com-3 to entryPoint https"
time="2018-10-22T07:39:02Z" level=debug msg="Creating backend backend-alertmanager-prometheus"
time="2018-10-22T07:39:02Z" level=debug msg="Adding TLSClientHeaders middleware for frontend frontend-Host-alertmanager-monitor-local-example-com-3"
time="2018-10-22T07:39:02Z" level=debug msg="Creating load-balancer wrr"
time="2018-10-22T07:39:02Z" level=debug msg="Creating server server-alertmanager-4cb3f945666077858f35d4018dd76eaaat http://172.18.0.3:9093 with weight 1"
time="2018-10-22T07:39:02Z" level=debug msg="Creating route route-frontend-Host-alertmanager-monitor-local-example-com-3 Host:alertmanager.monitor.local.example.com"
time="2018-10-22T07:39:02Z" level=debug msg="Wiring frontend frontend-Host-metrics-monitor-local-example-com-Path-cadvisor-ReplacePath-metrics-4 to entryPoint metrics"
time="2018-10-22T07:39:02Z" level=debug msg="Creating backend backend-cadvisor-cadvisor"
time="2018-10-22T07:39:02Z" level=debug msg="Adding TLSClientHeaders middleware for frontend frontend-Host-metrics-monitor-local-example-com-Path-cadvisor-ReplacePath-metrics-4"
time="2018-10-22T07:39:02Z" level=debug msg="Creating load-balancer wrr"
time="2018-10-22T07:39:02Z" level=debug msg="Creating server server-cadvisor-e42dd141b28258fda36b39d92119a422 at http://172.19.0.2:8080 with weight 1"
time="2018-10-22T07:39:02Z" level=debug msg="Creating route route-frontend-Host-metrics-monitor-local-example-com-Path-cadvisor-ReplacePath-metrics-4 Host:metrics.monitor.local.example.com;Path:/cadvisor;ReplacePath:/metrics"
time="2018-10-22T07:39:02Z" level=debug msg="Wiring frontend frontend-Host-metrics-monitor-local-example-com-Path-node-exporter-ReplacePath-metrics-1 to entryPoint metrics"
time="2018-10-22T07:39:02Z" level=debug msg="Creating backend backend-node-exporter-node-exporter"
time="2018-10-22T07:39:02Z" level=debug msg="Adding TLSClientHeaders middleware for frontend frontend-Host-metrics-monitor-local-example-com-Path-node-exporter-ReplacePath-metrics-1"
time="2018-10-22T07:39:02Z" level=debug msg="Creating load-balancer wrr"
time="2018-10-22T07:39:02Z" level=debug msg="Creating server server-node-exporter-0fee9f2051d33a490d9f32e9bacbc888 at http://172.19.0.3:9100 with weight 1"
time="2018-10-22T07:39:02Z" level=debug msg="Creating route route-frontend-Host-metrics-monitor-local-example-com-Path-node-exporter-ReplacePath-metrics-1 Host:metrics.monitor.local.example.com;Path:/node-exporter;ReplacePath:/metrics"
time="2018-10-22T07:39:02Z" level=debug msg="Wiring frontend frontend-Host-monitor-local-example-com-2 to entryPoint https"
time="2018-10-22T07:39:02Z" level=debug msg="Creating backend backend-grafana-prometheus"
time="2018-10-22T07:39:02Z" level=debug msg="Adding TLSClientHeaders middleware for frontend frontend-Host-monitor-local-example-com-2"
time="2018-10-22T07:39:02Z" level=debug msg="Creating load-balancer wrr"
time="2018-10-22T07:39:02Z" level=debug msg="Creating server server-grafana-9733510c04552af6363b3428626e564f at http://172.18.0.5:3000 with weight 1"
time="2018-10-22T07:39:02Z" level=debug msg="Creating route route-frontend-Host-monitor-local-example-com-2Host:monitor.local.example.com"
time="2018-10-22T07:39:02Z" level=debug msg="Wiring frontend frontend-Host-traefik-traefik-monitor-local-example-com-0 to entryPoint http"
time="2018-10-22T07:39:02Z" level=debug msg="Creating backend backend-traefik-traefik"
time="2018-10-22T07:39:02Z" level=debug msg="Adding TLSClientHeaders middleware for frontend frontend-Host-traefik-traefik-monitor-local-example-com-0"
time="2018-10-22T07:39:02Z" level=debug msg="Creating load-balancer wrr"
time="2018-10-22T07:39:02Z" level=debug msg="Creating server server-traefik-780f230448df16d66397c0c29cebc062 at http://172.18.0.2:80 with weight 1"
time="2018-10-22T07:39:02Z" level=debug msg="Creating route route-frontend-Host-traefik-traefik-monitor-local-example-com-0 Host:traefik.traefik.monitor.local.example.com"
time="2018-10-22T07:39:02Z" level=info msg="Server configuration reloaded on :8080"
time="2018-10-22T07:39:02Z" level=info msg="Server configuration reloaded on :80"
time="2018-10-22T07:39:02Z" level=info msg="Server configuration reloaded on :443"
time="2018-10-22T07:39:02Z" level=info msg="Server configuration reloaded on :8484"
time="2018-10-22T07:39:02Z" level=debug msg="Try to challenge certificate for domain [monitor.local.example.com] founded in Host rule"
time="2018-10-22T07:39:02Z" level=debug msg="Try to challenge certificate for domain [alertmanager.monitor.local.example.com] founded in Host rule"
time="2018-10-22T07:39:02Z" level=debug msg="Looking for provided certificate(s) to validate [\"alertmanager.monitor.local.example.com\"]..."
time="2018-10-22T07:39:02Z" level=debug msg="No ACME certificate generation required for domains [\"alertmanager.monitor.local.example.com\"]."
time="2018-10-22T07:39:02Z" level=debug msg="Looking for provided certificate(s) to validate [\"monitor.local.example.com\"]..."
time="2018-10-22T07:39:02Z" level=debug msg="No ACME certificate generation required for domains [\"monitor.local.example.com\"]."
time="2018-10-22T07:39:03Z" level=debug msg="Building ACME client..."
time="2018-10-22T07:39:03Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory"
time="2018-10-22T07:39:03Z" level=info msg=Register...
time="2018-10-22T07:39:03Z" level=info msg="legolog: [INFO] acme: Registering account for email@example.com"
time="2018-10-22T07:39:03Z" level=debug msg="Using DNS Challenge provider: digitalocean"
time="2018-10-22T07:39:03Z" level=info msg="legolog: [INFO] [*.monitor.local.example.com, monitor.local.example.com] acme: Obtaining bundled SAN certificate"
time="2018-10-22T07:39:04Z" level=info msg="legolog: [INFO] [*.monitor.local.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/e_z7lOUvFfKROKTTQ05XenpudAPpfpBRuR7r5EKrXI4"
time="2018-10-22T07:39:04Z" level=info msg="legolog: [INFO] [monitor.local.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/khfXsaShFq29VdjRLJ5VO2ZH62SnnIf3ovjHDi4lUf4"
time="2018-10-22T07:39:04Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Could not find solver for: http-01"
time="2018-10-22T07:39:04Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Could not find solver for: tls-alpn-01"
time="2018-10-22T07:39:04Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Preparing to solve DNS-01"
time="2018-10-22T07:39:04Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Preparing to solve DNS-01"
time="2018-10-22T07:39:04Z" level=error msg="Error obtaining certificate retrying in 439.762423ms"
time="2018-10-22T07:39:04Z" level=info msg="legolog: [INFO] [*.monitor.local.example.com, monitor.local.example.com] acme: Obtaining bundled SAN certificate"
time="2018-10-22T07:39:05Z" level=info msg="legolog: [INFO] [*.monitor.local.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/e_z7lOUvFfKROKTTQ05XenpudAPpfpBRuR7r5EKrXI4"
time="2018-10-22T07:39:05Z" level=info msg="legolog: [INFO] [monitor.local.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/khfXsaShFq29VdjRLJ5VO2ZH62SnnIf3ovjHDi4lUf4"
time="2018-10-22T07:39:05Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Could not find solver for: http-01"
time="2018-10-22T07:39:05Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Could not find solver for: tls-alpn-01"
time="2018-10-22T07:39:05Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Preparing to solve DNS-01"
time="2018-10-22T07:39:05Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Preparing to solve DNS-01"
time="2018-10-22T07:39:05Z" level=error msg="Error obtaining certificate retrying in 415.556797ms"
time="2018-10-22T07:39:05Z" level=info msg="legolog: [INFO] [*.monitor.local.example.com, monitor.local.example.com] acme: Obtaining bundled SAN certificate"
time="2018-10-22T07:39:06Z" level=info msg="legolog: [INFO] [*.monitor.local.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/e_z7lOUvFfKROKTTQ05XenpudAPpfpBRuR7r5EKrXI4"
time="2018-10-22T07:39:06Z" level=info msg="legolog: [INFO] [monitor.local.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/khfXsaShFq29VdjRLJ5VO2ZH62SnnIf3ovjHDi4lUf4"
time="2018-10-22T07:39:06Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Could not find solver for: http-01"
time="2018-10-22T07:39:06Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Could not find solver for: tls-alpn-01"
time="2018-10-22T07:39:06Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Preparing to solve DNS-01"
time="2018-10-22T07:39:06Z" level=info msg="legolog: [INFO] [monitor.local.example.com] acme: Preparing to solve DNS-01"
time="2018-10-22T07:39:06Z" level=error msg="Error obtaining certificate: acme: Error -> One or more domains had a problem:\n[monitor.local.example.com] error presenting token: digitalocean: could not determine zone for domain: 'monitor.local.example.com'. unexpected response code 'NOTIMPL' for monitor.local.example.com.\n"
time="2018-10-22T07:39:06Z" level=error msg="Unable to obtain ACME certificate for domains \"*.monitor.local.example.com,monitor.local.example.com\" : unable to generate a certificate for the domains [*.monitor.local.example.com monitor.local.example.com]: acme: Error -> One or more domains had a problem:\n[monitor.local.example.com] error presenting token: digitalocean: could not determine zone for domain: 'monitor.local.example.com'. unexpected response code 'NOTIMPL' for monitor.local.example.com.\n"
@ldez
Copy link
Member

ldez commented Oct 22, 2018 •
edited

The error comes when Lego (the lib we use to manage Let's Encrypt) try to find the zone for the FQDN.
To find the zone, we do a DNS query to the name servers.

This error comes when the DNS server don't implement SOA query.

https://tools.ietf.org/html/rfc1035

4 Not Implemented - The name server does not support the requested kind of query.

So I suppose you defined some custom DNS server as name server and maybe you are using a DNS proxy or a corporate DNS proxy.

Could you give more information about your environment?

@dpsarrou
Copy link
Author

Thanks for the fast response during the Great Github Outage :)

Yes I also checked the source code of Lego library and found the relevant method. I was not sure however why the server would return a NOTIMPL and that is the reason I tried to use certbot to see if that works. How come certbot works but not Lego..? I assume there is something specific Lego needs..?

I'm not sure exactly what you mean by custom DNS server, as I'm using Digital Ocean's nameservers. The environment is described below:

On a typical macbook (high sierra) I have a local vagrant machine that runs docker and traefik.
In the vm I have installed a prometheus stack in docker containers (Grafana, prometheus, alertmanager, along with a couple of exporters).

I am using a domain (lets say example.com) that is managed by Digital Ocean nameservers.
I have setup a couple of A records pointing to the vm, some of those as an example:

  • local.example.com -> 192.168.33.126
  • *.local.example.com -> 192.168.33.126
  • *.monitor.local.example.com -> 192.168.33.126

I'm not using any custom DNS server or proxy, all dns queries go directly to DO nameservers which resolve to the local IP of the virtual machine.

Thanks again for looking into this!

@juliens juliens added area/acme kind/bug/possible a possible bug that needs analysis before it is confirmed or fixed. priority/P3 maybe and removed status/0-needs-triage labels Oct 23, 2018
@traefiker
Copy link
Contributor

Thanks for opening this issue!

We need further information to better understand the problem you're facing 🤔

Could you please join us on our Slack workspace and reach out to us on the (#support channel)?

We're looking forward to talking to you there!

@dpsarrou
Copy link
Author

Many thanks to the team for helping to resolve this!

In short:
Using --natdnshostresolver1 "on" in the virtual machine is not compatible with Lego library that needs to issue a SOA request.

The root cause was identified by issuing a dig <domain> SOA inside the virtual machine that did not return a SOA answer, while issuing the same command on the host returns the expected result.

@ldez ldez mentioned this issue Oct 24, 2018
Merged
2 tasks
@traefik traefik locked and limited conversation to collaborators Sep 1, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area/acme kind/bug/possible a possible bug that needs analysis before it is confirmed or fixed. priority/P3 maybe status/5-frozen-due-to-age
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@juliens @ldez @dpsarrou @traefiker