Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ACME-DNS-API not pulling a certificate for itself #324

Open
protogenxl opened this issue Dec 22, 2022 · 9 comments
Open

ACME-DNS-API not pulling a certificate for itself #324

protogenxl opened this issue Dec 22, 2022 · 9 comments

Comments

@protogenxl
Copy link

Hi
I have been working on setting up a acme-dns and have ran into an issue where the web API is not pulling it's own let's encrypt cert. I feel like I am missing something simple but I am to far in to see what is behind me.

Any help would be appreciated

halibut.juggedfish.com will be playing the part of my public dns record
and 8.8.8.8 will be playing the part of my public ip address

ERROR

root@halibut ~]# systemctl status acme-dns -l
? acme-dns.service - Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely
   Loaded: loaded (/etc/systemd/system/acme-dns.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-12-22 09:49:24 EST; 46min ago
 Main PID: 737 (acme-dns)
   CGroup: /system.slice/acme-dns.service
           +-737 /usr/local/bin/acme-dns

Dec 22 09:49:24 halibut.juggedfish.com systemd[1]: Started Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.
Dec 22 09:49:24 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:24-05:00" level=info msg="Using config file" file=/etc/acme-dns/config.cfg
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55191: no certificate available for 'halibut.juggedfish.com'"
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55192: no certificate available for 'halibut.juggedfish.com'"
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55193: no certificate available for 'halibut.juggedfish.com'"
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55194: no certificate available for 'halibut.juggedfish.com'"

CONFIG

# ================= DO NOT MODIFY THIS FILE =================
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at NethServer official site: https://www.nethserver.org
#
#

[general]
# dns interface
listen = "10.110.144.53:1053"
# protocol, "udp", "udp4", "udp6" or "tcp", "tcp4", "tcp6"
protocol = "both"
# domain name to serve the requests off of
domain = "halibut.juggedfish.com"
# zone name server
nsname = "ns1.halibut.juggedfish.com"
# admin email address, where @ is substituted with .
nsadmin = "halibut@juggedfish.com"
# predefined records served in addition to the TXT
records = [
    # default A
    "halibut.juggedfish.com. A 8.8.8.8",
    # A
    "ns1.halibut.juggedfish.com. A 8.8.8.8",
    "ns2.halibut.juggedfish.com. A 8.8.8.8",
    # NS
    "halibut.juggedfish.com. NS ns1.halibut.juggedfish.com.",
    "halibut.juggedfish.com. NS ns2.halibut.juggedfish.com.",
]
# debug messages from CORS etc
debug = true

[database]
# Database engine to use, sqlite3 or postgres
engine = "sqlite3"
connection = "/etc/acme-dns/acme-dns.db"


[api]
# listen ip eg. 127.0.0.1
ip = "10.110.144.53"
# disable registration endpoint
disable_registration = false
# listen port, eg. 443 for default HTTPS
port = "443"
# possible values: "letsencrypt", "letsencryptstaging", "cert", "none"
tls = "letsencrypt"
# only used if tls = "letsencrypt"
acme_cache_dir = "/etc/acme-dns/api-certs"
# CORS AllowOrigins, wildcards can be used
corsorigins = [
    "*"
]
# use HTTP header to get the client ip
use_header = false
# header name to pull the ip address / list of ip addresses from
header_name = "X-Forwarded-For"


[logconfig]
# logging level: "error", "warning", "info" or "debug"
loglevel = "warning"
# possible values: stdout, TODO file & integrations
logtype = "stdout"
# file path for logfile TODO
# logfile = "./acme-dns.log"
# format, either "json" or "text"
logformat = "text"

PUBLIC DNS


A
ns1.halibut
8.8.8.8
15 minutes


A
ns2.halibut
8.8.8.8
15 minutes


CNAME
_acme-challenge.halibut
halibut.juggedfish.com
15 minutes


NS
halibut
ns2.halibut.juggedfish.com
15 minutes

NS
halibut
ns1.halibut.juggedfish.com
15 minutes
@webprofusion-chrisc
Copy link
Contributor

Are port 53 DNS queries against your instance working? You are listening on 1053 but will need to port forward this externally from 53 for normal DNS queries to work (you may already be doing that). I'd imagine if that doesn't work then it also won't be able to use itself to complete a DNS challenge for it's own cert. There was/used to be an http-01 challenge mode but I don't know the config to use that instead of DNS validation.

@novakele
Copy link

novakele commented Dec 23, 2022 •
edited

Hi @protogenxl, I ran into a similar issue.

In my case, I was running the acme-dns.service as a non-root user, and the user did not have write permission in his home directory. By default, the service uses WorkingDirectory=~.

Are you running the service as root?

EDIT
Does the user runner the service has write permissions in /etc/acme-dns/api-certs?

@protogenxl
Copy link
Author

@webprofusion-chrisc yes the DNS forward on my firewall seems to working correctly

2022-12-27 08:11:12 Allow 8.0.38.4 8.8.8.8 dns/udp 53597 53 6-WAN 1-Trusted ProxyAllow: DNS question match   (LetsEncryptDNS-proxy-00) DNS-Incoming.1 proc_id="dns-proxy" rc="590" msg_id="1DFF-000E" proxy_act="DNS-Incoming.1" rule_name="*" query_type="A" question="nS2.halibut.juggedfish.com" | Traffic

2022-12-27 08:11:12 Allow 8.0.38.4 8.8.8.8 dns/udp 53597 53 6-WAN 1-Trusted Allowed 82 55 (LetsEncryptDNS-proxy-00)  proc_id="firewall" rc="100" msg_id="3000-0148" dst_ip_nat="10.110.144.53" dst_port_nat="1053" | Traffic

@novakele I am running the service as acme-dns and permissions appear to be correct

[root@halibut ~]# cd /etc/acme-dns/api-certs
[root@halibut api-certs]# ls -lah
total 0
drwx------ 4  992 acme-dns 100 Dec 27 08:03 .
drwxrwxr-x 3 root acme-dns  60 Dec 21 06:17 ..
drwx------ 3  992 acme-dns  60 Dec 20 23:53 acme
drwx------ 2  992 acme-dns 142 Dec 27 08:03 locks
-rw------- 1  992 acme-dns   0 Dec 21 06:17 rw_test_288443979776093768
-rw------- 1  992 acme-dns   0 Dec 21 06:17 rw_test_9142020181166123590

@novakele
Copy link

It is strange that the owner uid (992) does not resolve to the user acme-dns.

Could you provide the output of id acme-dns? Should the uid of acme-dns be anything else than 992, that is your problem.

Here are the permissions for my instance:

root@lighthouse:~# tree -pufidg /var/lib/acme-dns/
[drwxr-xr-x acme-dns acme-dns]  /var/lib/acme-dns
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/acme
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users/<EMAIL>
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/certificates
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/certificates/acme-v02.api.letsencrypt.org-directory
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/certificates/acme-v02.api.letsencrypt.org-directory/<DOMAIN>
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/locks
[drwx------ acme-dns acme-dns]  /var/lib/acme-dns/api-certs/ocsp

Also, here is the output of the same commands you ran (I use /var/lib/acme-dns instead of /etc/acme-dns for the home directory):

root@lighthouse:~# cd /var/lib/acme-dns/api-certs/

root@lighthouse:/var/lib/acme-dns/api-certs# ls -lah
total 24K
drwx------ 6 acme-dns acme-dns 4.0K Dec 23 22:19 .
drwxr-xr-x 3 acme-dns acme-dns 4.0K Dec 23 23:31 ..
drwx------ 3 acme-dns acme-dns 4.0K Dec 23 22:19 acme
drwx------ 3 acme-dns acme-dns 4.0K Dec 23 22:19 certificates
drwx------ 2 acme-dns acme-dns 4.0K Dec 23 22:19 locks
drwx------ 2 acme-dns acme-dns 4.0K Dec 23 22:19 ocsp

@p3l1
Copy link

p3l1 commented Dec 31, 2022

See #315, I encountered similar problems. I am using the Dockerfile to run acme-dns.
The v0.8 release works just fine. In my opinion this is not an environment problem, but a problem of the software itself.

On my profile I've got an improved Dockerfile based on the v0.8 release.

@jeffsf
Copy link

jeffsf commented Feb 2, 2023 •
edited

I'm seeing similar behavior to what has been reported here and in #315 on a new installation in FreeBSD 13.1-RELEASE-p5 and the current upstream compiled and packaged acme-dns-1.0_3,1 (installed today).

  • DNS queries from outside the perimeter, directed at the public DNS name of the delegation, are answered as expected (both expecting a result as well as expecting NXDOMAIN) Edit: SOA record also confirmed served to outside request
  • Trying to connect to the HTTP API using curl results in a TLS handshake error
  • There are no suggestions in the logs of any issues
  • There do not appear to be any permission problems on the file system (other than acme-dns configured by the port author to be running as root)
time="2023-02-01T19:13:53-08:00" level=info msg="Using config file" file=/usr/local/etc/acme-dns/config.cfg
time="2023-02-01T19:13:53-08:00" level=info msg="Connected to database"
time="2023-02-01T19:13:53-08:00" level=debug msg="Adding new record to domain" domain=<DELEGATED_NS_NAME>. recordtype=A
time="2023-02-01T19:13:53-08:00" level=debug msg="Adding new record to domain" domain=<DELEGATED_NS_NAME>. recordtype=NS
time="2023-02-01T19:13:53-08:00" level=debug msg="Adding new record to domain" domain=<DELEGATED_NS_NAME>. recordtype=SOA
time="2023-02-01T19:13:53-08:00" level=info msg="Listening HTTPS" domain=<DELEGATED_NS_NAME> host="0.0.0.0:443"
time="2023-02-01T19:13:53-08:00" level=info msg="Listening DNS" addr= proto=tcp
time="2023-02-01T19:13:53-08:00" level=info msg="Listening DNS" addr= proto=udp
time="2023-02-01T19:14:36-08:00" level=debug msg="Answering question for domain" domain=should.fail.<MY_DOMAIN>. qtype=A rcode=NXDOMAIN
time="2023-02-01T19:15:11-08:00" level=info msg="http: TLS handshake error from <TEST_HOST_IP>:57801: no certificate available for '<DELEGATED_NS_NAME>'"

On a subsequent restart, I additionally get

time="2023-02-01T19:29:33-08:00" level=info msg="2023/02/01 19:29:33 [INFO][FileStorage:/var/db/acme-dns/api-certs] Lock for 'issue_cert_<DELEGATED_NS_NAME>' is stale (created: 2023-02-01 19:13:53.163243238 -0800 PST, last update: 2023-02-01 19:29:23.133675565 -0800 PST); removing then retrying: /var/db/acme-dns/api-certs/locks/issue_cert_<DELEGATED_NS_NAME>.lock"

sudo find var/db/acme-dns/api-certs/ -type d -exec ls -ld {} \;
drwx------  4 root  wheel  4 Feb  1 19:13 var/db/acme-dns/api-certs/
drwx------  2 root  wheel  3 Feb  1 19:13 var/db/acme-dns/api-certs/locks
drwx------  4 root  wheel  4 Feb  1 19:15 var/db/acme-dns/api-certs/acme
drwx------  3 root  wheel  3 Feb  1 19:13 var/db/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory
drwx------  3 root  wheel  3 Feb  1 19:13 var/db/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users
drwx------  2 root  wheel  4 Feb  1 19:13 var/db/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users/<NOTIFICATON_EMAIL>
drwx------  3 root  wheel  3 Feb  1 19:15 var/db/acme-dns/api-certs/acme/acme-staging-v02.api.letsencrypt.org-directory
drwx------  3 root  wheel  3 Feb  1 19:15 var/db/acme-dns/api-certs/acme/acme-staging-v02.api.letsencrypt.org-directory/users
drwx------  2 root  wheel  4 Feb  1 19:15 var/db/acme-dns/api-certs/acme/acme-staging-v02.api.letsencrypt.org-directory/users/<NOTIFICATON_EMAIL>

$ egrep -v '^#' usr/local/etc/acme-dns/config.cfg
[general]
protocol = "both"
domain = "<DELEGATED_NS_NAME>"
nsname = "<DELEGATED_NS_NAME>"
nsadmin = "<HOSTMASTER@DOMAIN.NAME>"
records = [
    # domain pointing to the public IP of your acme-dns server
    "<DELEGATED_NS_NAME>. A <DELEGATED_NS_PUBLIC_IP>",
    # specify that auth.example.org will resolve any *.auth.example.org records
    "<DELEGATED_NS_NAME>. NS <DELEGATED_NS_NAME>.",
]
debug = true

[database]
engine = "sqlite3"
connection = "/var/db/acme-dns/acme-dns.db"

[api]
ip = "0.0.0.0"
disable_registration = false
port = "443"
tls = "letsencrypt"
acme_cache_dir = "/var/db/acme-dns/api-certs"
notification_email = "<NOTIFICATON_EMAIL>"
corsorigins = [
    "*"
]
use_header = false
header_name = "X-Forwarded-For"

[logconfig]
loglevel = "debug"
logtype = "stdout"
logformat = "text"

@jeffsf
Copy link

jeffsf commented Dec 24, 2023

Confirmed still an issue in FreeBSD 13.2 and package acme-dns-1.0_12,1

No changes in behavior identified

@tigeli
Copy link

tigeli commented Feb 9, 2024

Might be related to this https://github.com/joohoi/acme-dns/issues/339#issuecomment-1935928848

@jeffsf
Copy link

jeffsf commented Feb 12, 2024 •
edited

I did try adding the CAA record as described on #339 with no change in behavior

Removing the entire /var/db/acme-dns/api-certs directory and allowing it to be recreated also did not change the behavior.

It appears that the query is coming in for the TXT record. The following appears to repeat periodically.

time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=_Acme-cHALlENgE.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=_AcMe-chaLLEnGe.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<f4...UUID...b1>.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=_aCME-CHALlENgE.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<f4...UUID...b1>.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<f4...UUID...b1>.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR

The requests for the TXT record come in from a variety of IP addresses, including one that reverse-resolves to outbound1g.letsencrypt.org and another to outbound2e.letsencrypt.org

There is no certificate present anywhere under /var/db/acme-dns/api-certs and openssl s_client confirms that there is no certificate available.

Edit: For clarity, the instance works through the API to have other clients renew certificates. It just isn't able to take care of its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jeffsf @webprofusion-chrisc @tigeli @novakele @p3l1 @protogenxl