https: automatic tls certificate (or in plugin/tls)

[zzxap]

start a https server
https://www.mydomain.com {
bind myip
hosts {
10.6.6.2 sms.service
10.6.6.3 search.service
}
}
how to set https certificate？

[miekg]

[ Quoting <notifications@github.com> in "[coredns/coredns] start a https ser..." ]

start a https server https://www.mydomain.com { bind myip hosts { 10.6.6.2 sms.service 10.6.6.3 search.service } } how to set https certificate？

this has to be done manually. I also think Let's Encrypt; or generally the CA industry doesn't handout certs for IP addresses; so you need some kind of domain as well. I'm eyeing caddy 2 ways of setting and getting certs as this has been automated already (cc @mholt)

[mholt]

Hi - anything I can help with?

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] start a https..." ]

Hi - anything I can help with?

wondering if caddy's acme code can be tranferred/used from coredns to perform a simular function, but now for a DNS server.

[mholt]

Yep, totally.

In Caddy 2, there is the tls app which can manage the certs for you.

Or if you just need cert management features in any other Go program, you can use CertMagic: https://github.com/mholt/certmagic -- this is the same library that Caddy uses. (Caddy 2 uses it also, but Caddy 2 has the tls app which is a more centralized way to do it.)

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] start a https..." ]

Yep, totally. In Caddy 2, there is the `tls` app which can manage the certs for you. Or if you just need cert management features in any other Go program, you can use CertMagic: https://github.com/mholt/certmagic -- this is the same library that Caddy uses.

thanks! Would be good to see what's easiest to integrate (at some point), but that's homework we need to do. /Miek

…

-- Miek Gieben

[mholt]

No problem, let me know if you want any more specific guidance or have any questions!

[polarathene]

It would be nice to see CoreDNS able to handle an ACME DNS challenge.

Not sure how much work that is to implement, and I assume a CoreDNS plugin would be required in additional to a separate one for software like Caddy to use (caddy-dns plugins which are apparently simple to implement using Go libdns, using with Caddy guide here). go-acme/lego has quite a list of supported providers, but most are not providers that you can self-host (I recognize PowerDNS as one of the only ones there?).

[miekg]

I agree tls shoudl do acme

tls {
    acme <domainname>
}

or something as config and then it does the let's encrypt dance. Setting up those keys/certs manually is tedious. We do need a place to write the keys/certs though...

[mholt]

@polarathene

It would be nice to see CoreDNS able to handle an ACME DNS challenge.

What do you mean by this, exactly? Technically "handling" the DNS challenge is just a matter of setting a TXT record, which CoreDNS is more than capable of already.

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] https: automa..." ]

@polarathene It would be nice to see CoreDNS able to handle an ACME DNS challenge. What do you mean by this, exactly? Technically "handling" the DNS challenge is just a matter of setting a TXT record, which CoreDNS is more than capable of already.

ah ok. It's because with DoH - which is HTTP/2 - we need a full blown "get cert from let's encrypt" (in a way that by-passes DNS, if it all possible)

[polarathene]

I'm not sure about Lets Encrypt, but I have used acmebot(python ACME client) with bindtool which it calls to update my DNS records on CoreDNS via a zone file template, one part is an {{acme}} section that it reads JSON from acmebot to get the domain name and TXT record to generate a record for.

The only issue with that setup is the file plugin can't update by file watch event, has to wait until the reload timer has elapsed (which checks for SOA record update to know if it should update records).

acmebot also supports another way to update DNS records that I imagine most other DNS services are also doing with their DNS plugins for various ACME clients, "RFC 2136 dynamic DNS updates using nsupdate". I know @miekg has spoken of wanting to implement that before, perhaps a plugin for temporal records like ACME TXT DNS challenge is a good case?

[polarathene]

BTW, if it would be helpful, I presently have a docker-compose setup for my current approach. It uses

• CoreDNS(provides several DNS records, supports the ACME DNS challenge, enables an FQDN to be used)
• Smallstep (private CA, Caddy uses the same lib internally afaik)
• acmebot + bindtool
• A mail server (which I register two certificates for RSA and ECDSA)

I just need to bring the containers up and it automates the whole provisioning/setup. I used it for testing a TLS security update for a PR I'm contributing to said mail server project.

[polarathene]

ah ok. It's because with DoH - which is HTTP/2 - we need a full blown "get cert from let's encrypt" (in a way that by-passes DNS, if it all possible)

Oh.. was this about TLS certs for CoreDNS to use for it's own needs? My mistake 😅

Still if you go forward with it, ideally it's not tied to LetsEncrypt like some ACME clients are. Smallstep is a nice private CA / ACME provisioner server, it'd be great if that can also be used for getting certs from over ACME.acmebot handles this really nicely config wise btw. It could be used to provision a TLS cert for any domain that CoreDNS needs, just need to configure CoreDNS to use the certificate.

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] https: automa..." ]

ah ok. It's because with DoH - which is HTTP/2 - we need a full blown "get cert from let's encrypt" (in a way that by-passes DNS, if it all possible) Oh.. was this about TLS certs for CoreDNS to use for it's own needs? My mistake 😅 Still if you go forward with it, ideally it's not tied to LetsEncrypt like some ACME clients are. Smallstep is a nice private CA / ACME provisioner server, it'd be great if that can also be used for getting certs from over ACME.

yep, initially this would be for let's encrypt, but others can be added. Getting a special TXT record served by a plugin is probably the easiest thing to implement.

[balboah]

Hey guys. Now I'm having an issue with letsencrypt as well.
In my setup, I'm using Kubernetes and cert-manager from jetstack. This works great, it can generate the certificate via DNS api's on my provider.
The certificate will be mounted for coredns to use, I just provide tls /etc/coredns-tls/tls.crt /etc/coredns-tls/tls.key.

However when the certificate gets cycled every 90 days, the file will change but coredns won't notice. I was thinking it would be enough with a reload option in the tls plugin.
As many parts may use the cert, I believe an Instance restart similar to how the reload plugin works would be the most straight way to do it?

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] https: automa..." ]

Hey guys. Now I'm having an issue with letsencrypt as well. In my setup, I'm using Kubernetes and cert-manager from jetstack. This works great, it can generate the certificate via DNS api's on my provider. The certificate will be mounted for coredns to use, I just provide tls /etc/ coredns-tls/tls.crt /etc/coredns-tls/tls.key. However when the certificate gets cycled every 90 days, the file will change but coredns won't notice. I was thinking it would be enough with a reload option in the tls plugin. As many parts may use the cert, I believe an Instance restart similar to how the reload plugin works would be the most straight way to do it?

I thinks a better to keep the reloading contained in the reload plugin. What _could_ be is to add an option there to reload every X hours + jitter, so any changes that live outside of the main config file are picked up.

[balboah]

Having a reload ever X hours would work, but it would have to restart a lot more often than the TLS cert file changes since it can't know the difference between the certificate expiry and the start time of coredns.
The reload plugin already refers to auto to be able to reload zonefiles, that's why I thought it would be natural that tls should handle reloading of cert files. And then only one reload would be necessary.

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] https: automa..." ]

Having a reload ever X hours would work, but it would have to restart a lot more often than the TLS cert file changes since it can't know the difference

so?

between the certificate expiry and the start time of coredns. The reload plugin already refers to auto to be able to reload zonefiles, that's why I thought it would be natural that tls should handle reloading of cert files.

yeah, but I'm not a fan of every plugin doing this by itself

[balboah]

Doesn't this create unnecessary interruptions to client requests? You would prefer to have one global option that restarts for example every hour instead of only when Corefile, tls, or zonefiles change? Or that one reload loop should run last in line and handle the checking of all the external resources of other plugins?
I saw you also need something similar with #4406

[miekg]

shouldn't be an issue. Think it is better to focus on having builtin acme support (or whatever the protocol is)

[balboah]

Sure that would be convenient, but more complicated. And it wouldn't work when you have more than one instance. I would somehow need to share persistent storage between my coredns instances or move this stuff to another proxy a.k.a. ingress in front of coredns. This can already be solved with cert-manager. Just need to reload :)

[balboah]

But i guess this can be done from an external plugin as well.

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] https: automa..." ]

Sure that would be convenient, but more complicated. And it wouldn't work when you have more than one instance. I would somehow need to share persistent storage between my coredns instances or move this stuff to another proxy a.k.a. ingress in front of coredns. This can already be solved with cert-manager. Just need to reload :)

fair point, but I still think reloading via reload is simplest to add and neatly contains this functionality. (Could have done this for the host and (maybe) auto plugin as well)

[yongtang]

I think builtin acme support would be a nice feature.

Depending on the environment the share persistent storage could be easily available. For example NFS is available in may IT environment (on cloud such as AWS you have EFS). I think this is less of a concern and not necessarily tied to the acme support itself.

[miekg]

Caddy did stuff to support clusters retrieving certs. If that all is too complex we could still fallback to just reloading the certs (I mean that would be simplest to implement, with only more and more plugins doing so, but we could provide some infra for that)

…

On Fri, 22 Jan 2021, 19:49 Yong Tang, ***@***.***> wrote: I think builtin acme support would be a nice feature. Depending on the environment the share persistent storage could be easily available. For example NFS is available in may IT environment (on cloud such as AWS you have EFS). I think this is less of a concern and not necessarily tied to the acme support itself. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3460 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACWIW3S424U5DBN2VNEVPTS3HCFDANCNFSM4JON3IJQ> .

[yongtang]

/cc @greenpau

[yongtang]

As a reminder, the deadline for Google Summer of Code application is 04/13. We encourage the submission if haven't done so

/cc @jimil749

[jimil749]

Hi @yongtang @miekg! I'm sorry for not being active for last 2 weeks, I had planned to write the proposal earlier but unfortunately could not do so due to unavoidable circumstances. Since the GSoC deadline is in a couple of days it'd be difficult to write the proposal in such a short period. That being said, if the project is available post-gsoc, I would be happy to work w the community on this!

[yongtang]

@jimil749 Thanks for the update.

[yongtang]

All: At the moment this project is still open to GSoC applications. If anyone is interested, please submit through GSoC CNCF website. If we don't receive GSoC application for this project by 04/13, we will open this project to all community.

(Since CoreDNS has been a frequent participant of both Google's GSoC and Linux Foundation's CommunityBridge program, we may add this project to future GSoC or CommunityBridge if appropriate).

[yongtang]

We didn't receive GSoC application submission for this project (Don't know if we should have some PR to promote this?). Though the Linux Foundation's CommunityBridge program is coming up and is open to all student (similar to GSoC in format where student will receive a stipend upon success completion of the project). I am planning on redirect this project to CommunityBridge program. Submission is definitely encouraged.

[yongtang]

The pull request for LFX (formally Linux CommunityBridge) program has been created in cncf/mentoring#385

[chinzhiweiblank]

As assigned by LFX, I will write the RFC and submit it for approval. I also noticed that there is a RFC that is closed and would like to reference it.
@greenpau @yongtang

[yongtang]

@chinzhiweiblank Welcome and glad to see you getting started! I also sent you an email (to both ch...@....edu.sg and ch...@gmail.com). Let me know if you haven't receive it yet. /cc @greenpau

[yongtang]

For anyone that might be interested in ACME support in CoreDNS, as part of LFX program, RFC has been created in coredns/rfc#12

comments are welcomed.

[chinzhiweiblank]

Hi @zzxap , if you are still having problems with getting the tls certificate automatically, you could try out the plugin I created here. Feedback is welcomed.

[chinzhiweiblank]

Hi @polarathene and @balboah , I noticed you expressed interest in a CoreDNS ACME plugin. You could try out or review the plugin I created here. I would appreciate any feedback you have!

[chebro]

@yongtang i'm coming from the cncf gsoc 2022 ideas list. is this project still open for contributor proposals for gsoc 2022?

[yongtang]

@chebro Yes this project is for GSoC 2022 now and is still open. Contributor proposals are definitely welcomed!

[chebro]

@chebro Yes this project is for GSoC 2022 now and is still open. Contributor proposals are definitely welcomed!

sorry, i am a little confused about the status of this issue. after going through the comment chain, what i've gathered is this

1. jimil749 had previously proposed a solution but couldn't write the gsoc proposal in time.
2. chinzhiweiblank already wrote an rfc and (what looks like) an implementation for the same under the lfx program. (but the rfc was declared incomplete)

it would be helpful to know what is expected from a new contributor wanting to work on this issue, because it would make sense to avoid starting from scratch if there's progress had already been made.

[yongtang]

@chebro This project has been reserved for GSoC/LFX program in the past year. However, the past participant have either not complete the initial RFC, or have not been able to implement the RFC at the end of program. In other words, this project is back to the initial status.

The interest from community for automatic TLS certificate management remains high. As a result, coredns maintainers (me and @greenpau) have decided to re-open the project for this year's GSoC project (2022) and new contributors are welcomed.

CoreDNS is an open source community and this project is also open. New contributors are free to take necessary knowledge from past contributors (no need to start from scratch). Though you may want to reference past RFCs as a courtesy (we always want to give people credit even if the work is partial).

[mariuskimmina]

@yongtang I would like to work on this (also in the context of GSoC) but I am not sure I understood this fully.
I think I understand how ACME works for a webserver like Caddy - either by using the http-01 challenge and hosting a file on /well-kown/... or using the dns-01 challenge utilzing the API of a DNS provider to set a TXT Record.
I am not sure how to apply this to CoreDNS.
I also understand that CoreDNS, to serve DoT or DoH traffic, needs a valid certificate and we would like to automate obtaining such a cert and renewing it with ACME - So, now the part I don't understand yet: when using the dns-01 challenge should CoreDNS set this TXT Record or should it also utilize the API of a DNS provider? And what about the http-01 challenge, how would that work for a DNS Server such as CoreDNS, would we utilize the underlying Caddy server and open port 80 until the challenge is done? I am sorry if this should be obvious but it isn't to me.

[yongtang]

@mariuskimmina Sorry for the late reply as I was traveling the past several days. Basically we want to consider coredns as an independent system that does not relies on other software. In terms of ACME support, since CoreDNS is already a DNS server, as long as CoreDNS is server public DNS domains, then it is possible to allow Lets-Encrypt to validate through the very much same CoreDNS server to issue the certificate.

The scope of the project is to allow CoreDNS to get the certificate from Lets-Encrypt (or other vendor), and serving DoT or DoH traffic, without dependency of acme.sh or caddy, etc.

[mariuskimmina]

No problem @yongtang.
Okay, so CoreDNS would be running on e.g. dns.example.com and we would assume that CoreDNS is also the authoritative DNS for example.com and can fullfill the dns challenge (aka setting a TXT record at _acme-challenge.example.com)?
And the HTTP Challenge would then be completely out-of-scope?

[yongtang]

@mariuskimmina Yes. We only need to solve the issue of, when someone want to setup coredns and serve DoH/DoT traffic, there is no need to manually deal with certificate deploy/renew. Everything can be done by potentially add a line or two in Corefile.