LetsEncrypt wildcards #6
Comments
|
@chazer that's an excellent idea! I'm certainly open to the idea of a pull request. But I'll let you know one thing: wildcard requires dns-01 challenge, and that's very difficult to implement with the sslip.io setup: sslip is not your standard master/slave nameserver configuration; instead, it's four distinct master servers with common configurations (essentially a bash-script back-end). To dynamically add a TXT record for a specific entry would require a not-inconsiderable amount of work. And my experience with letsencrypt is that they're very rigorous with DNS (i.e. I probably won't be able to get away with adding the TXT record to merely one server — I'd need to add it to all four nameservers). I don't see an easy way to make it work, sorry. |
|
Ahh, but dns-01 does support the _acme-challenge record being a CNAME. In fact it most likely also supports dns delegation. So answering with an NS record would work (as long at there's a functioning DNS server like acme-dns running at the given IP address), eg:
In fact it's probably okay to do this for all subdomain records that start with an underscore, because underscore characters (“_”) MUST NOT be present in dNSName entries of certificates. It would be nice to forward all TXT/SRV/MX requests like this, but it's not valid to have CNAME and other types, and I don't think having NS and other types makes sense. Using the proposed scheme from above: these records would be supported as long as there's an underscore present, eg: asking for That doesn't really help if you need to put the TXT records at the root name though. You could run your own DNS server to answer all names and use
would allow https://192.168.0.1.ns.sslip.io and TXT (or any other type of record) for 192.168.0.1.ns.sslip.io as long as you're willing to run the DNS server at 192.168.0.1 to answer all queries. |
|
@normanr That's a clever solution, and if I understand it correctly the users would need to set up a DNS server at their IP address (e.g. 192.168.0.1) (Yes, we both know that's an RFC 1918 address & not routable, but it serves as an example). I'm hesitant to spend the time implementing this solution because if the user is going to go to the trouble of setting up a DNS server (which requires a certain amount of technical depth & discipline), then they might as well go all-in and register a domain (e.g. my-co.biz) & serve the domain's DNS records from their newly-set up DNS server and bypass the clunky sslip.io nomenclature; in other words 'cust1.my-co.biz' is a much better URL than 'cust1.192.168.0.1.sslip.io'. |
|
I was thinking this would more fit the scenario where the user doesn't want to register a domain and/or may not have the technical know-how to set up a DNS server. I was imagining scenarios where the LetsEncrypt integration (along with serving DNS replies) is mostly hidden from the user and the only information they need to confirm is their public IP address. They might need to set up port forwarding if they're behind NAT, but adding port 53 to the current requirement of 443 and 80 (or replacing 80 with 53) would be very minimal extra effort. For example I found sslip.io via webtorrent/webtorrent#1492 (comment) where there's a discussion about how to provision a certificate for https without involving the user. Of course this doesn't explain why supporting wildcard certs via DNS-01 would be useful (because if should be possible to implement the above using HTTP-01 on port 80). |
|
@normanr I don't think I fully understand your suggestion, but I would like to say that I'm very open to pull requests if you feel inclined to make the change you want. Here's the code that runs sslip.io: https://github.com/cunnie/bin/blob/master/pdns_pipe.sh . Yes, a shell-script. And here's the corresponding tests (believe it or not, also a bash script): https://github.com/cunnie/bin/blob/master/pdns_pipe_spec.sh Thanks for contributing! |
|
A use case for wildcard certs would be https://community.letsencrypt.org/t/how-to-create-certificates-for-a-development-environment-in-a-local-network/122910. If it were possible to get a cert for: *.external-ip.sslip.io, then internal-ip.external-ip.sslip.io could be used to access internal sites (assuming the issued SSL cert is installed correctly on internal-ip). |
|
That's an interesting use case. There's several more pieces that need to be put in place before implementing the wildcard certs. I'll think about the best way to do it. Now that I've moved to a hand-built DNS server, I have much more leeway than in the past (when this comment was initially proposed). I'm going to reopen this issue while I consider ways to accomplish this. |
In order to restore email service for the sslip.io domain, we need to return custom TXT records. The custom records are in the `xip.Customizations` variable. This lays the groundwork for ACMEv2 wildcard DNS, which, IIRC, works via TXT records. Drive-by: removed an unused constant, `MxHost`. That information is either in the `Customization` struct or generated on-the-fly. fixes: > Dear valued customer, We have disabled your domain sslip.io and all of its addresses. No emails will be received or sent for it. [#6]
Prior behavior was that the same trinity of NS records was returned for every NS query: - ns-aws.nono.io. - ns-azure.nono.io. - ns-gce.nono.io. This commit introduces a change in that behavior: IF the NS query includes the string `_acme-challenge.` AND the query has an embedded IP address THEN the NS record returned is the query with the `_acme-challenge.` stripped. For example: ``` dig +short ns _acme-challenge.104.155.144.4.sslip.io ``` Would return: ``` 104.155.144.4.sslip.io. ``` This is an attempt to enable [DNS-01](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) challenge for wildcard certs from Let's Encrypt or other CAs (Certificate Authorities). Note that the embedded IP address would need to be routable (NOT 10.x 172.16-31.x, or 192.168.x). Note that you would also need to run a DNS server such as [acme-dns](https://github.com/joohoi/acme-dns) at that address. Thanks @normanr ! [#6]
|
I apologize if it's not appropriate to post here, but I just started using this project for the local development use case. Perhaps it could be used for inspiration. In addition to the the dynamic ns trick, it gets wildcard certs from let's encrypt and serve them to /keys. It's obviously not secure, but is very easy to use. |
|
Hi @jpambrun , Thank you for bringing Corollarium's localtls to my attention—yes, it's something along the lines that I've been working on. Sure, there are cosmetic differences: I plan on using Golang+Docker, they're using Python, but the general idea is the same. I'll update this issue when I have something working. |
This small DNS server only returns one type of record, a TXT record, meant to be a token assigned by a certificate authority (e.g. Let's Encrypt) to verify domain ownership. The TXT record will be updateable by an API endpoint on the webserver (same executable as the DNS server), but I haven't yet written that portion. Drive-by: in our _other_ (main) sslip.io DNS server, I changed `break` → `continue` in the main loop. Had we gotten a malformed UDP packet, we would have exited, but now we continue to the next packet. Exiting is not that big a deal—`monit` would have restarted the server—but moving on to the next packet is a more robust approach. [#6]
The purpose of this commit is to enable Let's Encrypt DNS-01 challenges for wildcard certificates. To accomplish that, we'd like to delegate queries for ALL types (e.g. NS, SOA, A, AAAA) to the IP address of that server. For example, any query for `_acme-challenge.52-0-56-137.sslip.io` would be delegated to the DNS server `52-0-56-137.sslip.io` (whose IP address 52.0.56.137 would be supplied as well). Thanks @normanr ! On a personal note, I feel the code is getting bloated again. Also, I'm inconsistent with my parameters: `NSResponse()`, for example, has arguments which it mutates (`response`), and which are returned (`logMessage`). This offends my esthetics. [#6]
We are pleased to announce that these instructions finally work. [#6]
|
FYI, I've written instructions for procuring wildcard certificates for |
|
Yeah, that would work. Few comments:
Let's encrypt certs expire every 90 days and this process is a bit tedious.
I ended up setting up localtls. There is a couple of manual steps to get
the certs as well, but their instructions are slightly more streamlined.
Ideally this would automagical a la caddy. Having a top level wildcard is a
bit more convenient than 192.168.0.1.32-34-34-34.sslip.io (tho I admit it's
a neat trick) especially since I will have to change the IP when renewing
the certs.
I really like that subdomain.prefix-22-22-22-22.suffix.sslio.io resolve to
22.22.22.22. I had to modify localtls to achieve that.
Localtls has a http server bundled to serve the certs under /keys. It's
obviously super insecure, but very convenient for testing. If you could do
the same thing on sslip.io for the top level that would be awesome.
Making a single go binary with this DNS server, acme challenge, SSL cert
serving would be epic.
…On Wed, Jan 20, 2021, 7:50 PM Brian Cunnie ***@***.***> wrote:
@normanr <https://github.com/normanr> @jpambrun
<https://github.com/jpambrun>
FYI, I've written instructions
<https://github.com/cunnie/sslip.io/blob/master/docs/wildcard.md> for
procuring wildcard certificates for sslip.io domains. If you have time to
review them & have suggestions, I'm all ears. Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFPMMB6FLLZD7YX2VANF63S2526PANCNFSM4EVT5WKQ>
.
|
|
Thanks @jpambrun for the most-excellent suggestions—I'll see what I can do to make it even more streamlined. |
I don't think that there's a secure way to serve any keys at sslip.io - because then sslip.io has to request and maintain them etc. It's far more secure to make the client do the domain validation, key maintenance, etc. Providing NS records for About the only way to make it secure would be to add user account management, etc, and and at that point you've created another Dynamic DNS service. (I think this would be an awesome feature for existing Dynamic DNS services to provide). |
|
You can have both.. sslip.io can provide a insecure *.sslip.io cert that
can be used "securely" for testing on private networks, and you can use the
yy.yy.yy.yy.xx-xx-xx-xx.sslip.io trick on your own server with your secret
certs.
…On Thu, Jan 21, 2021, 5:22 PM Norman Rasmussen ***@***.***> wrote:
Localtls has a http server bundled to serve the certs under /keys. It's
obviously super insecure, but very convenient for testing. If you could do
the same thing on sslip.io for the top level that would be awesome.
I don't think that there's a secure way to serve any keys at sslip.io -
because then sslip.io has to request and maintain them etc. It's far more
secure to make the client do the domain validation, key maintenance, etc.
Providing NS records for _acme-challenge enables exactly that - you have
to control the external IP address in order to be able to reply for
_acme-challenge, and it's a once off change for sslip.io to implement.
About the only way to make it secure would be to add user account
management, etc, and and at that point you've created another Dynamic DNS
service. (I think this would be an awesome feature for existing Dynamic DNS
services to provide).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFPMMFY5VBTQBKIMVGS6Q3S3CSJNANCNFSM4EVT5WKQ>
.
|
I think it would be nice to have some simpler instructions as an introduction/overview. Basically it comes down to:
There are many different ways of fulfilling these requirements:
There are public services that have free DNS hosting (some for a limited number of domains), and I suspect that those will be a much easier way to get started (compared to setting up a docker container). |
No you can't, because anywhere that issues certificates will revoke them if the private keys are publicly available (because they can be misused in too many ways).
Note: the format is actually |
|
I have a DNS server like sslip serving the keys under /keys and they still
haven't been revoked.. I doubt let's encrypt would prevent me from getting
new ones. I suppose it could be an issue with a high traffic site sslip.io,
but then I could just run myself on a small VPS..
You are right about the dots that should have been dashes.
…On Thu, Jan 21, 2021, 5:54 PM Norman Rasmussen ***@***.***> wrote:
You can have both.. sslip.io can provide a insecure *.sslip.io cert that
can be used "securely" for testing on private networks,
No you can't, because anywhere that issues certificates will revoke them
if the private keys are publicly available (because they can be misused in
too many ways).
and you can use the yy.yy.yy.yy.xx-xx-xx-xx.sslip.io trick on your own
server with your secret certs.
Note: the format is actually
prefix1-yy-yy-yy-yy.prefix2-xx-xx-xx-xx.sslip.io, because certificate
wildcards only match at the same level.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFPMMGYSBARTZVSNUBYEQ3S3CWCRANCNFSM4EVT5WKQ>
.
|
That's only because it's a private server and the endpoint isn't well known. The only thing you need to request revocation of a certificate is access to the private key, so anyone that can get the private key can revoke the certificate.
Let's encrypt won't prevent you from getting a new certificate with new keys, but if they wish to remain a member of the CA/Browser Forum then they are required to revoke existing certificates if the private key is compromised. It's a bit unclear if a freely shared private key should be considered compromised or not (see https://news.ycombinator.com/item?id=10184866 for more details).
Right, which is part of what this issue is trying to solve (safely issuing certs for sub-domains of a high traffic site). If you're willing to pay for a VPS, then you could also just by a domain name, and you wouldn't need sslip.io. |
|
My personal need is really a way get a FQDN and valid certs for hosts on
private networks (I.e. that can't handle the normal let's encrypt
challenge). Wildcard certs with subdomain that resolves dynamic IP is
perfect for that. I don't mind paying or running my own VM.
…On Thu, Jan 21, 2021, 8:31 PM Norman Rasmussen ***@***.***> wrote:
I have a DNS server like sslip serving the keys under /keys and they still
haven't been revoked.
That's only because it's a private server and the endpoint isn't well
known. The only thing you need to request revocation of a certificate is
access to the private key, so anyone that can get the private key can
revoke the certificate.
I doubt let's encrypt would prevent me from getting new ones.
Let's encrypt won't prevent you from getting a new certificate with new
keys, but if they wish to remain a member of the CA/Browser Forum then they
are required to revoke existing certificates if the private key is
compromised. It's a bit unclear if a freely shared private key should be
considered compromised or not (see
https://news.ycombinator.com/item?id=10184866 for more details).
I suppose it could be an issue with a high traffic site sslip.io, but
then I could just run myself on a small VPS.
Right, which is part of what this issue is trying to solve (safely issuing
certs for sub-domains of a high traffic site). If you're willing to pay for
a VPS, then you could also just by a domain name, and you wouldn't need
sslip.io.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFPMMHX7UTBZLNDNLX73PDS3DIO7ANCNFSM4EVT5WKQ>
.
|
I've had certificates revoked on two occasions. The first was when I published the private key to *.sslip.io wildcard certificate. It was an unpleasant experience, and I had to promise the CTO of Sectigo (then Comodo) that I would not share the private key outside myself & the company at which I worked, at which point he agreed to issue me a second certificate. I'm extremely reluctant to do anything which will put me on the the wrong side of the certificate authorities. If other people want to fight that battle, it's fine with me, but I don't want to become a three-strikes loser. |
Interesting. I assume you'd prefer something along the lines of That would require some thought. And then some work. |
Thanks @normanr , that's pure gold. I'll take a second pass at the instructions. I'll follow the template that you described, and leave the instructions I initially wrote as a very specific example of procuring wildcard certificates. |
|
Maybe I wasn't clear. I am currently using the localtls project on a VPS.
It already does what you suggested. It gets wildcard certs for *.
jpambrun.com, respond to DNS queries for www-xx-xx-xx-xx.jpambrun.com and
serves the certs at jpambrun.com/keys for everyone to see..
I think this projects' implementation is slightly neater, I was just hoping
it could take inspiration from localtls.
I didn't know about the whole story. The hn thread was a good read, but
It's pretty old. Now so many web APIs are locked behind HTTPS and
self-signed certs break many things.. maybe such an approach would not be
as frowned upon today..
…On Fri, Jan 22, 2021, 9:21 AM Brian Cunnie ***@***.***> wrote:
I think it would be nice to have some simpler instructions as an
introduction/overview. Basically it comes down to:
Thanks @normanr <https://github.com/normanr> , that's pure gold. I'll
take a second pass at the instructions. I'll follow the template that you
described, and leave the instructions I initially wrote as a very specific
example of procuring wildcard certificates.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFPMMGLTZKHDC37ZK4SK3DS3GCXXANCNFSM4EVT5WKQ>
.
|
I suspect that the preferred workaround for getting access to APIs that require HTTPS these days are services like https://ngrok.com/, https://localhost.run/, https://smee.io/, Cloudflare Argo Tunnel. I'm excited to see what https://tailscale.com/ are doing with their Magic DNS offering, and I suspect that they could integrate support for |
I reached out to my contact, but I haven't heard back from him. In the meantime, I read the Let's Encrypt Subscriber Agreement, and it looks like that by allowing anyone to procure a cert for
Admittedly the existing ability to procure certificates for sslip.io subdomains (e.g. |
By "procure a certificate", do you mean "gain access to the (existing) private key corresponding to the public key in the Note that certificates only contain public data and they can be shared freely. It's the private key that must be properly protected.
I'm not sure why. I don't think it's that different from a dynamic dns service where you can register a sub-domain of your choice, or ones where you can delegate an entire sub-domain to a nameserver of your choosing. As far as I understand the reason that private keys need to be protected is so that no one else is able to impersonate the domain name that the corresponding certificate is securing (whether or not the original domain considers it "allowed" or not). By using sub-domains, everyone using the service has their own unique certificate and private key. One user can't impersonate another user's domain name because the domain names in their certificates are different. |
|
Isn't the process described in wildcard.md going to lead to "everyone" hitting rate limits on Let's Encrypt will give you 50 "Certificates per Registered Domain" per week, and unless you're able to get |
We want people to report rate-limiting so we know to request an increase. [#6]
|
For another take on the "necessary steps to maintain control of, secure, properly protect and keep secret and confidential the Private Key corresponding to the Public Key in Your Certificate", I've been building |
|
Hi @ncruces ,
Thanks! I've updated the website to include a blurb, "If your request for an "sslip.io" certificate is rate-limited, please open a GitHub issue and we'll request a rate-limit increase." I use Let's Encrypt for the "sslip.io" web server, and I haven't noticed a rate-limit yet, but it's good to be prepared for that eventuality. I was expecting the process to procure a wildcard cert to be cumbersome enough so that maybe 2 people would go through it, and probably renew on a monthly schedule, which would be about ½ request per week. But that's pure speculation. I'm fascinated about the public suffix list, but it looks like they've accounted for domains such as sslip.io/xip.io/nip.io:
|
|
Hi @cunnie,
It's less than that. Renewals with the exact same domain names don't count, so it's only "new" certificates that are rate limited to 50/week. But if someone decides to build something (like Plex) on top of this, where it gets automated on behalf of users, 50 new users per week is not that much. Unfortunately, anything less than one certificate per user is subject to MitM, so to build something like Plex, you either get your users to buy a domain, or you need to partner with a CA. I think |
|
I ended up hacking what I wanted here: I am not trying to plug this, this is poor quality, soon to be unmaintained.. but maybe it can help someone. |
|
@jpambrun : nice, tight code! I wish I had been able to make my code as compact. |
|
btw, one additional suggestion if using a public dns service, would be to always add a prefix to the external ip. Otherwise another account on the same service might have already added the same domain and you won't be able to. If you just use a unique prefix (like your account name) then that problem is solved. I also found that it's relatively easy to find out who's using TLS certs with sslip.io because the certificate transparency logs are publicly searchable: https://crt.sh/?Identity=sslip.io&exclude=expired |
|
Actually the Censys logs show roughly 1000 valid certs from Let's Encrypt, which is suspicious and makes me think that rate limiting is being applied. |
Thanks, that's a great resource—I had no idea that something like Censys existed. I'm going to try to filter how many of those 1k certs are renewals & how many are new. To have 1,142 active certs with 90-day window would mean that 88 certs are being registered each week—well above the 50/week limit, so obviously many of them are renewals, which don't count against the rate-limit, but it'd be nice to know how close to the ceiling we are, if we've not already bumped into it. |
|
I was using nip.io at some point and often faced LE rate limit issues.
That's what first prompted me to run a dynamic NS myself.
See exentriquesolutions/nip.io#4
…On Sun, Feb 14, 2021, 11:34 AM Brian Cunnie ***@***.***> wrote:
@normanr <https://github.com/normanr>
Actually the Censys logs show roughly 1000 valid certs from Let's Encrypt,
which is suspicious and makes me think that rate limiting is being applied.
Thanks, that's a great resource—I had no idea that something like Censys
existed. I'm going to try to filter how many of those 1k certs are renewals
& how many are new. To have 1,142 active certs with 90-day window would
mean that 88 certs are being registered each week—well above the 50/week
limit, so obviously many of them are renewals, which don't count against
the rate-limit, but it'd be nice to know how close to the ceiling we are,
if we've not already bumped into it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFPMMCY3YFX53XVOHYJZJDS673S5ANCNFSM4EVT5WKQ>
.
|
|
nip.io and xip.io seem to have ~11k valid certs. I'm not sure if that's just they're more well known, or they have asked Let's Encrypt for an increased rate limit. It looks like it's easy enough to ask, but it does take a couple of weeks to rollout (so it's not a temporarily working around rate limits solution). |
I know nip.io has asked for a rate-limit increase (thanks @jpambrun for the link), and based on your research they have ~10x the number of certs that sslip.io has, so I've asked to bump the rate-limit by 20x (i.e. from 50/week to 1000/week). Here is how I filled out their form:
Updated Answers 2022-04-07:
|
|
@ncruces By the way, I'd like to compliment you on keyless ; I wasn't even aware that was possible (providing TLS without exposing the key). The obvious implication would be to use something like keyless for *.sslip.io, but I sadly discarded that notion: although keyless adheres to the letter of the law, it doesn't adhere to the spirit, and I don't want to get into trouble again. |
|
keyless looks great. I assume by the fact that it can only sign that it only supports DH handshakes, and not RSA handshakes I suspect that "take, all appropriate, reasonable, and necessary steps to maintain control of, secure, properly protect and keep secret and confidential the Private Key" means that the signing endpoint is subject to the same restrictions (for the "properly protect" part). So I don't think it could be used for sslip.io without a significant amount of work to add user authentication, etc, at which point it's no longer sslip.io. My hope is that some of the existing "users" (spammers?) who are requesting multiple certificates with many names will switch over to wildcard certificates, and the number of new certificates will drop, because they're covered by the wildcard certificates. It looks like there are plenty doing two level names, so not all usages will be covered, but many would be. |
|
I used ECDSA only to simplify matters ("pick one"), but it should work with RSA. Cloudflare offers a similar service for enterprise customers who, for compliance issues, need to e.g. provide EV certificate, and can't share their private keys (which may be hardware backed). They support RSA. In terms of authentication, my use case is an app (like Plex) with an embed webserver which users access through a browser, mostly on localhost, but maybe on their LAN, not sure if I'll enable it over the internet. My current plan is to embed a client certificate and private key pair, and do mTLS with the key server. This is a relatively common “kinda-secure” to do on mobile (where, esp. iOS it's “hard” to get the key), but in the end, it's obfuscation, DRM, etc. It's a (small?) step up from distributing the actual private key as the folks from |
|
It would be better to generate a separate private key on first run, instead of embedding a single cert/key in the app. You can sign it with an in-house CA (because it'll only ever be used to secure the connection to the keyless server by a client that you control). Then you have a way to differentiate between connections to the keyless server, and can revoke individual certificates should it ever be required. |
|
@normanr , @ncruces , @jpambrun : FYI, Let's Encrypt bumped our rate limit to 3,000 this morning:
Thanks for all your help! |
|
No one's commented in over three months; there's a mechanism to generate wildcard certificates; closing. |
A few days ago Let'sEncrypt turn on the ACMEv2 protocol.
Do you have any plans to implement some API for issuing free wildcard certificate by request from same IP?
The text was updated successfully, but these errors were encountered: