Blacklist domain while permitting issuance to subdomain(s)

[jcjones]

One Let's Encrypt subscriber whose domain is blacklisted (we'll say example.com) wanted to keep the blacklist entry for the registered domain, but permit one or more individual subdomains for issuance (like blog.example.com and mumble.example.com).

I do not believe that the current blacklist / whitelist interaction permits this style of permissions, though it is desirable for some.

[jsha]

To flesh this out a little: The idea is that we would have one blacklist-including-subdomains and a separate FQDN-blacklist. Currently our whole list of high risk domains is on the former. If a domain owner of a high risk domains wants to issue with Let's Encrypt but also wants to keep certain FQDNs blacklisted, we could remove their domain from the former list, and add a set of specific FQDNs to the latter.

[yaronf]

The FQDN-blacklist should have both the high-level blacklisted domain (bigcorp.com) as well as the whitelisted subdomains (sub1.bigcorp.com, sub2.bigcorp.com etc.). So it is a bit more than a simple list. In other words, you don't want to try to infer the blacklisted domain from the allowed subdomains.

[seveas]

I would actually use it the other way around at first: test.example.com would be whitelisted while all other subdomains would still be blacklisted.

[stevewiesel]

Need this capability for a project I am working on at work. In looking at policy-authority-data.go, whitelists are only checked if requireWhitelisted, and then are overridden regardless by an existing blacklist entry. It would seem to me that one possible solution to this would be that if a specific whitelist entry existed AND there was NO specific blacklist entry then ALLOW. But if a blacklist entry existed regardless of the existence of a whitelist entry then REJECT. Seems like a trivial fix. What am I missing?

[jsha]

The whitelist code you see in the repo is a leftover from the public beta, when we were issuing only to whitelisted domains. It's not really suitable for the plan here, and we intend to rip it out.

A little more detail on why the solution I describe is preferable to implementing a whitelist of specific domains that overrides the blacklist: If we maintain a whitelist of overrides, we will have to process a request to add to the whitelist each time a domain owner wants to use Let's Encrypt for a new subdomain, which is likely to be frequent for very large domains. The idea is that domain owners know better than we do which FQDNs within their domain are truly sensitive, and should be able to provide a list of those for more fine-grained blacklisting.

[stevewiesel]

Interesting. We would want the flip-side of that. A very select few would NOT be sensitive in our eyes and those and only those should receive the exceptional processing. Your solution is heavily slanted towards the reverse. I wonder how unique we are in that respect? We are one of those BIG DOMAIN kind of guys.

[stevewiesel]

In corresponding with Josh it seems like this is not very high on the release priority list. We need it, and are willing to write the code and issue the PULL request. I am just trying to figure out what makes sense for the larger community vs. our specific needs.

[stevewiesel]

My solution seems like it has the virtue of being backward compatible, a small number of lines of code and one SQL statement, and doesn't force BIG DOMAIN guys that have only a small hand full of non-sensitive domains to pay an onerous penalty to get that setup. Already blacklisted and want to stay that way..great. Already blacklisted with a small handful of sensitive subdomains...add a few whitelist entries and you are done. It really boils down to whether the belief is that the BIG DOMAIN folks will have a handful of subdomains they want to whitelist or blacklist. I dont think we are unusual in that respect. I think many of the other BIG DOMAIN guys will want the vast majority blacklisted and the minority whitelisted.

[jsha]

I agree with you that from the perspective of a large domain operator, it's preferable to whitelist only a handful of subdomains at a time. That is definitely the conservative approach. However, the set of subdomains for which you want to issue will surely grow over time, as new products or subdomains are added. For instance, when I was at Twitter, we'd frequently spin up one-off subdomains hosted by third parties (like investor.twitter.com). If Let's Encrypt was around at the time, and those third parties had integrated with it, LE would have provided a great way to easily turn on HTTPS. However, LE's high risk domain list would have gotten in the way.

If LE were to use a whitelist of subdomains on top of the blacklist, each such new subdomain would mean that Let's Encrypt will have to field a change request from the domain owner. We're not set up to field a high volume of change requests. Our ability to provide large numbers of secure certificates for free is predicated on being able to automate our service as much as possible.

I agree that my proposed approach makes it harder for domains on the high risk list to start using Let's Encrypt, and that's a bummer. But, after the initial work (on the domain owner's part) of generating a most specific list of high risk FQDNs, it makes future use of Let's Encrypt much easier for new subdomains. And it also means the work on LE's site is one-time, rather than ongoing.

[stevewiesel]

Totally get it. However, I think you will find that for the big guns like us that makes it very difficult for us to use the service. There are so many subdomains that a blanket blacklist with an exceptional whitelist truly works best for us. Somewhere down the line there will be change requests to either add new blacklisted subdomains or new whitelisted ones whichever way this goes. At least for us (although we may be atypical) there will be far fewer requests to whitelist than blacklist. Because we are a high value target we want as much as possible to be on your blacklist. Because of the magnitude of scale, any process that requires us to create blacklist entries for everything we want blacklisted now and in the future would be a significant effort and may in fact be prohibitive.

Sent from my iPhone

On Feb 19, 2016, at 18:19, Jacob Hoffman-Andrews notifications@github.com wrote:

I agree with you that from the perspective of a large domain operator, it's preferable to whitelist only a handful of subdomains at a time. That is definitely the conservative approach. However, the set of subdomains for which you want to issue will surely grow over time, as new products or subdomains are added. For instance, when I was at Twitter, we'd frequently spin up one-off subdomains hosted by third parties (like investor.twitter.com). If Let's Encrypt was around at the time, and those third parties had integrated with it, LE would have provided a great way to easily turn on HTTPS. However, LE's high risk domain list would have gotten in the way.

If LE were to use a whitelist of subdomains on top of the blacklist, each such new subdomain would mean that Let's Encrypt will have to field a change request from the domain owner. We're not set up to field a high volume of change requests. Our ability to provide large numbers of secure certificates for free is predicated on being able to automate our service as much as possible.

I agree that my proposed approach makes it harder for domains on the high risk list to start using Let's Encrypt, and that's a bummer. But, after the initial work (on the domain owner's part) of generating a most specific list of high risk FQDNs, it makes future use of Let's Encrypt much easier for new subdomains. And it also means the work on LE's site is one-time, rather than ongoing.

—
Reply to this email directly or view it on GitHub.

[yaronf]

Adding to what Steve said, the proposed fix (whitelist by default) would probably make it too risky for big organizations to join LE. It will be a very hard sell internally.

[seveas]

As the person who caused this ticket to be filed, I have to say that the proposed solution is entirely contrary to what we want, and we'd much favour an explicit whitelist. If a whitelist-by-default approach is taken, we would definitely still not use Let's Encrypt for subdomains of our main domain.

What @stevewiesel describes also definitely applies to our situation. It would for us also be a lot more work to maintain the blacklist than to maintain the whitelist. Subdomains we'd whitelist for let's encrypt would be things we delegate to third parties. At the moment that would be at most a dozen or two subdomains and that set changes slowly. And hundreds of subdomains should definitely not use it, with subdomains being added on a fairly regular basis (I added a few dozen the other day for instance).

[rolandshoemaker]

It should be noted that our internal blacklist is not the only place for you to implement measures that will prevent us from issuing for you domains. This is almost exactly what the CAA DNS record specification was intended for and since Boulder actively supports CAA (and Let's Encrypt enforces it's use in its CPS) it is a viable alternative.

If you wish to blanket subdomains under an apex from issuance all you need to due is add a CAA record with the tag issue and an empty (or non-lets-encrypt.org) value. CAA checks are designed to descend to the apex until they find a record, so if I add a CAA record to example.com as detailed I will also be preventing any issuances to a.example.com, a.b.example.com, etc.

If you then wanted to whitelist a single domain you could add a CAA record with a issue tag that matches the issuers domain (i.e. for Let's Encrypt lets-encrypt.org). Boulder will encounter this record before the one on the apex domain which will allow us to issue for that specific subdomain.

Implementing blacklist by default would become a huge administrative hassle for us, mainly because we would have to have some process in place to decide if a domain really needs to be on the blacklist because it truly is 'High-Value' and not just a random site that the submitter wants on the list and how do we verify that the submitter is the owner. This goes beyond our scope of implementing an automated CA.

[yaronf]

So to clarify, what you suggest that we (as owners of example.com) do is:

• Add an "empty" CAA record to example.com.
• Request Let's Encrypt to remove example.com from the blacklist.
• Add specific CAA records to any domains we would like to whitelist.

Is this correct?

[hlandau]

That will work fine.
The empty CAA record looks like this:
example.com. IN CAA issue ";"

[riking]

branch:exact-blacklist

blocked on tests infrastructure

[jsha]

This is unblocked now that #1773 landed.