renew --allow-subset-of-names causes removed failures to not be renewed again during normal run

[mahriman]

My operating system is (include version):

FreeBSD 12.1-RELEASE-p3 GENERIC amd64

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

OS package manager (pkg)
Name : py37-certbot
Version : 1.0.0,1

I ran this command and it produced this output:

certbot renew --allow-subset-of-names --force-renewal
It created a cert for my domain as CN and my subdomains as SAN as expected, missing two SAN subdomains that were the reason for me using --allow-subset-of-names in the first place.
After fixing the problems with the two subdomains, I re-ran with
certbot renew --force-renewal upon which certbot only renewed the existing cert with CN and Subject Alternative Names from the current certificate, not with the SANs for the two subdomains that had now been fixed.

Certbot's behavior differed from what I expected because:

I expected certbot to respect the configuration file for the domain and not choose what CN and SANs to renew based on the latest certificate. Running certbot renew --force-renewal seems to renew based on latest certificate's CN and SANs rather than use the configuration file to see what domains should be renewed for that certificate.

Additional context

• I confirmed all the domains were still in the domain.org.conf file
• I tried fixing this by re-linking /usr/local/etc/letsencrypt/live/domain.org/<files> to an older certificate (cert1, fullchain1....) in archive/domain.org and re-running, which resulted in
  Found a new cert /archive/ that was not linked to in /live/; fixing... I.e., certbot found a newer certificate (2, 3 and namely 4) and linked live/domain.org/<files> to the errornous 4-version and the problem persisted.
• Upon removing version 2, 3 and 4 from archive/domain.org/<files> and re-linking live/domain.org/<files> to their archive/domain.org/<files> counterparts again and once again re-running certbot renew --force-renewal, my certificate was now holding my previously missing SANs that had once failed renewal.
• I noticed after the fix was done that allow_subset_of_names = True was added to all my domains configuration files, it's quite possible changing this to false/removing the configuration entry would've fixed my issue much easier. If so, I'd expect much clearer output from certbot that it's omitting renewal of domains due to previous allow_subset_of_names

Steps to reproduce

1. Create a domain configuration with several domains and obtain certificate
2. Make one or more domains faulty, e.g. by setting wrong permissions on webroot folder, making the root of the domain non-accessible due to 401 in webserver config etc.
3. Run certbot renew --allow-subset-of-names
4. Fix the issue with the faulty domains
5. Run certbot renew
6. (latest cert is not including the faulty domains as SANs that has since been fixed)

[alexzorin]

I expected certbot to respect the configuration file for the domain and not choose what CN and SANs to renew based on the latest certificate

AFAIK the renewal config file has never been used to define the SAN list for the certificate. I don't think there's any such option in the renewal config file.

The lineage's latest certificate is used to determine the SAN list for the next renewal.

Maybe this is a documentation fix to warn that --allow-subset-of-names is destructive.

I confirmed all the domains were still in the domain.org.conf file

Could you share what the file looks like? Are you referring to the webroot map for the webroot authenticator?

[mahriman]

AFAIK the renewal config file has never been used to define the SAN list for the certificate. I don't think there's any such option in the renewal config file.

The lineage's latest certificate is used to determine the SAN list for the next renewal.

Maybe this is a documentation fix to warn that --allow-subset-of-names is destructive.

Yes, perhaps that is the solution. I had no idea the latest certificate was used to determine the SAN list but imagined the renewal configuration would be taken into account. That explains my issues, then.

Could you share what the file looks like? Are you referring to the webroot map for the webroot authenticator?

Yes.

version = 1.0.0
archive_dir = /usr/local/etc/letsencrypt/archive/domain.org
cert = /usr/local/etc/letsencrypt/live/domain.org/cert.pem
privkey = /usr/local/etc/letsencrypt/live/domain.org/privkey.pem
chain = /usr/local/etc/letsencrypt/live/domain.org/chain.pem
fullchain = /usr/local/etc/letsencrypt/live/domain.org/fullchain.pem

[renewalparams]
account = (removed)
authenticator = webroot
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
bowser.domain.org = /usr/local/www/apache24/data/domains/domain.org/bowser
domain.org = /usr/local/www/apache24/data/domains/domain.org
mail.domain.org = /usr/local/www/apache24/data/domains/mail.domain.org
pwnad.domain.org = /usr/local/www/apache24/data/domains/pwnad.domain.org/webroot
wiki.domain.org = /usr/local/www/apache24/data/domains/wiki.domain.org
www.domain.org = /usr/local/www/apache24/data/domains/domain.org

For reference, bowser.* failed due to a misconfiguration in Apache (duplicate entries for the vhost, the latter pointing to another location than the webroot) and wiki.* failed due to 401.

[schoen]

As @alexzorin notes, the Certbot option --allow-subset-of-names has never worked this way. We did once consider making it work this way, but we ended up not going this route. I think we originally called the idea something like "aspirational name coverage". Perhaps it would still be a useful option for some users; we could also call it --irredentist. 🙂

That would be a totally different mechanism than exists in Certbot right now, as there is no official way that Certbot can track desired or aspirational names.

I agree with the suggestion for the present option to be more explicitly documented as "destructive", both in --help and in the documentation page. I created issue #7929 to track this.

I'm just still curious about

I confirmed all the domains were still in the domain.org.conf file

What did that look like and how did that get there?

[mahriman]

What did that look like and how did that get there?

See my post above. It was generated from certbot certonly
The only difference in the file from before and after running with --allow-subset-of-names was the allow_subset_of_names = True directive

[github-actions]

We've made a lot of changes to Certbot since this issue was opened. If you still have this issue with an up-to-date version of Certbot, can you please add a comment letting us know? This helps us to better see what issues are still affecting our users. If there is no activity in the next 30 days, this issue will be automatically closed.

[github-actions]

This issue has been closed due to lack of activity, but if you think it should be reopened, please open a new issue with a link to this one and we'll take a look.