NixOS 24.05 creates a new letsencrypt account and breaks CAA

[sbourdeauducq]

After updating to 24.05, it appears that the ACME client (with the default infrastructure: security.acme/services.nginx.xxx.enableACME etc.) registers a new account for obtaining letsencrypt certificates. If you had CAA with account binding enabled on your domains, the certificate renewals will now fail.

If you are affected by the problem, here is how you can figure out the new account URI in order to update your CAA entries:

1. ls -l /var/lib/acme/.lego/accounts/ and take the most recent directory
2. cat /var/lib/acme/.lego/accounts/xxxxx/acme-v02.api.letsencrypt.org/email@example.com/account.json

This should probably be added to the release notes.

[sbourdeauducq]

And you will, of course, very rapidly hit the ultra-frustrating letsencrypt rate limits again. They really ought to revise those policies as it makes relatively minor bugs like this a much bigger pain in the ass than they should be..

[flokli]

cc @NixOS/acme

[arianvp]

Did you update from 23.11?

[sbourdeauducq]

Yes.

[stephank]

Suspect 4494fca, because the two different directories I have now match the SHA256 of "https://acme-v02.api.letsencrypt.org/directory ec256 <email>" and " ec256 <email>". (Note the space at the start, which is the toString null of the old default for the server option.)

[arianvp]

I honestly have no idea why this happened. The module hasn't had any significant changes for the past few years and nothing stands out in the recent history to me.

[arianvp]

Wow @stephank good catch. Yes sounds like that is the issue

[flokli]

Do what's the plan forward? IMHO we should fix this whitespace and backport it to stable, assuming there's more people who didn't yet upgrade to the latest release vs that did.

Also, there might be a chance people who already did the upgrade still have their old accounts available, no?

[arianvp]

Apparently this bug was brought up during review: #270221 (comment) :(

[sbourdeauducq]

Is this a solid assumption? With a maintainance window of just one month, there is quite some pressure to update NixOS.

Also, there might be a chance people who already did the upgrade still have their old accounts available, no?

The issue is having to update the CAA records in the DNS, being able to still use the old account does not really matter.

[stephank]

Maybe we can keep the current method, but if the correct accounts directory doesn't exist, check if one exists under the old hash and move it?

One thing I'm unclear about: only the account hash should've changed, I believe? So why is it renewing all domains as well? Is there a link between domains and account ID somewhere, I'm not seeing? (EDIT: looks like server is part of the certDir hash as well, my bad.)

[flokli]

What's really bad is neither @NixOS/acme got highlighted on the issue, nor did the release note update happen. This currently is a tire fire for everyone switching to it, either having them run into rate limits or having to update CAA records (after they notice, at all).

[arianvp]

We could create a patch that migrates the directory like @stephank proposes. Suggestion in the NixOS ACME/LetsEncrypt channel was similar. Something along the lines of:

if [ -d $oldHash ]; then 
  if [ ! -d $newHash ]; then
   mv $oldHash $newHash
  else
   echo "You are dedge please fix"
   exit 1
  fi
fi

In any case we still should also add a note to the release notes as this migration doesn't help people who have already upgraded.

[SuperSandro2000]

This currently is a tire fire for everyone switching to it

I've updated probably 25 VMs all with certs and didn't run into an problems and already forgot that this was a problem 4 months ago. I assume this is only a problem if you have the account ID in the caa record or if you have way to many http challenge certs with different accounts.

IMHO we should fix this whitespace and backport it to stable, assuming there's more people who didn't yet upgrade to the latest release vs that did.

We cannot just do that, as that would cause the opposite direction for anyone on unstable for the last 4 months, that already migrated or that recently installed 24.05.
We probably need some state convergence.

[arianvp]

Since #106857 landed the rate-limit issue is probably less of an issue in practise. I think we really only should care about breaking people's CAA setup.

In that case I think we can get away with just an entry in the release notes and state convergence can be seen as a nice to have but not a must have.

[osnyx]

the rate-limit issue is probably less of an issue in practise.

At the Flying Circus, there's a high likelihood we're going to be affected by this nonetheless, despite the head-of-line registration blocker setup you mentioned, @arianvp. We have several hundred machines that'll request a new account when rolling out 24.05.

You can create a maximum of 10 Accounts per IP Address per 3 hours. You can create a maximum of 500 Accounts per IP Range within an IPv6 /48 per 3 hours.

https://letsencrypt.org/docs/rate-limits/

When having to fix this downstream, I'd probably adopt a state convergence activation script approach.
Alternatively, @stephank had proposed to re-allow null as the account URL again, which would allow getting back the previous account hash string. If we adopt this here in NixOS upstream, that's a decent solution as well.

[arianvp]

I'll work on a patch to re-allow null tonigh. Will post here for review in a bit!