[debops.pki] Nginx OCSP issue with DST Root CA X3 expiration, pki-realm script and ca-certificates package update

[alo-is]

Hi,

Following the depreciation of LE DST Root CA X3 and its removal from ca-certificates package, it looks like the pki-realm script fails to grep the CA in /etc/ca-certificates.conf folder and thus, to create the required files for the Nginx role (/etc/pki/realms/<domain>/trusted.crt, which is used for OCSP validation). This causes Nginx to fail after ca-certificates has been upgraded to it's last version.

The following line seems to cause the error :

debops/ansible/roles/pki/files/usr/local/lib/pki/pki-realm

Line 1171 in c24d514

if [ -r "${config['acme_root_ca_path']}/${config['acme_root_ca_file']}" ] && grep -q "${config['acme_root_ca_file']}" /etc/ca-certificates.conf ; then

Which has an hardcoded value to :

debops/ansible/roles/pki/files/usr/local/lib/pki/pki-realm

Line 192 in c24d514

config["acme_root_ca_file"]="mozilla/DST_Root_CA_X3.crt"

It looks like that it should now be replaced by mozilla/ISRG_Root_X1.crt, but I don't know how to handle the full chain right now, as Nginx still complains after the update.

Related discussions :

• https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1944481
• https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

[jptreen]

With that single edit that you suggested to debops/ansible/roles/pki/files/usr/local/lib/pki/pki-realm, my nginx will happily restart following a run of service/nginx and service/pki.

Out of interest, how is your nginx failing after the update?

[alinalexandru]

I confirm that the fix works

[imrejonk]

I believe this has been fixed with #1881, feel free to re-open if you are still having issues.

[imrejonk]

After updating my host's PKI realm scripts using the patched debops.pki role, I had to update my PKI realms like this:

1. Update all realm configurations: ansible -b -m shell -a 'grep -rl DST_Root_CA_X3 /etc/pki/realms | grep "config\/realm\.conf" | xargs -I{} sed -i "s/DST_Root_CA_X3/ISRG_Root_X1/g" {}' debops_all_hosts
2. Update all acme/root.pem symlinks: ansible -b -m shell -a 'find /etc/pki/realms -wholename "*/acme/root.pem" | xargs -I{} ln -fs /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt {}' debops_all_hosts
3. Manually update public/full.pem and public/intermediate_root.pem files that still contain the DST root. You can find those with ansible -b -m shell -a 'grep -rl "Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ" /etc/pki/realms' debops_all_hosts. Remember to replace both the intermediate and root certificates in those files. I haven't been able to come up with a more clever way to do this; if you know any, please share.

@pdobrigkeit

[imrejonk]

You can also verify that you are not using the old cross-signed R3 CA anymore with ansible -b -m shell -a 'grep -rl "UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==" /etc/pki/realms' debops_all_hosts. This didn't return any hits on my infrastructure, so this one should probably be fine.