Let's Encrypt SSL certificates are being linked to an expired root certificate

[MLeFrench]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

New generated/re-issued SSL certificates with Let's Encrypt via the ACME Client are linked to an expired R3 Root Certificate.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior:

1. Go to Services
2. Click on ACME Client
3. Go to Certificates
4. Issue new Certificate that's with an ACME Account linked to Let's Encrypt
5. Validate the Certificate when created with a free service eg: https://www.sslshopper.com

Expected behavior

A valid SSL Certificate is generated with no issues in its chain.

Describe alternatives you considered

Using another SSL certificate provider

Screenshots

Relevant log files

n/a

Additional context

The R3 root certificate expired yesterday:

https://scotthelme.co.uk/lets-encrypt-old-root-expiration/
https://techcrunch.com/2021/09/21/lets-encrypt-root-expiry/

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 21.7.3_1-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021

[fichtner]

Wrong repo and duplicate.

[MLeFrench]

Thanks @fichtner, 🤦‍♂️ found it in plugins #2550

[fichtner]

No worries, hotfix going out later today :)

[ssbarnea]

In fact I am quite worried as opnsense updates failed to due to invalid certificate.

[AdSchellevis]

@ssbarnea likely these issues are due to a certificate locally installed, just try to update from a fresh install (works like a charm from here). Most of this is caused due to how let's encrypt handles their expired certificate (I assume you can't access their website either from the same install as it distributes the same chain)

[ssbarnea]

My OpenSense box was configured to auto-update and I am running 21.7.3_1 which right now is unable to get any updates, from any mirror! I doubt all of them are using the new certificate.

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 21.7.3_1 (amd64/OpenSSL) at Fri Oct  1 09:45:52 BST 2021
Fetching changelog information, please wait... Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
4435996811264:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/21.7/sets/changelog.txz.sig: Authentication error
Updating OPNsense repository catalogue...
...

The funny bit is that my router has an ACME certificate configured on it for it own interface and I was able to use ACME client to regenerate certificates. That part worked but the router itself is not able to get any updates and I don't really understand why this happens.

Wasn't the "default" mirror option designed to provide resiliency for the case when one of few mirrors go down? I tried picking 4-5 and I still get the same kind of errors,... and interestingly they all involve https://pkg.opnsense.org. I guess that its certificate needs the new intermediate which for some reason was not loaded by the router (no idea why considering auto-update was enabled).

I guess I need to manually add the new intermediate ACME certificate to fix it but I am unable to find it.

[ssbarnea]

I found the certificates on https://letsencrypt.org/certificates/ but is not clear which files i need to download and upload to the router. The cross-signed or the self-signed ones? I guess pem format.

Update: I uploaded both cross-signed root ones but updates still do not work. Considering that this is breaking core updates, I am not sure if we can really claim that is a plugin issue. Or maybe we need a new ticket to track that issue.

[AdSchellevis]

It's likely that you have one certificate installed which you shouldn't trust anymore, which is the same as the webbrowsers have implemented. Let's encrypt sends two paths, if the expired one is trusted on your end, the path is expired. Which is why you likely can access https://pkg.opnsense.org from a browser without issues (and also from a clean install).

[AdSchellevis]

some context for RHEL which explains the issue quite nicely (in my humble opinion): https://blog.devgenius.io/rhel-centos-7-fix-for-lets-encrypt-change-8af2de587fe4 . When using LibreSSL, the trust first doesn't seem to be available yet (https://ftp.openbsd.org/pub/OpenBSD/patches/6.8/common/032_cert.patch.sig thanks @fichtner), in which case you really need to make sure "DST Root CA X3" isn't trusted on your end.

[fraenki]

in which case you really need to make sure "DST Root CA X3" isn't trusted on your end

AFAICT, the expired X3 CA is still shipped with ca_root_nss-3.69_1 (and installed in /usr/local/etc/ssl/cert.pem). Could this be the root cause?

[ssbarnea]

• performed manual upgrade using env SSL_NO_VERIFY_PEER=1 pkg update && pkg upgrade just to get the latest 21.7.3_3 with patches.
• removed the expired DST Root CA X3 root cert
• changed mirror to http://mirror.ams1.nl.leaseweb.net/opnsense/FreeBSD:12:amd64/21.7 and that allowed the update process to finish with error (but some warning about cert still there). With default cert it does still fail!
• List of current CA:
• I regenerated router certificate using the upgraded acme client and now the browser recognizes its certificate as correct.

Still, the main issue here is that router itself does not have a correct set of CA certificates, so update process is still partially broken. I do suspect that should also be the case for a long list of opnsense users. I see some kind of vicious cycle: new CA certs needed in order to get updates, you cannot update due to outdated CA certs.

[AdSchellevis]

@fraenki that depends if people suffering from this issue are all using LibreSSL, but if the expired cert is still shipped, eventually it should be removed.

[fraenki]

@AdSchellevis I'm on OpenSSL and am seeing the same issue here. It's unrelated to os-acme-client. In the link you've posted they said that RedHat removed the expiring CA from the root CA list as a fix. An alternative could be to blacklist the expired CA, because I figure customizing ca_root_nss is not an option.

[ssbarnea]

I failed to mention that my "flavour" is still on default. Should I change it to something else?

[AdSchellevis]

@fraenki the issue doesn't seem to exist on a clean install, none of our machines over here are suffering from this.... @ssbarnea no, OpenSSL is the default, which is what we use as well.

[AdSchellevis]

@fraenki what happens if you replace /usr/local/etc/ssl/cert.pem and /etc/ssl/cert.pem with the contents of /usr/local/share/certs/ca-root-nss.crt (backup first or trigger system_trust_configure() to revert)?

[fraenki]

@AdSchellevis Same result. And I think this is expected, because /usr/local/share/certs/ca-root-nss.crt contains the expired X3 cert.

[AdSchellevis]

@fraenki well, the same cert is also available on my end, but my machine works like a charm (tested both pkg and curl)

[fraenki]

@AdSchellevis Then there's something wrong with how the trust chain is build/updated, right? I don't know how this could be os-acme-client's or a user's fault.

[fraenki]

what happens if you replace /usr/local/etc/ssl/cert.pem and /etc/ssl/cert.pem with the contents of /usr/local/share/certs/ca-root-nss.crt

OK, I have to reiterate on that. During my previous test the file got overwritten by the system before I've started my test.

So after running a new test I can confirm now that overwriting both files fixes the warning.

And I can confirm that having the current and not expired X3 cert in System: Trust: Authorities leads to the problem in the first place, but why?

EDIT: Just tested this on a fresh installation. Importing the new X3 cert immediately causes the validation issue. Removing it fixes it. Does this make sense?

Full X3 chain for testing:

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

[fraenki]

Just tested this on a fresh installation. Importing the new X3 cert immediately causes the validation issue. Removing it fixes it.

OK, the sort order in /usr/local/etc/ssl/cert.pem seems to be the key. The chain provided by Let's Encrypt contains the CA chain in the following order:

1. Subject: C = US, O = Let's Encrypt, CN = R3
2. Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

As a result, it gets added to System: Trust: Authorities in this order and finally gets added to /usr/local/etc/ssl/cert.pem in the same order. Reversing this order (by manually altering cert.pem) solves this issue.

This issue seems to only affect base tools such as pkg and fetch, but curl and openssl seems to verify the chain just fine.

Not sure how to proceed from here...

[AdSchellevis]

@fraenki base uses its own openssl version, which might have issues with the dual path. I'm not sure when X509_V_FLAG_TRUSTED_FIRST was added to openssl, but this might explain the issue at hand. What I don't understand is why pkg/fetch doesn't work on your end when only the original ca-root-nss.crt is used.

The easiest fix from our end is probably to ship a ca-root-nss.crt in the future without the expired cert, some sort of "blacklist" mechanism might also be something to consider in the long-run, as these type of issues aren't exclusive to this single cert.

Am I right to assume that you only have seen this issue on machine which do have certificates imported locally (in System: Trust)? Or is there an edge case I haven't seen yet?

[fraenki]

What I don't understand is why pkg/fetch doesn't work on your end when only the original ca-root-nss.crt is used

In another test this worked as expected. Sorry for the confusion that my first test caused.

Am I right to assume that you only have seen this issue on machine which do have certificates imported locally (in System: Trust)?

Only when importing the new X3 CA chain.

@AdSchellevis, do you have any idea why the sort order matters? This might be key for fixing this issue. I've also seen that that other firewall project uses a different approach for handling root certificates...

[AdSchellevis]

@fraenki I can only assume that openssl in base doesn't come with X509_V_FLAG_TRUSTED_FIRST, in which case it likely always chooses the first (or last) path it finds.

I'm not sure if/how others sort certificates, but adding logic in these things usually is asking for issues later on. As an experiment you could try to feed your certificates before the standard set in system_trust_configure (https://github.com/opnsense/core/blob/d5d52ac975a62c083b82758dcc4ab2ab3a68c856/src/etc/inc/system.inc#L825).

I'm a bit hesitant for large changes and logic around certificate generation as this likely irons itself out in time since the ports packages seem to be doing the right thing and I don't expect the functional difference in ordering is intentional.

[fraenki]

As an experiment you could try to feed your certificates before the standard set in system_trust_configure

I've manually put the CA chain on top of /usr/local/etc/ssl/cert.pem but it did not change a thing. Only changing the sort order (so that the X1 cert is loaded before the R3 cert) seems to fix it.

[kulikov-a]

@AdSchellevis @fraenki hi! can i join the discussion? I also tested different options and so far I got this explanation for myself: the matter is generally in the presence of the cross-signed ISRG Root X1 certificate in the trust store.
and it seems to me that the trusted_first is default behavior now (https://www.openssl.org/docs/manmaster/man1/openssl-verification-options.html). And seeing the cross-signed ISRG X1 intermediate certificate in the trusts, openssl builds a chain to the expired root (ignoring short chain).
ideas that come to mind: use a short chain in the acme plugin (and this will kill compatibility with old androids) and use some blacklist of CA that can be used in the system_trust_configure to exclude this certs from the openssl cert.pem (as I understand, the result will have to resemble the action of the certctl https://www.freebsd.org/cgi/man.cgi?query=certctl&sektion=8). I like the second option: in this way the config will contain a long chain and services will be able to build their chains based on the config. and openssl client connections will not trust to cross-signed intermediate CA that chains to the expired root. (and imho the ability to block some CA trust can be useful in other cases as well)

[AdSchellevis]

@kulikov-a In principle I'm certainly not against an explicit blacklist, although with trusted_first (which doesn't seem to be supported in the base system, hence the difference between fetch and curl) the need is less urgent hough that probably covers some of the common scenarios where multiple valid paths are available.

Building a user friendly blacklist will have some challenges of it's own, we really would like to keep system_trust_configure() as simple as possible as really bad things happen if the trust chain isn't correct (as we have seen in the last couple of days) and the blacklisting action itself should be very easy and obvious to use (and not using legacy components in my humble opinion).

[kulikov-a]

@AdSchellevis on a vm without a acme plugin certificate, I added the ISRG Root X1 cross-signed certificate to the trusted and made a quick POC (kulikov-a/core@083c7e6)
with ISRG Root X1 cross-signed cer in Trusted CA there is Verify error (expired) for openssl, /usr/local/bin/openssl, fetch.
if i place https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem in blacklist folder and trigger system_trust_configure() then all of this works.
curl works all the time. but may be curl have some own logic on this (like browsers do)?

[fichtner]

Thanks for all the discussion! So if I see correctly, if both X1 root and R3 intermediate are installed in this order in System: Trust: Authorities this works? And if none are present this works too?

[kulikov-a]

@fichtner yes. order matters. openssl verify also gives different results when changing the order

root@OPNsense:~ # openssl verify -verbose -CAfile /etc/ssl/cert.pem /var/le_site.pem
/var/le_site.pem: OK

root@OPNsense:~ # openssl verify -verbose -CAfile /etc/ssl/cert.pem /var/le_site.pem
O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup: certificate has expired
error /var/le_site.pem: verification failed

[kulikov-a]

the question is whether changing the order makes it right or are we just tricking openssl with the wrong order
https://datatracker.ietf.org/doc/html/rfc5246#section-7.4.2

   certificate_list
      This is a sequence (chain) of certificates.  The sender's
      certificate MUST come first in the list.  Each following
      certificate MUST directly certify the one preceding it.  Because
      certificate validation requires that root keys be distributed
      independently, the self-signed certificate that specifies the root
      certificate authority MAY be omitted from the chain, under the
      assumption that the remote end must already possess it in order to
      validate it in any case.

[fichtner]

A little history. Users nagged us to move firmware updates to HTTPS and FreeBSD still doesn’t to it. Also, Adding CA certificates to trust store was a feature request by users a while back. I would hardly call this “a different approach” to trust stores. It looks like the usual price of being progressive and listening to users despite core team reservations and potential pitfalls.

Essentially we can’t take a lot of care validating and reordering arbitrary input so that “OpenSSL does the right thing” when CA certificates are renewed and their chain is altered. It obviously is a weak point in OpenSSL by itself and is best avoided. Commercial CAs do this too and it always ends in problems, I can tell you this much from previous work experience with S/MIME in particular.

It might make sense to add reordering to the system: trust: authorities page or try to reorder using issuer/subject CN while appending to trust store if that doesn’t take an a awful lot of computing time. Or just let people know that chains need to be reversed in the documentation to make sense of what’s going on.

FWIW, “trusted_first“ seems to be set in base OpenSSL as well so that’s definitely not our issue.

I also think that next ca_root_nss package will remove the expired CA and that should be it.

[kulikov-a]

thanks for the explanation!
I absolutely understand the requests of users to add certificates to the store: this allows us to quickly and globally add trust to the internal CAs of the organization (as I did with mine)) (another thing is that it may be worth doing this only for root certificates by default).
if I correctly understood the logic of the openssl devs from the description of the trusted_first and no_alt_chains options (https://www.openssl.org/docs/man1.1.0/man1/verify.html#OPTIONS), including the intermediate certificates in the store is the way to force the openssl to select a specific chain. therefore I consider the current write order to be correct, and the behavior of the openssl as intended (but of course I did not find clear evidence of my version other than the openssl docs and TLS RFC).

I also think that next ca_root_nss package will remove the expired CA and that should be it

as my tests showed (manualy delete DST cert from ca_root_nss and trigger system_trust_configure()), this shouldn't and doesn't fix the situation. only the reason of the error will change: "unable to get issuer certificate"

I checked another option for a quick hotfix (not a complete solution with the ability to block trust of any certificate, but a PoC that allows users to exclude certs in System: Trust: Authorities from store). something like kulikov-a/core@636f3a0
(of course, this only makes sense if I correctly understood the openssl chains logic)