security/acme-client: Lets Encrypt certificates are provided by obsolete DST Root CA X3 #2550

cedvan · 2021-09-28T10:02:24Z

Important notices
[x] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
[x] I have searched the existing issues and I'm convinced that mine is new.
[x] When the request is meant for an existing plugin, I've added its name to the title.

Is your feature request related to a problem? Please describe.
Lets Encrypt certificates are provided by DST Root CA X3 in acme plugin. But this root CA is deprecated and expire in september 2021 (https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/)

Describe the solution you'd like
Use new root certificate ISRG Root X1 is a new way. How use it with acme plugin ?

fraenki · 2021-09-28T12:14:03Z

@cedvan Maybe I'm missing something here? As far as I'm aware, Let's Encrypt added a new cross-signed CA which should work beyond 30.09.2021:

R3 ← ISRG Root X1 ← DST Root CA X3

I've manually renewed one of my certificates and the two parts of the chain seem to confirm that it's using the new cross-signed CA:

# openssl x509 -noout -text -in /tmp/chain1.crt 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3
...

# openssl x509 -noout -text -in /tmp/chain2.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
...

And viewing the resulting cert chain in Firefox 92 it seems that it already prefers the X1 route over the expiring X3. However, I can confirm that Chrome still shows the the same output that you've posted in the screenshot.

Yes, there is an alternative chain available:
R3 ← ISRG Root X1
But it provides worse compatibility with older clients and should not provide any benefits.

Does this make sense to you? Is there something wrong with this explanation?

cedvan · 2021-09-28T13:58:13Z

Hum,

On my servers without DST Root CA X3, certificates provided by acme-client are not valid.

curl command respond :

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

My servers have ISRG_Root_X1.pem but no DST_Root_CA_X3.pem in CA certificates list.

fraenki · 2021-09-28T14:00:08Z

@cedvan Which service uses the certificate, is it the OPNsense WebUI or something else?

cedvan · 2021-09-28T14:00:45Z

I use certificates with HAProxy frontend

fraenki · 2021-09-28T14:01:46Z

@cedvan Can you verify the chain by using openssl on the CLI?

openssl s_client -showcerts -connect YOUR_HAPROXY_FQDN:443

cedvan · 2021-09-28T14:04:39Z

CONNECTED(00000005)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:CN = YOUR_HAPROXY_FQDN
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
-----END CERTIFICATE-----
---
Server certificate
subject=CN = YOUR_HAPROXY_FQDN

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3530 bytes and written 398 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: EF399C96D212C47AB2E7F5D29D96A2C8F0DB987A0E92BC5EC4A5100DAF86F531
    Session-ID-ctx: 
    Resumption PSK: 3EB8086999FF6711495E6CA92FB4A81963224A9FA2BFF8E752A8D13C77FD9FE38F80583E7FD32B5D8E8F343360DDD032
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 3e 0e 69 78 07 b9 e4 57-b4 16 66 41 bd 09 73 ad   >.ix...W..fA..s.
    0010 - c9 3e aa bf c0 c1 09 bd-ca 97 bf a8 e6 85 68 75   .>............hu

    Start Time: 1632837812
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: BF92C190D8CE371F9BD2B177946C7E6FD23548E2A432052CEADA4037CB3BB7AC
    Session-ID-ctx: 
    Resumption PSK: 07582B9D51AB5E9FFDF81E544E47CB0F95E9CC42D1F3585DC8CCA4DDA7CC23280EBE01539DF4EE1600F052365703DD57
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - a1 17 1a 9d 9e 35 51 00-0a 9d 04 29 58 d8 c2 17   .....5Q....)X...
    0010 - e6 5c 65 60 74 68 78 f7-a5 b2 73 32 86 79 84 4a   .\e`thx...s2.y.J

    Start Time: 1632837812
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

cedvan · 2021-09-28T14:06:16Z

I have latest version to opnsense and all plugins. My certificate regenerated today, but no presence to ISRG_Root_X1

cedvan · 2021-09-28T14:18:01Z

My account in Acme client use Let's Encrypt[default]

cedvan · 2021-09-28T14:19:22Z

Maybe need recreate new account or recreate new certificate to get good CA hierarchy ?

fraenki · 2021-09-28T14:23:01Z

@cedvan Let's check one more thing:

go to System: Trust: Certificates
find your Let's Encrypt certificate
What is the issuer? Please paste the full name here.

cedvan · 2021-09-28T14:24:55Z

Issuer to all generated certificates with acme client is R3 (Let's Encrypt)

fraenki · 2021-09-28T14:35:55Z

OK, so the real problem is that os-acme-client failed to import the cross-signed R3 CA certificate and as a result applications such as the WebUI and HAProxy will only present the expiring R3 CA certificate to clients. I'll look into this.

cedvan · 2021-09-28T14:38:12Z

Ok, I look forward to your feedback and a solution :)

kulikov-a · 2021-09-28T15:26:58Z

@cedvan

My servers have ISRG_Root_X1.pem but no DST_Root_CA_X3.pem in CA certificates list.

sorry, what https://www.ssllabs.com/ssltest/index.html tells about cert path?
imho DST_Root_CA_X3 still needed for End-entity certificate ← R3 ← ISRG Root X1 ← DST Root CA X3 chain (default Android compatible chain)

cedvan · 2021-09-28T15:38:32Z

@kulikov-a Yes of course DST_Root_CA_X3 is still needed for old Android systems. To do this Let's Encrypt added a new cross-signed CA. This allows generated certificates compatible with ISRG_Root_X1 and DST_Root_CA_X3.

But ACME client generate certificates with old CA R3 (no compatbile with ISRG_Root_X1) instead of new cross-signed R3 CA. On recent systems, DST_Root_CA_X3 is removed because obsolete. So certificates become invalid on recent systems...

cedvan · 2021-09-28T15:43:06Z

This bug is quite urgent, because some applications will become unreachable after September 2021 (rest only 2 Days)

kulikov-a · 2021-09-28T16:00:28Z

@cedvan

But ACME client generate certificates with old CA R3 (no compatbile with ISRG_Root_X1) instead of new cross-signed R3 CA

not quite so (ACME client does not generate certificate)
so https://www.ssllabs.com/ssltest/index.html reports broken chain for your domain?

fraenki · 2021-09-28T16:10:31Z

I've found the bug, give me some minutes to test the fix :)

kulikov-a · 2021-09-28T16:17:37Z

@fraenki ok, thanks!)
i have production 20.7.7_1's and already working cross-signed LE certs. i just can't remember if i imported the new R3 cert manually or not )

fraenki · 2021-09-28T16:19:42Z

OK, here we go. The following patches for OPNsense 21.7.3 should fix the bug:

# apply the fix
opnsense-patch -c plugins ae69739 9220a41 247408e 774374a4

# restart backend service
service configd restart

Besides fixing the underlying bug in the cert/CA import logic, it also adds a new "import" button to Services: ACME Client: Certificates. By using this button the certificate should be linked to the correct CA. Afterwards you need to restart the service that uses the certificate.

I could only test it on my test server, so make a configuration backup before applying/testing this bugfix.

In my case it fixes the bug, the example shows the ouput when using the certificate for the OPNsense WebUI:

# without bugfix
$ openssl s_client -showcerts -connect opnsense:443
CONNECTED(00000003)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = opnsense
verify return:1

# after applying the bugfix and re-importing the cert
$ openssl s_client -showcerts -connect opnsense:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = opnsense
verify return:1

fraenki · 2021-09-28T16:25:17Z

(The width of the "commands" column had to be changed after adding the new button, this is fixed in 774374a.)

cedvan · 2021-09-28T17:09:08Z

Ok super! Thank you :)

So just I execute
opnsense-patch -c plugins ae69739 9220a41 247408e 774374a4 ?

And regenerate my cert and finally restart haproxy?

fraenki · 2021-09-28T18:44:13Z

You don't need to regenerate your certificates, just use the new "import" button:

Besides fixing the underlying bug in the cert/CA import logic, it also adds a new "import" button to Services: ACME Client: Certificates. By using this button the certificate should be linked to the correct CA. Afterwards you need to restart the service that uses the certificate.

kulikov-a · 2021-09-28T20:04:08Z

@fraenki Sorry, one more question: are you planning to add the option to select the preferred chain (https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain)?
thanks)
and thanks for the great job!

fraenki · 2021-09-28T20:44:58Z

are you planning to add the option to select the preferred chain (https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain)?

I'm still undecided. Adding a text input field is really error prone: a minor typo and it breaks. But adding a dropdown list of possible values puts the maintenance burden on me. Not sure which route to choose.

kulikov-a · 2021-09-29T05:22:27Z

got it, thanks! (will be glad to any option. a text input with a link to the wiki is quite convenient imho)

cedvan · 2021-09-29T07:11:02Z

I test patch, but not working :/.

My step :

Go to OpnSense machine in CLI and select shell option
Execute opnsense-patch -c plugins ae69739 9220a41 247408e 774374a4 => All patches have been applied successfully. Have a nice day.
Go to Services: ACME Client: Certificates and click on new button Re-import on my FQDN row
Restart HAProxy
Try curl call, same result : curl: (60) SSL certificate problem: unable to get local issuer certificate
Execute openssl s_client -showcerts -connect FQDN:443 : Same result, missing X1
Go check System: Certificats: FQDN issuer => R3 (Let's Encrypt)

cedvan · 2021-09-29T07:25:52Z

Ok, I regenerate my cert and restart HAProxy works now !!!

cedvan · 2021-09-29T07:31:14Z

Thank you 🙏

hny-gd · 2021-09-30T06:39:38Z

Any plans to release an official OPNSense patch for this or is manual patching like described by @cedvan above the way to go?

fichtner · 2021-09-30T06:41:26Z

Can we wait for this to be merged to any sort of branch in the project before we talk release timelines?

mimugmail · 2021-09-30T07:01:40Z

@hny-gd Just delete the old CA in Authorities, reissue all LE certs and link them again in the Services (HAProxy, Postfix etc.). Last step is important!

hny-gd · 2021-09-30T08:00:13Z

Thanks a lot, @mimugmail, that indeed is the way to go.

fraenki self-assigned this Sep 28, 2021

fraenki added the feature Adding new functionality label Sep 28, 2021

fraenki changed the title ~~acme-client : Lets Encrypt certificate are provided by obsolete DST Root CA X3~~ security/acme-client: Lets Encrypt certificates are provided by obsolete DST Root CA X3 Sep 28, 2021

fraenki added bug Production bug and removed feature Adding new functionality labels Sep 28, 2021

fraenki added a commit to fraenki/plugins that referenced this issue Sep 28, 2021

security/acme-client: fix CA association, closes opnsense#2550

247408e

cedvan closed this as completed Sep 29, 2021

matlink mentioned this issue Sep 30, 2021

ACME Client: expired let's encrypt intermediate CA is still retrieved #2552

Closed

opnsense locked as resolved and limited conversation to collaborators Sep 30, 2021

fraenki pinned this issue Sep 30, 2021

fichtner linked a pull request Sep 30, 2021 that will close this issue

security/acme-client: release 3.2 #2551

Merged

caizixian pushed a commit to caizixian/plugins that referenced this issue Oct 22, 2021

security/acme-client: fix CA association, closes opnsense#2550

cc5c337

fraenki unpinned this issue Dec 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

security/acme-client: Lets Encrypt certificates are provided by obsolete DST Root CA X3 #2550

security/acme-client: Lets Encrypt certificates are provided by obsolete DST Root CA X3 #2550

cedvan commented Sep 28, 2021 •

edited

fraenki commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021

cedvan commented Sep 28, 2021

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021 •

edited

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021

kulikov-a commented Sep 28, 2021

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

kulikov-a commented Sep 28, 2021

fraenki commented Sep 28, 2021

kulikov-a commented Sep 28, 2021

fraenki commented Sep 28, 2021 •

edited

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021 •

edited

fraenki commented Sep 28, 2021

kulikov-a commented Sep 28, 2021

fraenki commented Sep 28, 2021

kulikov-a commented Sep 29, 2021

cedvan commented Sep 29, 2021

cedvan commented Sep 29, 2021 •

edited

cedvan commented Sep 29, 2021

hny-gd commented Sep 30, 2021

fichtner commented Sep 30, 2021

mimugmail commented Sep 30, 2021

hny-gd commented Sep 30, 2021

security/acme-client: Lets Encrypt certificates are provided by obsolete DST Root CA X3 #2550

security/acme-client: Lets Encrypt certificates are provided by obsolete DST Root CA X3 #2550

Comments

cedvan commented Sep 28, 2021 • edited

fraenki commented Sep 28, 2021 • edited

cedvan commented Sep 28, 2021 • edited

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021 • edited

cedvan commented Sep 28, 2021 • edited

cedvan commented Sep 28, 2021

cedvan commented Sep 28, 2021

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021 • edited

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021

kulikov-a commented Sep 28, 2021

cedvan commented Sep 28, 2021 • edited

cedvan commented Sep 28, 2021 • edited

kulikov-a commented Sep 28, 2021

fraenki commented Sep 28, 2021

kulikov-a commented Sep 28, 2021

fraenki commented Sep 28, 2021 • edited

fraenki commented Sep 28, 2021

cedvan commented Sep 28, 2021 • edited

fraenki commented Sep 28, 2021

kulikov-a commented Sep 28, 2021

fraenki commented Sep 28, 2021

kulikov-a commented Sep 29, 2021

cedvan commented Sep 29, 2021

cedvan commented Sep 29, 2021 • edited

cedvan commented Sep 29, 2021

hny-gd commented Sep 30, 2021

fichtner commented Sep 30, 2021

mimugmail commented Sep 30, 2021

hny-gd commented Sep 30, 2021

cedvan commented Sep 28, 2021 •

edited

fraenki commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

fraenki commented Sep 28, 2021 •

edited

cedvan commented Sep 28, 2021 •

edited

cedvan commented Sep 29, 2021 •

edited