HTTP error: Peer certificate cannot be authenticated with given CA certificates

[Ageless93]

Today, September 30 2021, the IdentTrust DST Root CA X3 expired, which may be the cause that a lot of projects using this Let's Encrypt certificate cannot be contacted anymore by BOINC. We need an update in ca-bundle.crt

HTTP error: Peer certificate cannot be authenticated with given CA certificates
Scheduler request failed: Peer certificate cannot be authenticated with given CA certificates

Projects we know of that are affected are: GPUGrid, CPDN and iThena. Probably more.
Let's Encrypt switched to its own ISRG Root X1 certificate earlier this year.

For more on this: https://techcrunch.com/2021/09/21/lets-encrypt-root-expiry/

[nicolas17]

The new ISRG X1 certificate has been in ca-bundle.crt since cb0832b so that's not the problem.

Is BOINC for Windows still shipping an old OpenSSL version?

[nicolas17]

Apparently BOINC is using OpenSSL 1.0.2s (released in Sept 2019 and probably with several known security issues).

OpenSSL 1.0 sees the X3 certificate expired and gives up, instead of noticing the project's certificate is also signed with X1 which is still valid. The fix is updating to OpenSSL 1.1.

I already mentioned this in 2020 when the AddTrust certificate expired and we had the same problem, which would have also been avoided by using a newer OpenSSL.

[cwallbaum]

Yep, and if you run Linux and don't get any updates or cannot update your packages atm, you can deactivate the certificate at least on Debian based distros via

# sudo sed -i 's/mozilla\/DST_Root_CA_X3/!mozilla\/DST_Root_CA_X3/' /etc/ca-certificates.conf
# sudo update-ca-certificates

[nicolas17]

https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

It seems projects can work around this by using certbot --preferred-chain "ISRG Root X1" when getting their letsencrypt certificate, this gives compatibility with old OpenSSL (while breaking compatibility with old Android).

As a workaround on the client, it seems the expired X3 certificate can be removed from ca-bundle.crt.

(I haven't actually tested any of this myself)

The real fix is still to update OpenSSL.

[cwallbaum]

These are the certificates which will expire next:

Not After : Dec 15 08:00:00 2021 GMT
        Subject: OU=GlobalSign Root CA - R2, O=GlobalSign, CN=GlobalSign

Not After : Dec 15 08:00:00 2021 GMT
        Subject: O=Cybertrust, Inc, CN=Cybertrust Global Root

Not After : Dec  8 11:10:28 2022 GMT
        Subject: C=NL, O=Staat der Nederlanden, CN=Staat der Nederlanden EV Root CA

Not After : Mar  3 12:09:48 2023 GMT
        Subject: C=TR, L=Ankara, O=E-Tu\xC4\x9Fra EBG Bili\xC5\x9Fim Teknolojileri ve Hizmetleri A.\xC5\x9E., OU=E-Tugra Sertifikasyon Merkezi, CN=E-Tugra Certification Authority

Not After : May 15 04:52:29 2023 GMT
        Subject: C=HK, O=Hongkong Post, CN=Hongkong Post Root CA 1

Not After : Sep 30 04:20:49 2023 GMT
        Subject: C=JP, O=SECOM Trust.net, OU=Security Communication RootCA1

[Ageless93]

Other projects affected are WuProp, Amicable Numbers, SiDock@Home.
Can we get an ETA on if this is going to be fixed within a reasonable time period?

Else, Richard has put up a ca-bundle.crt with the affected certificate removed, it's available from his Google Drive.

[RichardHaselgrove]

I see we have a new client_release/7/7.16.20 branch this morning. I've downloaded it, compiled it under VS2013 (no errors), and started running it. Looks like a clean start so far, but I'll keep prodding it.

But we still have
04/10/2021 11:55:02 | | Libraries: libcurl/7.42.1 OpenSSL/1.0.2s zlib/1.2.8
so we're storing up for yet another round of this type of emergency in the future.

Crazy thought: I don't think there's anything magical about the order in which the certificates are assembled in the ca-bundle.crt file. Could we assemble our own bundle, with the longest-lived certificates at the start, and the ones likely to expire soonest at the end? It won't be trivial, but I know how to do it - just boring and fiddly.

It would at least stop an expired certificate near the start (this time, it was number 28 in the bundle which went bad) blocking OpenSSL's onward search for a replacement.

[AenBleidd]

Could we assemble our own bundle, with the longest-lived certificates at the start, and the ones likely to expire soonest at the end?

I'd prefer to use the file that is assembled and maintained by someone else (preferably more experienced) like Mozilla, then we don't need to do an extensive testing of it

[AenBleidd]

Should have been fixed by #4540

[RichardHaselgrove]

Well, we've got to change something so that we don't keep letting down the science projects that rely on us for their research results. A better version of OpenSSL would be better, agreed, but we've been told already that we need it, and done nothing.

A replacement bundle could at least be updated 'in the field', unlike the runtime DLLs which you incorporated into the application under VS2019.

[AenBleidd]

7.16.20 is going to be a hotfix release and nothing more.
I'm looking into building curl for Windows that will not use ca-bundle at all in favor of Windows certificate store.
That and updated 3rdparty dependencies are gonna be a next major release

[RichardHaselgrove]

Thanks, but please make sure it doesn't drop off the 'ToDo' list.

We've already been warned that the 'GlobalSign Root CA - R2' certificate is due to expire in just over 2 months time, on 15 December this year, and that's number 2 in the new bundle

[AenBleidd]

Thanks, but please make sure it doesn't drop off the 'ToDo' list.

#4542

We've already been warned that the 'GlobalSign Root CA - R2' certificate is due to expire in just over 2 months time, on 15 December this year, and that's number 2 in the new bundle

Assuming that I have only 1 major ticket to do for the next major release and it's currently in progress, 2 months should be enough

[Ageless93]

VeriSign Class 3 Public Primary Certification Authority - G5 has a first next expiration date of 2021-11-07 23:59­:59 UTC but many after that till 2035 and further, so it may not be a problem ( https://ssl-tools.net/subjects/b9e9b287028503f8eca5fb42e13e0f49c72426e2)

…

-- Jord

On Mon, Oct 4, 2021 at 1:58 PM Vitalii Koshura ***@***.***> wrote: Thanks, but please make sure it doesn't drop off the 'ToDo' list. #4542 <#4542> We've already been warned that the 'GlobalSign Root CA - R2' certificate is due to expire in just over 2 months time, on 15 December this year, and that's number 2 in the new bundle Assuming that I have only 1 major ticket to do for the next major release and it's currently in progress, 2 months should be enough — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4530 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACS5WU3XVIKZYWXKZ7HJXVDUFGJFNANCNFSM5FC4IPOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[cwallbaum]

@Ageless93: The one found in our ca-bundle.crt (for years now - as far as i see) will expire in 2036:

    Data:
        Version: 3 (0x2)
        Serial Number:
            18:da:d1:9e:26:7d:e8:bb:4a:21:58:cd:cc:6b:3b:4a
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc. -
 For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5
        Validity
            Not Before: Nov  8 00:00:00 2006 GMT
            Not After : Jul 16 23:59:59 2036 GMT
        Subject: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc.
- For authorized use only, CN=VeriSign Class 3 Public Primary Certification Authority - G5
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                ...

But: Mozilla already removed Symantec's certificates (VeriSign, GeoTrust, Thawte...) from Firefox in October 2018. That's why it's also not found in the latest ca-bundle.crt anymore which Vitalii committed today.

Let's hope for the best... ;)

[nicolas17]

We've already been warned that the 'GlobalSign Root CA - R2' certificate is due to expire in just over 2 months time, on 15 December this year, and that's number 2 in the new bundle

Why does it matter that it's "number 2"?

Does it make any difference if you move the expired X3 certificate to the end of the file? I wouldn't expect the order to matter at all.

[RichardHaselgrove]

I suggested moving the dangerous ones to the end, but that was not welcomed.

Here are the first 8 to expire:

BOINC certificate expiries.pdf

• numbers 2, 24, 72, 56, 28, 9, 5, 23

The problem is OpenSSL stopping processing when it encounters an expired one. If we can lose OpenSSL, it'll be easier.

[brevilo]

Does it make any difference if you move the expired X3 certificate to the end of the file? I wouldn't expect the order to matter at all.

And you are right, it doesn't for OpenSSL 1.0.2 since any untrusted chain gets preferred over the trusted ones.

[RichardHaselgrove]

Confirmed for our current situation: I moved the expired certificate to the very end of the bundle, and it still blocked comms.

I said it was a crazy idea, and so it turned out to be. Consign to history.

[tristanolive]

It seems projects can work around this by using certbot --preferred-chain "ISRG Root X1" when getting their letsencrypt certificate, this gives compatibility with old OpenSSL (while breaking compatibility with old Android).

I can confirm that this works with BOINC clients and also other applications compiled with libssl1.0. We have a stats process that uses curl on an old Debian 9 system (where curl is linked to libssl1.0.2) and experienced the same problem and the same short-term solution (upgrade of the OS being the appropriate step to take next).

However, we also have a significant number of old clients connecting to Charity Engine that do not even have the ISRG Root X1 certificate in ca-bundle.crt. In this case, the certbot workaround is ineffective; it is a strict incompatibility. So projects that intend to continue to support old clients can instead acquire certificates from another CA.

Determining which CA to use is an individual judgment call, but it helps to examine an old version of ca-bundle.crt to find a root certificate that:

• would be present in old clients; and
• will not be approaching expiration for a long time.

In our case, we put a low cost certificate in place with a cross-signed / dual chain of USERTrust RSA Certification Authority and AAA Certificate Services, which are valid until 2038 and 2028, respectively. No doubt there are other options. (If there's any need at projects, a subset of such certificates could be identified. Find suitable certs included in Firefox and cross reference with the desired commit history of ca-bundle.crt to find a CA that can support old builds).

Upgrading the version of OpenSSL included with BOINC seems like a very good move for future consideration. I also like the idea of relying on the OS certificate store, though I expect there are edge cases where that might not work well. This whole topic is just an area where there is no single solution that will work for everyone, so cautious development is good.

[mblumberg]

(i) As a point of reference, here's a draft document outlining server-side updates for affected projects: Let's Encrypt Certificate Issues.
-- It's a rough draft; open to public updates for anyone who wants to contribute.