ISRG Root X1 Certificate not trusted

[chris246]

Hi,

for some reasons, Nextcloud Desktop started throwing errors today about being unable to securely connect to my server that uses a valid Let's Encrypt certificate signed by R3 (intermediate) and ISRG Root X1 (root certificate).

Visiting Nextcloud via Browsers (Firefox, Edge, Internet Explorer) all works on the same PC.
On a different PC, the issue resolved itself after updating to Nextcloud Desktop 3.3.5, but on this PC that did not help.

I am aware of Let's Encrypt switching to a new root CA: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
The errors started showing up today, after the previously used cross signed intermediate/root expired: https://letsencrypt.org/certificates/

The problem seems to be, that the certificate of the issuer DST Root CA X3 expired and therefore the certificate ISRG Root X1 can't be validated, even though it should be trusted on its own.

ISRG Root X1: https://crt.sh/?id=3958242236
DST Root CA X3: https://crt.sh/?id=8395 (expired September 30)

---

Solution:

1. Open this link (or any other website with a valid Let's Encrypt certificate in MS Edge or Internet Explorer: https://valid-isrgrootx1.letsencrypt.org/ (This updates the Microsoft Certificate Store)
2. Restart Nextcloud Desktop

See: #3858 (comment)

---

Client configuration

Client version: 3.3.3, 3.3.4, 3.3.5
All three versions show this error

Operating system: Windows 10

OS language: German

Full error message (sorry, German only):

Kann keine sichere Verbindung zu **** herstellen:
Das Zertifikat des Ausstellers eines lokal gefundenen Zertifikats konnte nicht gefunden werden
mit Zertifikat ISRG Root X1
Organisation: Internet Security Research Group
Einheit: <nicht angegeben>
Land: US
Fingerabdruck (SHA-256): 6d:99:fb:26:5e:b1:c5:b3:74:47:65:fc:bc:64:8f:3c:d8:e1:bf:fa:fd:c4:c2:f9:9b:9d:47:cf:7f:f1:c2:4f
Fingerabdruck (SHA-512): 7a:dc:2b:5f:11:e5:d1:2d:f7:ad:b6:ce:e9:5e:04:f7:ec:a7:14:40:4b:ff:58:84:9a:36:0b:91:0f:3a:fb:dc:37:23:5c:dd:99:e3:3b:4e:82:ef:ee:e1:6d:59:85:73:a4:e3:46:e0:a6:bd:c4:1f:70:b3:60:3c:6f:43:24:fa

Datum des Inkrafttretens: Mi Jan 20 19:14:03 2021 GMT
Ablaufdatum: Mo Sep 30 18:14:03 2024 GMT
Aussteller: DST Root CA X3
Organisation: Digital Signature Trust Co.
Einheit: 
Land: 

[heeplr]

Happening here as well (also german Windows 10 Systems). Version 2.6.2stable-Win64 (build 20191224) and 3.2.3 seem unaffected.

[aPollO2k]

Ich have the same issue. Only with the nextcloud client. I use the certificate in Browsers, Mail Clients, FTP Servers with no issues. But Nextcloud client prints this warning.

My Workaround is to remove the old DST Root CA X3 Certificate from the Chain (WARNING: This breaks the compatibility with older client see here for more details: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ )

If you use acme.sh you can re issue the certificate with the --preferes-chain "ISRG Root X1" argument. See here: https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain

Example
acme.sh --issue -d nextcloud.com --preferred-chain "ISRG Root X1" --force --server letsencrypt

[megmug]

On my windows machines, I see the same problem. Interestingly, on my Ubuntu desktop with Nextcloud Client 3.3.4 it still works.

[martin-rueegg]

Same here. My workaround - after verifying the fingerprint - was to trust the certificate anyway (needs to be done for every client installation and for every linked account therein separately).

SHA256-Fingerprint of the cross signed certificate can be done as follows:

$ curl https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem | openssl x509 -in - -noout -sha256 -fingerprint
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1955  100  1955    0     0   7548      0 --:--:-- --:--:-- --:--:--  7548
SHA256 Fingerprint=6D:99:FB:26:5E:B1:C5:B3:74:47:65:FC:BC:64:8F:3C:D8:E1:BF:FA:FD:C4:C2:F9:9B:9D:47:CF:7F:F1:C2:4F

[ElchWG]

I have the same problem on my Windows machine with client version 3.3.4 and we don't even use Let's Encrypt on our server installation. We have a valid DigiCert wildcard certificate for our domain. Access via browser works fine with that certificate.

I also found a workaround for an older version (2.x.y-ish) of the desktop client in the german help forum (see https://help.nextcloud.com/t/ungultiges-zertifikat-nextcloud-client/45327) describing the exact same problem with another certificate that broke.
Workaround: Once the error pops up simply close the app and restart it, then it doesn't bother with the broken cert and works fine.

The workaround did the trick for me this morning, so may be the bug is also related to a similar problem like the one in 2019?

[vekin03]

Same here.

Since yesterday, my NextCloud client (3.3.4) on Windows 10 display an error regarding my (Let's Encrypt) certificate.

I tried to force renew the certificate, reboot the server (Debian 9), install the last updates... but nothing works.

In the meantime, the web UI of NextCloud works without warning and so for the Android app.

[qupfer]

on the website, there is now an new version 3.3.5 which seems to fix the issue

[chris246]

This only fixed the issue on one PC for me, on the other I still get the error...

[msklywenn]

Updating to 3.3.5 didn't fix the issue for me.

[ph00lt0]

Issue seems only to appear on windows clients.

[FlexW]

Just to let you know. We are working on a fix for that.

[jheyens]

This also breaks the auto updater. The domain "updates.nextcloud.org" is also signed using an LE certificate rooted in ISRG Root X1. You won't be able to push the update to affected Windows users, unless you temporarily switch to another trusted certificate on at least the domain "updates.nextcloud.org"

[FlexW]

The people having problems, do you have the ISRG Root X1 certificate installed in your certificate store on Windows? You can find out if you type certmgr into the Windows search and then open the folder "Trusted Root Certification Authorities" -> Certificates and then look after the ISRG Root X1 certificate.

[jheyens]

Yes.

[jheyens]

[UeliDeSchwert]

I don't have a trusted root CA named "ISRG Root X1" (or anything with ISRG for that matter) in my trusted root CAs, but I have the same problem with the nextcloud client.

This certificate, however, is there in Firefox. Might this have any influence?

[jheyens]

I don't have a trusted root CA named "ISRG Root X1" (or anything with ISRG for that matter) in my trusted root CAs, but I have the same problem with the nextcloud client.

This certificate, however, is there in Firefox. Might this have any influence?

Firefox ships with its own certificate storage. The Windows system store isn't used by Firefox.

[FelixSchwarz]

I had the problem but it is gone now after installing Windows updates.

However I think I can tell what changed so maybe this is helpful:

• Previously in certmgr.msc (like 1h ago) I did not see "ISRG Root X1" in "Trusted Root Certification Authorities" but only in "Drittanbieter-Stammzertifizierungsstellen" ("Third-party Root Certification Authorities"?).
• I could access web sites with Let's Encrypt just fine even with Microsoft Edge.
• Then I installed KB5005539 and KB5005611, rebooted
• "ISRG Root X1" is now present in "Trusted Root Certification Authorities", NextCloud does not show errors anymore :-)

[jheyens]

That seems to solve the issue here, too.

[theOehrly]

Yes, you're right. It's suddenly fixed now without even restarting Windows. If it was a Windows update, then it was applied automatically in background for me.
I do have "ISRG Root X1" in "Trusted Root Certification Authorities" now. I don't know if it was there before.

Edit: I did need to restart the Nextcloud Client though.

[jheyens]

I may have looked into the system store during/shortly after the application of the update.

I did not need to restart the Nextcloud Client.

[FelixSchwarz]

The Windows updates may be a red herring though: ISRG Root lazy loading problem + missing from (random) updated Windows 10 versions.

Basically what is said in the Let's Encrypt forum is that the ISRG root certificate can be lazy-loaded into the "Trusted Root Certification Authorities" and that can be triggered with various applications. From what I read just using Microsoft Edge to access any modern Let's Encrypt site could trigger the inclusion of the X1 root?

[UeliDeSchwert]

@FelixSchwarz I think I can approve what you just said: I did not have ISGR Root X1 in certmgr.msc, then I used Edge to access a page that uses a LE-Cert, and now the ISGR Root X1 is there in the certmgr.msc

[theOehrly]

@FelixSchwarz I checked my windows update history and you are correct. There were no recent updates applied.

[chris246]

Just took a look at my affected laptop (was shut down overnight, not rebooted):
ISRG Root X1 is now present in the windows certificate store and Nextcloud Desktop syncs again...

[vekin03]

@FelixSchwarz I think I can approve what you just said: I did not have ISGR Root X1 in certmgr.msc, then I used Edge to access a page that uses a LE-Cert, and now the ISGR Root X1 is there in the certmgr.msc

I suppose that's what this page is supposed to do.

[Tras2]

I agree that the fix is to use Edge to browse to https://valid-isrgrootx1.letsencrypt.org/ which causes the new root certificate to be trusted.

I believe its the "Cryptographic Services" service which updates the trusted root store dynamically

[mgallien]

@FelixSchwarz @UeliDeSchwert and others thanks a lot for your help in better understanding the issue
I guess there is not much we can do within the desktop client to fix that

[FelixSchwarz]

@mgallien Well, if Edge can trigger the migration maybe the desktop client could do that as well?

[Gomez]

We at https://portknox.net (Nextcloud hosting provider) decided to remove the "DST Root CA X3" certificate from our chain. Looks like this solves the issues with clients (and the Joplin app - all Electron apps?), but could make problems on older devices/software.

On Debian:

• In /etc/ca-certificates.conf add a "!" before mozilla/DST_Root_CA_X3.crt
• Run update-ca-certificates (one cert should be removed)
• Make sure you config PREFERRED_CHAIN="ISRG Root X1" for certbot/acme/dehydrated (Should be the default, now?)
• Force a renew of the certificate
• Restart webserver

Our customers reported no further problems..

[leolivier]

My 2 cents: I had the same issue and after visiting the https://valid-isrgrootx1.letsencrypt.org/, the certificate was installed but I had to restart the nextcloud desktop client to get it working again.

[megmug]

My 2 cents: I had the same issue and after visiting the https://valid-isrgrootx1.letsencrypt.org/, the certificate was installed but I had to restart the nextcloud desktop client to get it working again.

Same for me. I think it would be worth trying to find out how the nextcloud client could trigger the same effect by itself.

[EVOTk]

Thank you for the explanation.
For French users, I made an explanatory note of the procedure here: [Mémo] Régler le soucis du certificat DST Root CA X3 expiré

[leolivier]

@EVOTk, great but you didn't say the link has to be fetched with Edge to be sure to download the certificate (e.g. it won't work with Firefox)

[EVOTk]

Hello,
@leolivier thx you for the comment.
I will add to be note. On the other hand I am surprised because I did it on several PCs including 2 via Firefox without worries.

[leolivier]

I was referring to some comments above saying that Firefox is not using the windows certificate store but it's own one

[rondadon]

Interesting. Even after Updating Windows, restarting etc. the Problem was not resolved.
I needed to open MS Edge Browser and access a Website which uses LE-Certificate (for obvious reasons you could acces your Nextcloud instance from Edge) and onyl after that the "ISRG Root X1" Root Certificate was installed/listed in "certmgr.msc" under "Trusted Root Certificates" . So indeed it is a Windows Problem. Was reported to me from two clients and I could replicate the problem on another Windows Client.

This was a PITA! Now its working again and no Untrusted Certificate message in the Nexcloud Desktop Client anymore.!

Thank you for discussing and finding the culprit!
Greetings

[mgallien]

We at https://portknox.net (Nextcloud hosting provider) decided to remove the "DST Root CA X3" certificate from our chain. Looks like this solves the issues with clients (and the Joplin app - all Electron apps?), but could make problems on older devices/software.

On Debian:

* In `/etc/ca-certificates.conf` add a "!" before `mozilla/DST_Root_CA_X3.crt`

* Run `update-ca-certificates` (one cert should be removed)

* Make sure you config `PREFERRED_CHAIN="ISRG Root X1"` for certbot/acme/dehydrated (Should be the default, now?)

* Force a renew of the certificate

* Restart webserver

Our customers reported no further problems..

Thanks for letting us know about your procedure
That is an alternate way to solve it that could help organizations hosting Nextcloud.

[mgallien]

@mgallien Well, if Edge can trigger the migration maybe the desktop client could do that as well?

hard to say
I have no idea which API is needed to force Windows to trust the root certificate that is now the only valid one for let's encrypt certificates.

[splitt3r]

What about pinning this issue for a month or so? So users of the client will find this issue quicker. And also adding the solution as a note to the original question?

[chris246]

Thanks @splitt3r: I update the original question as suggested

[Alternativend]

Im desperately waiting for a fix.
Any news on this?

[alpha14]

The provided solutions didn't work for me (the website https://valid-isrgrootx1.letsencrypt.org/ was displaying correctly though)

For a Windows 10 client, I ended up manually installing the cert by going directly to https://letsencrypt.org/certs/lets-encrypt-r3.der (see https://letsencrypt.org/certificates/ for more info)

[FelixSchwarz]

Background for lazy loading + some technical background how to trigger loading the certificate (without user interaction) in the Let's Encrypt community: https://community.letsencrypt.org/t/microsoft-windows-root-certificate-lazy-loading/160389/4

[ansha-2]

Hello,

on Windows 10 OS I have follow the solution in the head but the problem was persist.

I resolved after I have install the follow R3 CA as trusted CA on my Windows system.

https://letsencrypt.org/certs/lets-encrypt-r3.der

Thanks you.

[mrEscow]

1. conan remote remove conan-center
2. conan remote add conancenter https://center.conan.io --insert 0

[mgallien]

let's close this as I do not think we can do anything meaning full here

[lmamane]

For those that run Debian and have the same problem, note that Debian bug 1016085 has as consequence the same symptoms if you have libssl-dev installed in any other version than 3.something AND you have libqt5network5 version 5.15.4. If you do, either uninstall libssl-dev or upgrade it to version 3.

I think you'll also get the version if you have libqt5network5 version 5.15.2 and libssl-dev any other version than 1.1.something. Fix accordingly.

[Justinzobel]

Looks like this started happening again today for me.