Fixing: Unacceptable TLS error

[jonathon-love]

hi,

this is a dupe of #502, and i think relevant to #2123

the command flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo fails on appveyor, with error: Can't load uri https://flathub.org/repo/flathub.flatpakrepo: Unacceptable TLS certificate i'm using the Ubuntu2004 image.

there is some discussion of how this might be fixed on #502, however there seems to be assumed knowledge as to what is actually failing here, which i don't possess. any tips on how this might be addressed?

i note that it's possible to install the flathub remote as a by-product of perfoming this sort of install:

flatpak install --user -y https://dl.flathub.org/build-repo/....;

i can install the remote by performing an installation as the above, then uninstall that, then i can install from flathub with:

flatpak install flathub ...

with thanks

[jonathon-love]

hi,

i think i've figured this out. this stems from the let's encrypt expiry. if i copy flathub.flatpakrepo to a server not protected with a let's encrypt certificate, the following succeeds:

flatpak remote-add --user --if-not-exists flathub https://my-server-here/path/to/flathub.flatpakrepo

if flathub can update to a non-let's encrypt cert, this will fix things for people using older OSes, etc.

[mkrajnak]

I can reproduce this on older RHEL installation (flatpak-1.0.9-12) and I found a workaround which is installing flathub through gnome-software.

[mcatanzaro]

Can you try to figure out which of curl's TLS backends flatpak is using on Ubuntu? As far as I can tell, I think the default is OpenSSL, which is the same we have on Fedora. But there should be no way OpenSSL should be failing with this simple chain unless it is way older than you have in Ubuntu 20.04.

Please paste the output of:

• curl -I https://flathub.org
• openssl s_client -verify_return_error -connect flathub.org:443
• For fun: gnutls-cli flathub.org

Your distro's command-not-found handler should be able to install these tools for you if you are missing them.

I can reproduce this on older RHEL installation (flatpak-1.0.9-12) and I found a workaround which is installing flathub through gnome-software.

Not good. What version of RHEL?

[mcatanzaro]

Can you try to figure out which of curl's TLS backends flatpak is using on Ubuntu?

I mixed this up with #2123. Suffice to say: distro is very relevant. The curl TLS backend used varies by distro and by distro version. RHEL 8 should be openssl. I think RHEL 7 would be... NSS? Or perhaps GnuTLS? I'm really not sure. Would have to check to be certain.

[mcatanzaro]

Also check to ensure you have ISRG Root X1 in your trust store. How to do this varies by distro. Using seahorse would be a safe bet that should work everywhere.

[kerivkennedy]

Hopefully someone can help me with this. I'm a total newb to linux
trying to install flatpak . I want to install Blender on my new Chromebook. I installed the flatpak and attempted the next step (flathub repository) and get the same error TLS certificate.

hopefully someone can explain in super easy terms (this was a Christmas present, so I;m just learning this. back in the mid 90s I thought nothing of using DOS commands to do everything on a computer. then windows went and made everything to easy. So I am not computer illiterate, just have been dumbed-down .

I know my chromebook is Lenovo and is using the ARM (my husband has the pixel slate and has helped trying to set up Prusa Slicer)

[hfiguiere]

Blender is not available ARM so it's gonna be pointless.

[jonathon-love]

Also check to ensure you have ISRG Root X1 in your trust store.

so i think there's two audiences interested in this issue ... 1. is people trying to install software, 2. is people publishing software.

for 1. i'm like "sure, i don't mind noodling around in my computer, figuring out what this seahorse thing is", etc.

for 2. i'm like ... "wait? so exactly what do i communicate to my [often complete novice] users?"

the exciting thing about flatpaks is that we can make software available to chromebooks, a whole class of user traditionally cut off from the linux ecosystem.

so it would be great if these sorts of issues could generate a more sympathetic response, and perhaps even a solution. it's easy to dismiss novice users, but i think there's real value in making things accessible to as broad an audience as possible.

a reasonably simple solution here would be to use a non-let's encrypt cert, or if that's too much, simply mirroring the flathub.flatpakrepo file on a separate server using a different cert, and instructing people to install from there instead.

with thanks

[mcatanzaro]

so it would be great if these sorts of issues could generate a more sympathetic response, and perhaps even a solution. it's easy to dismiss novice users, but i think there's real value in making things accessible to as broad an audience as possible.

I replied above with a request for several bits of info, which you haven't provided yet. Maybe you missed that? This issue is blocked on you, not on devs. If you provide the requested info, I'll take a look.

a reasonably simple solution here would be to use a non-let's encrypt cert,

No, we need to figure out what is actually wrong.

[mcatanzaro]

Also check to ensure you have ISRG Root X1 in your trust store. How to do this varies by distro. Using seahorse would be a safe bet that should work everywhere.

seahorse is a graphical application (possibly still installed by default on Ubuntu, last I checked?) so it should be pretty simple, but this step can be optional, as it's unlikely that you would be missing this cert.

[jonathon-love]

this is from the focal image running on appveyor

curl -I https://flathub.org
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  7451    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200 
server: nginx/1.18.0 (Ubuntu)
date: Mon, 27 Dec 2021 01:42:23 GMT
content-type: text/html
content-length: 7451
vary: Accept-Encoding
last-modified: Tue, 28 Sep 2021 14:41:06 GMT
vary: Accept-Encoding
etag: "61532982-1d1b"
accept-ranges: bytes
expires: Wed, 26 Jan 2022 01:42:23 GMT
cache-control: max-age=2592000
cache-control: public, 
o-transform

openssl s_client -verify_return_error -connect flathub.org:443
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = flathub.org
verify return:1
DONE
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = flathub.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGSjCCBTKgAwIBAgISA2hVQJ+XublatKmkBpXh7vpVMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTExMjcxNjQ4MDdaFw0yMjAyMjUxNjQ4MDZaMBYxFDASBgNVBAMT
C2ZsYXRodWIub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAt+7n
wacOzjEIMVNuqHvkqTdA/8Dti16ov6neWr7kLi4dO5bp4TQzszeMhJAg7USuZyPT
tTIIpNb6kOyCBVXTcPSOWQ1FYeHyOQW37GMdZMgwbB+/WTvzWOopyo2ZnMH+UOPt
raLLAkOD1Z7+2Qp/LTrgjzW5KbfpDCvoB5fO3UtMRypYpgKE0cLye9A68VsVCMe9
tvSIT0x3h5b8q4m4ES5dKpj7Djc6a08DOlCk52C0lh3Pj9EEhpbhqU3HLqn8Napg
MC2akq2qaf2WYeL1CpaAcAT4F7Owhw7B+mCpBhIYGt5Wdn0wL8M+3tQeHrLwE1nw
vgbtnMHEvH5wuPTqmMBwpdad2VjZ4wMT2WlfTVi7UMWkPzZjXObc0TViUaXky9yE
sGA7ZIHrb+zzCIqraV5+NgboJ6lU3JzyirUTOb0jRPEefj4Jg8SX9SruOoW/gGBC
ArHqFADECLUM7eVUwKE/SF4SleO97JB3t3+7/3oN2RFbeUQQwiZlioAkmqvp9kQ9
hChT5OS9tqKtvM1Mq04kBbB0eCUzIOAgMRh90xYb5pWV4Kl30wJ9QIlDOt5oiUqr
3BAefv1pNgqGirhhsDXSOTUjaYH+7gp68duWw6xj0fKjhGB1jwWUgs1/dobM9yIV
m9SyoZqVWSTcUsW76tm+pbi96CboCw969cXnJ9MCAwEAAaOCAnQwggJwMA4GA1Ud
DwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0T
AQH/BAIwADAdBgNVHQ4EFgQUGaHdM4ymQW3cTiLALicxIhzUwsEwHwYDVR0jBBgw
FoAUFC6zF7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUF
BzABhhVodHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9y
My5pLmxlbmNyLm9yZy8wRAYDVR0RBD0wO4ILZmxhdGh1Yi5vcmeCG2ZsYXRodWIu
d2ViYXBwcy5mbGF0aHViLm9yZ4IPd3d3LmZsYXRodWIub3JnMEwGA1UdIARFMEMw
CAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9j
cHMubGV0c2VuY3J5cHQub3JnMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHcA36Ve
q2iCTx9sre64X04+WurNohKkal6OOxLAIERcKnMAAAF9YoLsFwAABAMASDBGAiEA
mah6pmanMfd1QRm9MEZ8UmBhVDom6xaRfwTPj7pOn9QCIQCIHYW55/ruTYG7gN4n
nuHVoEaXxdVRwrMHVv9WlrGF/AB1AEalVet1+pEgMLWiiWn0830RLEF0vv1JuIWr
8vxw/m1HAAABfWKC7EMAAAQDAEYwRAIgKYYHJmws7alhezlTvQW6Qd3amZXc5vS8
GSvyvsXjL/YCIGp67kw8fwH0NAI8sW7sUBkd3vRiv4RYXNsDgR2ShkgtMA0GCSqG
SIb3DQEBCwUAA4IBAQCchy/xw5k3sQIvgJAO7I6EMB+RYkB/Frtod8CYj4rd5UVT
MAflOuel/hWQ0n9P2vcmSOvQrclowwi/FlnSaIpYfsHNMAo/+77ox7aNVRa9lfL/
OYoEM8jrtJAxgkRtlc4FCDA9XNGB80wQ/l+0+tu9SvYsuJzoRRJ/oxfdWf0NYM63
11yTBmRsh/CeJ9YF7pcqGcnJQB5Dlta+uDVNN4d7bLJVTki95qen0E1+yY4jRyM7
lc2fay3E3/Q3W9zNdy7uUvUgG7JCb7F0+zAEnrD8q/gIn+3Z5wwNCLJ9ewlBX374
RjQAqNt4sOjJkAAvU2JKv1t55xwB/kv6NTVsPyxl
-----END CERTIFICATE-----
subject=CN = flathub.org
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5126 bytes and written 383 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

gnutls-cli flathub.org
Processed 128 CA certificate(s).
Resolving 'flathub.org:443'...
Connecting to '46.235.230.111:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `CN=flathub.org', issuer `CN=R3,O=Let's Encrypt,C=US', serial 0x036855409f97b9b95ab4a9a40695e1eefa55, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-11-27 16:48:07 UTC', expires `2022-02-25 16:48:06 UTC', pin-sha256="8UTCF0B/aVq34SPe6dHfibtHlmhJqeCpKre23k1yckc="
	Public Key ID:
		sha1:9170621f8b1c2b5301824069beffebb1542da8ce
		sha256:f144c217407f695ab7e123dee9d1df89bb47966849a9e0a92ab7b6de4d727247
	Public Key PIN:
		pin-sha256:8UTCF0B/aVq34SPe6dHfibtHlmhJqeCpKre23k1yckc=
- Certificate[1] info:
 - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[2] info:
 - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US', issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30 18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is trusted. 
- Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Options: OCSP status request,
- Handshake was completed
- Simple Client Mode:
- Peer has closed the GnuTLS connection

[mcatanzaro]

Er, but all of those succeeded. Hmm. Please try your original failing command again to confirm it is really still failing:

$ flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Is it really still broken? If only flatpak fails to handle flathub.org, that would be weird, but would certainly indicate a flatpak bug.

I just noticed this is the flathub issue tracker. This issue should be closed as there is obviously nothing wrong with flathub here. We can continue here to figure out which component is really to blame, then move to a more appropriate location.

[mcatanzaro]

I just noticed this is the flathub issue tracker. This issue should be closed as there is obviously nothing wrong with flathub here.

BTW I should clarify: having examined the TLS certificate chain, it's clear that flathub.org is presenting a good chain. Whatever is going wrong is a bug on the client side software (ca-certificates -> openssl -> libcurl -> flatpak), not on flathub's end.

That's not to say that this was a bad place to originally report the issue, but that we now know enough to confidently say it's not the right place to fix it.

[Aditya-ds-1806]

Trying to install a flatpak on RHEL 7, facing the same problem. Was there a fix found?
@mkrajnak Can you elaborate what you did? I am on RHEL 7 and flatpak 1.0.2

[mcatanzaro]

Trying to install a flatpak on RHEL 7, facing the same problem. Was there a fix found?

No, the last action in this issue was a request for me for further info. Unless you have the exact same problem (have run the same commands requested above and verified your output is the same) and can provide the last requested info, please don't assume this issue is really the same as yours. Actually, even then, please use a separate issue, because this ticket is about Ubuntu 20.04, which is way different from RHEL 7. libcurl on RHEL 7 uses NSS, so it's a completely different environment from anywhere else. Whereas Ubuntu 20.08 uses OpenSSL, which is at least expected to work.

P.S. There is no way flatpak applications will work properly on RHEL 7 because the host system is just not new enough. You're going to be missing p11-kit-server, xdg-desktop-portal[-gtk], and more. So honestly, I don't recommend even trying to get it to work. But if you really want to try, I would create a separate issue report for starters.

[Aditya-ds-1806]

@mcatanzaro I was able to install flatpak on a rhel 7 using flatpak remote-add --user --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo. I also feel it is a client-side issue as I had to clear the entries in /home/<user>/.local/share/flatpak/repo/config, after which it seemed to have worked.

[mcatanzaro]

I also feel it is a client-side issue as I had to clear the entries in /home//.local/share/flatpak/repo/config, after which it seemed to have worked.

Er... are you sure you simply didn't have it configured already?

If you deleted the configuration that was causing the command to fail without saving it somewhere, then of course you've probably lost the reproducer required for a bug report....

[Aditya-ds-1806]

It was on a virtualbox RHEL 7.6 VM. I can try to reproduce the problem on a new VM and get back to you. @mcatanzaro

[nanonyme]

My hunch is the original case has something to do with https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ (and the fact that the second root CA expired) but I can't figure out how.

[mcatanzaro]

My hunch is the original case has something to do with https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ (and the fact that the second root CA expired) but I can't figure out how.

That was my first thought too, but it shouldn't be happening because Ubuntu 20.08 has the ISRG Root X1 certificate (we know this because all the commands that I asked for succeeded). And the OpenSSL version in Ubuntu 20.08 is also new enough to handle the chain of trust.

IMO there's nothing more to do here besides to wait for Jonathon to confirm whether the original command is really still failing. Maybe there really was a bug with flathub.org, but it was transient and already long since fixed? Who knows.

It was on a virtualbox RHEL 7.6 VM. I can try to reproduce the problem on a new VM and get back to you.

If you're a Red Hat customer, you can report this downstream using a customer case, and the NSS developers might be willing to investigate. I see the ca-certificates package in RHEL 7.6 has been upgraded recently, so it ought to have ISRG Root X1, but maybe the old version of NSS is tripping over the expiration of DST Root CA X3. I certainly wouldn't expect this to work due to the very unusual configuration and old software versions, so from an upstream perspective, I wouldn't worry about this any further unless you can reproduce in RHEL 8, where things are much more normal (curl using a recent version of OpenSSL) and would be expected to work. If you really really need things to work in RHEL 7, I would try rebuilding the libcurl package to use OpenSSL instead of NSS, which ought to be able to handle this chain fine. One thing's for sure: it's extremely unlikely to be the same problem that Jonathon is experiencing, since you are using entirely separate distros with different curl backends and different ca-certificate packages.

[jonathon-love]

IMO there's nothing more to do here besides to wait for Jonathon to confirm whether the original command is really still failing.

hi, apologies for the delay,

yes, the command seems to be now working. i'll touch base with some of my users and see if it is resolved for them now too.

i'd close this issue, but i'm not quite sure if i should with the subsequent discussion ... so i'll leave it to you guys.

thanks for you help.

[nanonyme]

Something I noticed: libsoup in Debian does not pull in ca-certificates while libcurl does. Curl package also does. So most likely in various installations by the time you start debugging CA issues, they become fixed through testing.

[mcatanzaro]

i'd close this issue, but i'm not quite sure if i should with the subsequent discussion ... so i'll leave it to you guys.

It should be closed.

[nanonyme]

@jonathon-love yes, please, close this. The ca-certificates dependency mess is something Debian should resolve.

[iamihgam]

This is what I did and It resolved the problem for me:
$ sudo dpkg-reconfigure ca-certificates
On entering please deselect "DST Root CA X3". save and run the flathub repo script. It works .
Please see:
https://serverfault.com/questions/1079199/client-on-debian-9-erroneously-reports-expired-certificate-for-letsencrypt-issue

[mcatanzaro]

On entering please deselect "DST Root CA X3". save and run the flathub repo script. It works .

Interesting! This should definitely not be required. That certificate should be ignored because it is expired. I think you have proved that there is a Debian bug here. If you are using a supported version of Debian (either Buster or Bullseye) then you should (1) determine whether there is a problem with libcurl's OpenSSL backend, or a problem with OpenSSL itself, using the commands I've provided here, then (2) report a bug against the appropriate component on Debian's bug tracker.

My suspicion is that somehow Debian's OpenSSL is too old, and there is nothing wrong with the ca-certificates. But it's hard to be certain, since there are now three different users in this issue with at least two different problems, which is confusing things. Moving to downstream bugtracker is certainly required now.

[nanonyme]

FWIW Debian 9 -> https://packages.debian.org/stretch/libcurl3 OpenSSL 1.0.2d.

[mcatanzaro]

OpenSSL 1.0.2d.

OK, wow, you are right. Debian 9 (Stretch) is expected to be broken, then. I got confused because I saw the default version of OpenSSL in Stretch was 1.1.0l, but that doesn't matter because curl is using the older version of OpenSSL.

You need at least OpenSSL 1.1.0 to use Flathub.

[nanonyme]

Well, I would even state you need Debian 10 or higher to use the Internet with anything curl-based given how common Let's Encrypt is.

[ShakalnyKot]

i on artix linux got same problem, but on
flatpak update

[SamsonovAnton]

In case someone runs an elderly version of openSUSE, the solution is:

sudo rm /usr/share/pki/trust/DST_Root_CA_X3.pem
sudo update-ca-certificates

Be sure not to replace the existing ISRG_Root_X1.pem with a freshly downloaded version.

[nanonyme]

@SamsonovAnton is that yet another bug in crypto library that fails to ignore expired CA?

[SamsonovAnton]

is that yet another bug in crypto library that fails to ignore expired CA?

This is the same bug as referred to by @iamihgam in the comment above. Funny enough, out of all tests

curl -I https://flathub.org
openssl s_client -verify_return_error -connect flathub.org:443
gnutls-cli flathub.org    # For fun.

the only one that was failing for me was the last one with gnutls-cli.

[bkauler]

I will throw in our experience with EasyOS. We have a fix, but it is very strange...

EasyOS has "Flapi", Flatpak Installer, and it works great. No problem with registering with flathub.org, but then I received a report from one person who got that "unacceptable TLS certificate" when try to register with flathub.

So far, three people have reported this failure. Which is very puzzling, as it works for the rest of us.

Then one of those three stumbled upon a fix: to set the time from the Internet.

Another of those three has tried that, and it also fixed the problem.

Yeah, but why? Federico has suggested why on page 6 where we have been discussing this on the Puppy Linux Forum:

https://forum.puppylinux.com/viewtopic.php?t=9038

Yet federico has no problem with TLS anywhere else, only with flathub.

Oh, and for the record, EasyOS has flatpak 1.12.8. The reason we are not using a later series is because Easy has 'appstream-glib' and the later series have dropped that and require more dependencies that Easy doesn't have.

[bkauler]

Also, as this person reported, updating the ca-certificates did not fix the problem, but syncing time from Internet did:

https://itsfoss.com/unacceptable-tls-certificate-error-linux/

Why this problem is occurring only with flathub.org, and why the sync time with Internet fixes it?

[hfiguiere]

and why the sync time with Internet fixes it.

If that's the fix then that's a problem in the client machines whose clock isn't on tim. Why it (seems to) happen only for flathub is left as an exercise to the reader. But it could just mean the clock drift cause a certificate to be invalid (future or past), and that could also happen with many other certs.