Accessing Host Trusted Certificates

[LorbusChris]

Linux distribution and version

Fedora Silverblue 29

Flatpak version

v1.2.3

Description of the problem

Flatpaks apps do not pick up PEM certificates placed in /etc/pki/ca-trust/source/anchors/ on the host.

Steps to reproduce

• Place custom trusted cert in /etc/pki/ca-trust/source/anchors/
• Install Riot.im or Firefox Nightly flatpak
• In Riot.im: Try to log in to a Matrix server that uses that cert -> not possible
• In Firefox Nightly: Navigate to an URL that uses that cert -> 'Warning: Potential Security Risk Ahead' page is shown

[LorbusChris]

Confirmed not to work on Fedora 29 standard as well.

[TingPing]

Riot probably uses OpenSSL which doesn't respect PKI anyway AFIAK. Firefox should though.

[LorbusChris]

Right, that could be, as I only reproduced it with Riot this time. Firefox Nightlies aren't up to date at the moment, so I didn't test that.

Any idea how this could be made compatible with OpenSSL?

[TingPing]

Hmm, it looks like Fedora builds openssl with --openssldir=/etc/pki/tls while freedesktop-sdk has the default /etc/ssl. So that is probably relevant.

EDIT: Grepping around Flatpak doesn't seem to expose the pki dirs anyway. I thought there was some work on doing so though...

[matthiasclasen]

@amigadave is this something you could look into for our fedora flatpak builds ?

[mcatanzaro]

I think Firefox should be fixed nowadays, since https://gitlab.com/freedesktop-sdk/freedesktop-sdk/commit/a11bd7c94e163b34e4ac209361c47f914997932f, which is in 18.08. Can you confirm?

Then Riot.im is an Electron app, right? So you would need to add p11-kit support to Chromium (CrNet) if you expect it to be able to access host certificates. I don't think it uses OpenSSL.

Accessing host certificates requires use of the p11-kit server running on the host OS, so if your TLS library doesn't support using p11-kit as its trust store, this is simply not expected to work. So you should report a bug to Chrome (good luck with that) for Riot.im, not here.

It's true that OpenSSL is not expected to work either, though I think neither of your example apps use OpenSSL. Eventually, OpenSSL will be improved to support p11-kit. Once that happens, we will need to change freedesktop-sdk to build OpenSSL with p11-kit configured as its default trust, like we already do for GnuTLS. freedesktop-sdk also has NSS patched to use p11-kit, like Fedora does, so NSS should work too. (I think OpenSSL is the last remaining significant holdout that doesn't support using p11-kit as its trust store; eventually it will get fixed.)

So, expected to work:

• Anything using GnuTLS, including all GNOME stuff and anything else that uses GLib sockets (glib-networking)
• Anything using NSS, including Firefox
• freedesktop-sdk only: libcurl (configured to use GnuTLS), and anything using libcurl

Not expected to work and not our problem:

• Chrome/Electron stuff
• Anything using OpenSSL
• Fedora runtime only: libcurl (configured to use OpenSSL), and anything using libcurl

TL;DR: this bug should be closed, unless Firefox is still broken. (It should not be broken anymore.)

[mcatanzaro]

• Chrome/Electron stuff

Note: this will be using CrNet, and beneath that, BoringSSL. It's probably bundled in the Chromium source code under third_party/.

[mcatanzaro]

OK one last tip. If you want a workaround, then instead of Riot.im you can use Revolt, which just uses WebKitGTK to display the Riot website instead of Electron. That means you get GLib sockets -> glib-networking -> GnuTLS -> p11-kit which should have access to the host certificates.

[AdrianVovk]

There seem to be mechanisms to generate CA bundles for openssl & java & everything else in p11-kit: trust extract --format=openssl-bundle --filter=ca-anchors --overwrite /tmp/openssl-bundle.pem

As a distributor, the fd.o sdk can override /usr/libexec/p11-kit/trust-extact-compat, and then just run trust extract-compat. p11-kit runs this automatically whenever trust anchor is run, but I'm not sure how it handles the daemon/client relationship used by Flatpak.

[mcatanzaro]

Not sure I understand. How could that help?

[AdrianVovk]

@mcatanzaro All the SDKs have to do is run trust extract [...] at some point and then OpenSSL will use the system's CA certs, right? Or am I missing something. Just extract it into the correct place for OpenSSL inside of the runtime will pick it up

Edit: Or Flatpak can extract these on the host side and place them into, i.e. /run/flatpak/openssl-bundle.pem, and then bind-mount this into the runtime env

[mcatanzaro]

Yes, that would work, but it would conflict with the certificates provided by the runtime itself. The runtime would have to stop shipping its own certificates. I'm not sure whether that would be desirable or not. There would be... implications. :)

Proper fix is to finish and land openssl/openssl#8200 to bring openssl out of certificate dark ages and into modern times.

[AdrianVovk]

@mcatanzaro That's fair

[dustymabe]

I think I hit this today. Trying to use firefox in a flatpak on f31 silverblue and I have system wide certs added to my host (to access websites hosted by my employer). These certs don't make it into the flatpak AFAICT.

[mcatanzaro]

No, as stated above, Firefox uses NSS which uses p11-kit for the trust store therefore should work fine. After making sure you have p11-kit-server running on your host, file a new bug against the Firefox flatpak packaging if it is no longer working.

[mcatanzaro]

BTW this bug should be closed since there is nothing to be changed in flatpak here. Everything should be using host certificates nowadays except (a) Chrome/BoringSSL ecosystem, which requires a Chrome bug report, (b) stuff using OpenSSL, openssl/openssl#8200, and (c) minor/unconventional TLS libraries we shouldn't worry about.

[dustymabe]

After making sure you have p11-kit-server running on your host, file a new bug against the Firefox flatpak packaging if it is no longer working.

https://bugzilla.redhat.com/show_bug.cgi?id=1766340

[TingPing]

@mcatanzaro I don't think flatpak does anything to expose any of this actually.

[mcatanzaro]

I don't think flatpak does anything to expose any of this actually.

But p11-kit does, in the flatpak container it connects to p11-kit-server on the host. You should be the expert on this by now, not me. :)

[aviwad]

Problem remains in Fedora silverblue 31

[dustymabe]

I still have to follow the directions in https://bugzilla.redhat.com/show_bug.cgi?id=1766340#c3 every time I re-setup my system. Not sure what needs to be done to fix this, but it would be nice.

[mcatanzaro]

This bug really needs to be closed.

BTW this bug should be closed since there is nothing to be changed in flatpak here.

Please report bugs to the affected applications that don't use p11-kit. Don't comment here. Nothing is wrong with flatpak.

The issue with Fedora's Firefox build is reported against Fedora because it is a bug in Fedora's Firefox flatpak packaging. If you have the same issue with Flathub's Firefox, please report it to Flathub. Don't comment here. Flatpak is setting everything up properly, but it can't force applications to use the host certificates and it's not flatpak's fault if they don't.

[Erick555]

If you have the same issue with Flathub's Firefox, please report it to Flathub

No such thing yet 😄

[mcatanzaro]

So everybody's just using Fedora's build? We already know that's a Fedora bug. (Well, ideally it would be fixed upstream instead of requiring every distro to patch Firefox to use p11-kit, but in practice this is how all distros package Firefox and only Fedora's flatpak build gets it wrong.)

[TingPing]

I don't believe this bug should be closed. Even if p11-kit works many applications do not use p11-kit for certificates and it is entirely possible for Flatpak to expose host certificates. Maybe we don't want to do that but it hasn't really been explored here.

[dyskette]

Is this the issue I hit?

I'm using Steam Flatpak on Fedora 32 and trying to play Dirt Rally 2 game, however it needs a certificate from codemasters (a .pem file). I added it using the fedora documentation, but it doesn't seem to have any effect within the game, as it still doesn't let me play online.

Certificate: https://www.upload.ee/files/9626680/codemasters.pem.html

[mcatanzaro]

Iirc GnuTLS has supported certs from host through p11kit from like 1.6 (not taking bugs into account). nss grew support in 19.08 by @valentindavid . I don't know which curl uses here.

curl uses GnuTLS and GnuTLS uses p11-kit, certainly, but my concern is that curl is compiled with --with-ca-bundle= which tells libcurl to use a certificate bundle on disk rather than GnuTLS's default trust store (the p11-kit trust). I'm not sure if libcurl can be told to do the right thing; I haven't investigated. There should be some option to use the default trust store rather than a certificate bundle. I would expect it to be the default behavior, but the warning comment in the bst element indicates otherwise. Moreover, I see curl in Fedora is compiled to use the certificate bundle, so that causes me to suspect it's not going to work.

It's possible for curl to use p11-kit for PKCS#11 without using it for its trust store.

Unsure if OpenSSL route support p11kit, probably not.

It ought to work... the history here is that OpenSSL upstream refused to play ball, so p11-kit developed an OpenSSL engine.

[nanonyme]

Right, we don't have libp11 in the runtime though. I created issue for openssl p11kit support, probably won't materialize before 20.08 is out.

[nanonyme]

@valentindavid said there may be legal implications to TLS backend for curl so we probably need to keep the OpenSSL one as well.

[Yuri6037]

I've hit this as well: I require features that do not exist in p11-kit so I have to rely on loading the bundle. However the bundle is incorrect in the flatpak container as I can see many more CA certificates on my host Ubuntu system than in the flatpak container.

This really needs to be fixed. Also not even considering .NET Core apps, Java apps and possibly many other frameworks...

The flatpak container should just use the certificate bundle from the host system so that all applications are consistent on a given system...

[nanonyme]

It's not really as simple as "just using certificate bundle". Certificate ecosystem has been a true mess for a long time and without glue like p11-kit bringing things together it's not really possible to have something that works same way in all distros.

[zicklag]

Yeah, it is a surprisingly difficult problem because tons programs have been using their own way of getting the certificate bundle since forever. Python programs get it from a certify python package, some C programs like curl on different distros might be compiled to literally look in a specific directory. Everything has its own idea of where to get the certs and p11-kit is one of the few developing standards to start pushing everybody to use a standardized way of getting the cert, if I understand correctly.

[nanonyme]

I was trying to get https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/merge_requests/3798 done which might help some (probably not really backportable material since it's quite invasive) but it hit some static analysis tools with the project and I ran out of time figuring out how to fix them.

[nanonyme]

FWIW freedesktop-sdk 21.08 is going to carry libp11 to make OpenSSL hopefully play ball with p11kit. First beta is going to be done real soon.

[j1mc]

It looks like freedesktop-sdk release versions 21.08 and 21.08.1 were tagged within the last week.

Can anyone verify if the issue is resolved or if it still persists?

[nanonyme]

Okay, so the thing is, OpenSSL clearly can access host certs as can GnuTLS. But curl appears to use both in a way that results in embedded CA certs being used. I don't understand why.

[nanonyme]

You can observe this by that currently with org.freedesktop.Platform//21.08

curl -L https://mesa.freedesktop.org/archive/glu/glu-9.0.2.tar.xz fails

CURL_SSL_BACKEND=gnutls curl -L https://mesa.freedesktop.org/archive/glu/glu-9.0.2.tar.xz fails

gnutls-cli --ca-verification mesa.freedesktop.org succeeds

openssl s_client -connect mesa.freedesktop.org:443 succeeds

So remaining problem is clearly making curl not do something stupid. Crypto libraries work fine, curl does not.

[nanonyme]

@mcatanzaro looks like "causes me to worry that libcurl is not designed to do the right thing" was entirely correct.

[mcatanzaro]

looks like "causes me to worry that libcurl is not designed to do the right thing" was entirely correct.

To be clear: we must get rid of --with-ca-bundle from curl.bst if we want curl to have any chance at using host certificates (and we almost surely do want that). That does exactly what it says: it forces curl to use certs from the bundle installed in the runtime, rather than the host.

That said, I see this warning in the buildstream element:

# Curl must be configured --with-ca-bundle in order to find the
# correct certificate bundle at runtime, which is later generated
# by update-ca-certificates. Other applications such as git, rely
# on curl knowing the default location of the certs.

which indicates that libcurl defaults to using no certs at all? That doesn't sound right. If that's still true, it would be a curl bug IMO: it should default to using the system trust, not no trust at all.

How this is supposed to work is flatpak drops a configuration fragment under /etc which makes p11kit under flatpak use host trust store. curl is already linked against p11kit. Curl is a bit weird since it builds against both nss and GnuTLS and builds a dependency on p11kit on both directions.
Looks like for possibly historic reasons curl also links against OpenSSL and has OpenSSL backend which will not result in a working p11kit host cert usage if that ends up being used.

For avoidance of doubt: curl lets you choose the TLS backend at build time. Normally you would pick just one. Don't expect any consistency between distros here. Debian builds all three and lets applications pick which one they want; only one of them gets used by default, but I don't remember which. Fedora used to use NSS (I think) but nowadays uses OpenSSL. I see freedesktop-sdk uses both NSS and GnuTLS, and I'm not sure why. My suggestion would be to drop the NSS support and see if anything breaks: my guess is probably not. Sorry, after discussing with @nanonyme I see freedesktop-sdk actually builds all three and lets applications pick, just like Debian, but OpenSSL is default.

FWIW freedesktop-sdk 21.08 is going to carry libp11 to make OpenSSL hopefully play ball with p11kit. First beta is going to be done real soon.

If you've got OpenSSL hooked up to p11-kit, then that's cool. p11-kit is the primary reason to prefer GnuTLS vs. OpenSSL.

Of these three, NSS would be the least-preferred backend.

P.S. It seems I've requested this bug be closed multiple times in prior comments, since its scope is too broad and it's obviously not a flatpak issue. The flatpak issue tracker would be more useful if misplaced issues were closed more aggressively.

[nanonyme]

Well, it defaults to trying to figure out where platform-specific CA bundle is. I'm currently trying to build curl with --without-ca-bundle to disable its bundle logic completely. Note that currently OpenSSL, GnuTLS and NSS should be using p11kit correctly. So backend shouldn't matter as long as curl doesn't try to tell crypto library to use the runtime CA bundle.

[nanonyme]

I would also note that currently curl does the wrong thing even with GnuTLS backend.

[mcatanzaro]

So backend shouldn't matter as long as curl doesn't try to tell crypto library to use the runtime CA bundle.

I believe so, yes. Certainly that is correct for GnuTLS.

[shmu26]

Just wanted to mention that I am having this certificate issue with the Zoom flatpak app.
It is not honoring the certificate store of my system.
Ubuntu 22.04.

[nanonyme]

For the record, --with-ca-fallback in combination with -without-ca-bundle was the right solution. Not all apps will ever work correctly if they have eg bundled curl.

[shmu26]

For the record, --with-ca-fallback in combination with -without-ca-bundle was the right solution. Not all apps will ever work correctly if they have eg bundled curl.

As a home user without a technical background, is there anything I can do about this, on my side? I don't understand the terms you mentioned.

[nanonyme]

I mean, fix has been done. Runtime and flatpak work correctly. Remaining bugs are on app side and this should be closed.

[smcv]

As a home user without a technical background, is there anything I can do about this, on my side?

Not really, other than reporting it as a bug to the apps that are affected if nobody else has already done so.

(The maintainers of the Flathub package for Zoom probably cannot fix it, because Zoom is proprietary software, but at least having a bug report would record that limitation.)

It seems I've requested this bug be closed multiple times in prior comments, since its scope is too broad and it's obviously not a flatpak issue

Closing this as not Flatpak's bug.

What is now meant to happen is that apps (or the SSL/TLS libraries used by apps) should use p11-kit as their trust store. If they do that, then they will automatically pick up a socket forwarded by Flatpak from the host system, and trust whatever certificates the host trusts. If they do not do that, then that's a bug (or at least a limitation) in that app or the SSL/TLS library that it uses, and not a bug in Flatpak itself.