Pre Qt 5.10 app cannot find OpenSSL 1.0 on Arch Linux

[trollixx]

Following up on zealdocs/zeal-packaging#34. This is not really a bug, but rather an explanation of the current situation.

Arch Linux has switched to Qt 5.10, which now depends on OpenSSL 1.1 [1], and now OpenSSL 1.0 is not installed by default. See [2] for packages still depending on version 1.0, but there are not many left.

So, unless a packaged app is built with Qt 5.10, or bundles OpenSSL 1.0 itself, it won't be able to establish secure connections on up-to-date Arch Linux system.

1. https://wiki.qt.io/New_Features_in_Qt_5.10
2. https://www.archlinux.org/packages/core/x86_64/openssl-1.0/

[TheAssassin]

I'm asking myself, what exactly is the problem here? Are you bundling an application on Arch Linux (which is a bad idea anyway), or is it that an Arch Linux machine can't run this zeal-packaging AppImage? Actually, it should be bundled (see https://github.com/AppImage/AppImages/blob/master/excludelist#L159-L165 for details), and shouldn't load any libraries from your system which are not in the exclude list.
Could you provide a link to your AppImage for us to inspect it and maybe try to reproduce the issue?

[probonopd]

The issue seems to be that applications compiled against openSSL 1.0 do not run on systems that come with openSSL anymore, at least not in all cases.

If that is true, then rolling release distribution users are the first ones to encounter issues, but all others will follow.

Why can't openSSL ensure that applications written for 1.0 "just work" on 1.1 as well?

Relevant details:
https://wiki.openssl.org/index.php/OpenSSL_1.1.0_Changes#No_longer_works

So it looks like they are fully aware that their new version breaks a lot of third-party stuff including Qt, and still they couldn't seem to care less?!

This kind of stuff is exactly what makes "Desktop Linux" unpleasant to use. Certainly something we might want to mention on https://github.com/AppImage/AppImageKit/wiki/Desktop-Linux-Platform-Issues

[trollixx]

@TheAssassin See https://github.com/probonopd/zeal/releases

@probonopd Right, OpenSSL 1.0 and 1.1 are binary incompatible. And Arch Linux is almost OpenSSL 1.0 free.

[trollixx]

Are you bundling an application on Arch Linux (which is a bad idea anyway)

While this ticket is result of running an AppImage packaged on Ubuntu on Arch Linux, why not? Seems like the easiest way to get up-to-date Qt and other dependencies.

[probonopd]

@probonopd Right, OpenSSL 1.0 and 1.1 are binary incompatible.

Aaaaaaaaarrrrgggggghhhhhh!

[probonopd]

OT:

Are you bundling an application on Arch Linux (which is a bad idea anyway)
why not? Seems like the easiest way to get up-to-date Qt and other dependencies.

... that depend on other "up-to-date" dependencies in turn, which users of LTS or Enterprise distributions will not have (e.g., glibc). For details, please see https://github.com/AppImage/AppImageKit/wiki/Creating-AppImages#creating-appimages-that-are-compatible-with-many-systems.

[probonopd]

Our expectation is that if an application is built against version 1.0 of a library that is on every target system, then the same application will continue to work when the target system is updated to 1.0.1 or 1.1 of the same library.

If that basic assumption does not hold anymore, then we are effectively forced to bundle the library with the Application in a private version, which we want to avoid.

Especially for a library like OpenSSL that is a) on every system b) important for security, it would be desirable for applications not to bundle private copies but being able to rely on whatever the system provides. For this to work, though, OpenSSL must not break applications compiled against earlier versions of itself.

This discussion should probably be taken on some OpenSSL developer mailing list.

Please note that this has nothing to do with AppImage specifically but with compiled applications on the Linux platform in general. Take Firefox, for example. Mozilla distributes binaries that are supposed to run on most systems. If they cannot trust that whatever OpenSSL is in the operating system will continue to work, they are essentially forced to bundle a private copy, which means that a) there is more duplication, eating more system resources b) Firefox will not benefit from security updates in OpenSSL that the operating system might do. Same for Chrome and any other application that a) uses OpenSSL and b) is distributed outside of the Linux distribution.

[trollixx]

It appears [1] that with Qt 5.10 OpenSSL backend has to be chosen at the compile time. Once Qt is built against one, it will not use another.

1. http://lists.qt-project.org/pipermail/interest/2017-October/028512.html

[probonopd]

The key issue is that OpenSSL 1.0 and 1.1 seem to be binary incompatible. In other words, OpenSSL breaks binary compatibility in a minor release. Which is bad. According to https://semver.org/,

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.

So OpenSSL should have added "functionality in a backwards-compatible manner" going from 1.0 to 1.1. Instead they broke backward binary compatibility. People should complain about this big time at OpenSSL.

On a related note, it seems like they are still maintaining the 1.0 series, so no immediate need to switch to the binary-breaking 1.1.

[trollixx]

OpenSSL predates, and clearly not using the Semantic Versioning. A lot of projects have their own versioning systems.

[probonopd]

I've opened an issue for it:
openssl/openssl#5042

[probonopd]

OpenSSL predates, and clearly not using the Semantic Versioning.

They are just now going from 1.0 to 1.1 - so this is the time where they need to think about this, imo. Before 1.1 find its way into LTS distributions. At which point the mess will not be easily reversible anymore.

[trollixx]

Let's see, but I believe that will have no effect. They documented and announced breaking BC. It took Qt over a year to add support for OpenSSL 1.1.0.

[probonopd]

It seems to me that since OpenSSL 1.1 is in effect like a "major version" (despite it not following semver), distributions need to install both OpenSSL 1.0 and 1.1 in parallel.

The OpenSSL project continuing to update both versions seems to support this notion.

What a mess.

[probonopd]

They documented and announced breaking BC.

The issue with this kind of stuff is that people only realize once the breakage trickles down into LTS and Enterprise distributions, at which point it is way too late to complain.

[trollixx]

That's what they do. Arch has both packaged, but nowadays very few packages depend on 1.0, and often it is not installed.

[probonopd]

That's what they do. Arch has both packaged, but nowadays very few packages depend on 1.0, and often it is not installed.

That's the Arch user's problem, then. (It's not nice, but we have to thank OpenSSL for that.) Let's see what the Enterprise distributions will do in their default installations.

[trollixx]

They all are doing the same, just with some delay. For example, see information for Debian [1].
1.https://wiki.debian.org/OpenSSL-1.1

[trollixx]

I'd say since Qt uses dlopen, it's not a big deal. AppImage should just check if system provides a newer compatible OpenSSL, and use that, otherwise fallback to bundled version.

[trollixx]

Actually, it has nothing to do with how Qt loads OpenSSL.

[TheAssassin]

What I still don't fully understand is, why does it even bother to load the system libssl, since it should be bundled. If it's really loaded by Qt via dlopen(), then linuxdeployqt should add it as a dependency, and bundle it in any case.

[probonopd]

It is not bundled in https://github.com/probonopd/zeal/releases/download/continuous/Zeal-3361d7c-x86_64.AppImage

Why should it?

Searching for libssl shows only one match, libQt5Network.so.5:

me@host:~$ grep -r libssl squashfs-root/
Binary file squashfs-root/usr/lib/libQt5Network.so.5 matches

But libssl is not a dependency that ldd and hence, linuxdeployqt, could detect (presumably because it is using dlopen():

me@host:~$ ldd squashfs-root/usr/lib/libQt5Network.so.5  | grep libssl

So linuxdeployqt (currently) has no chance of knowing that libssl should be bundled, nor which version of it.

Even grepping it for strings doesn't yield it:

me@host:~$ strings squashfs-root/usr/lib/libQt5Network.so.5 | grep '\.so'
libc.so.6
libQt5Network.so.5
libstdc++.so.6
libpthread.so.0
libQt5Core.so.5
libz.so.1
libproxy.so.1
libQt5Network.so.5.5.1

Only this does:

me@host:~$ strings squashfs-root/usr/lib/libQt5Network.so.5  | grep libssl
libssl.*

[trollixx]

Why should it?

To let the application work on systems that don't have OpenSSL 1.0 installed?

[probonopd]

We currently assume that all systems have OpenSSL 1.0 installed, although we might have to reconsider that notion given what we've had to learn today (if it turns out that distributions won't ship it anymore in their default installations)...

With "Why should it?" I meant, what facts inside the AppDir could trigger linuxdeployqt to bundle OpenSSL? And as my analysis shows, even if we decided to bundle OpenSSL as needed, there would be no straightforward way for linuxdeployqt to even know that it is a dependency...

[probonopd]

Now that we have found out that Qt softly "depends" on libraries abbreviated by .*, we can systematically check whether more libraries are loaded this way:

grep -r '\.\*' squashfs-root/ | cut -d " " -f 3 > filenames
for FILENAME in $(cat filenames) ; do strings $FILENAME | grep 'lib.*\.\*' ; done

And indeed, we find:

libssl.*
libcrypto.*

So linuxdeployqt would possibly have to do a similar search over the whole AppDir content, and then somehow conclude how to translate the found strings like libssl.* and libcrypto.* into file paths that could be deployed.

[trollixx]

Right, that's what I mentioned in the beginning. OpenSSL is not a link dependency. Even on Windows we have to copy OpenSSL DLLs after running windeployqt.

[probonopd]

What I wonder is:

• What is the reason for Qt using dlopen() in the first place? Smells like it has to do with OpenSSL not following semver.org and/or bumping SONAMEs
• Why does OpenSSL 1.1 not get loaded by pre-5.10 libQt5Network.so.5, it also matches libssl.*?
• What would happen if we would bundle Qt 5.10, a) on systems with OpenSSL 1.0, b) on systems with OpenSSL 1.1, c) on systems with both?

[probonopd]

Even on Windows we have to add OpenSSL DLLs after running windeployqt.

Are you aware of other such "specialties"? How can we know, apart from trial and error?

[d1vanov]

Side question: would it be possible for those impatient folks who would like to have some hacky solution to this OpenSSL problem right now to just manually copy the relevant version of libssl.so.* to the AppDir after calling linuxdeployqt -bundle-non-qt-libs and before calling linuxdeployqt -appimage? Would the bundled application prefer bundled libssl over the system one in that case?

[TheAssassin]

There is no need to bundle the certificates, but bundling the SSL libs might be possible. We do that with gnutls for instance...

[probonopd]

Maybe we are just overly concerned and should just bundle our own OpenSSL. In fact, I checked what Ultimaker Cura (a PyQt application) does for macOS, they ship a private copy of libssl.1.0.0.dylib inside their .app bundle.

[TheLastProject]

I've had this issue myself on Fedora (27, but still with 28 and 29) and reported by Ubuntu and Mint users too: Pext/Pext#135.

Does https://bugreports.qt.io/browse/QTBUG-68156 help in any way?

[TheAssassin]

I guess Qt 5.11.3 will contain a fix, @TheLastProject.

[probonopd]

I guess Qt 5.11.3 will contain a fix

Do you have some evidence to base this hope on?

[TheAssassin]

That's what they say in that bug report...

[claell]

@probonopd I am facing this issue as a user of Sync-My-L2P now on Ubuntu 19.04 (see referenced issue above). Is there anything that can be done to solve this (apart from manually installing OpenSSL 1.0)?

In general, what is the normal procedure, when a major release of a shared lib gets published which breaks compatibility with previous versions? The only difference here is, that the major release did not follow common naming conventions, still it was thought to be a major release and the name is not really relevant here.

[probonopd]

In general, what is the normal procedure, when a major release of a shared lib gets published which breaks compatibility with previous versions?

Then either distributions ship both versions for some time (which they do e.g., for Gtk 2 vs. 3 but don't do in the case of OpenSSL; why?), or a private copy needs to be bundled inside the AppImage.

[probonopd]

• https://www.mail-archive.com/tech@openbsd.org/msg36437.html
• https://news.ycombinator.com/item?id=13284648
• Please follow semantic versioning openssl/openssl#5042

[claell]

Then either distributions ship both versions for some time (which they do e.g., for Gtk 2 vs. 3 but don't do in the case of OpenSSL; why?)

Alright, apparently this did not happen, unfortunately.

or a private copy needs to be bundled inside the AppImage.

So this is something that can be done currently? Because it was stated otherwise in the project's issue referring to this discussion: RobertKrajewski/Sync-my-L2P#140 (comment)

https://www.mail-archive.com/tech@openbsd.org/msg36437.html
https://news.ycombinator.com/item?id=13284648

Mainly complaining that the change could have been done better and asking whether it was worth it. Apparently OpenSSL did not care that much about breaking compatibility which is sad.

openssl/openssl#5042

I already read that issue. However I don't think that would have changed much, or do you think that is the main reason why distributions didn't ship both versions?

[probonopd]

do you think that is the main reason why distributions didn't ship both versions?

Somehow I think that it may have contributed to the mess, yes. Anyhow, I think that for now we need to ship a private copy of OpenSSL.

Now, the private copy of OpenSSL must either ship its own set of certificates (which would be very cumbersome to maintain), or load the certificates from the system. Unfortunately, the different distributions happen to put them in apparently "random" locations (for no apparent good reason other than "distribution policy"):

"/etc/ssl/certs/ca-certificates.crt",     // Debian/Ubuntu/Gentoo etc.
"/etc/pki/tls/certs/ca-bundle.crt",       // Fedora/RHEL
"/etc/ssl/ca-bundle.pem",                 // OpenSUSE
"/etc/pki/tls/cacert.pem",                // OpenELEC
"/etc/ssl/certs",                         // SLES10/SLES11, https://golang.org/issue/12139
"/usr/share/ca-certs/.prebuilt-store/"    // Clear Linux OS; https://github.com/knapsu/plex-media-player-appimage/issues/17#issuecomment-437710032
"/system/etc/security/cacerts"            // Android

Hence, the private copy of OpenSSL needs to be patched so that it loads certificates from all those known locations.

References:

• https://gitlab.com/probono/platformissues/blob/master/README.md#certificates
• https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/
• Patch by @darealshinji https://github.com/darealshinji/vlc-AppImage/issues/1#issuecomment-321041496
• Detect when included into AppImage; root CA and engines path openssl/openssl#7481

[claell]

Anyhow, I think that for now we need to ship a private copy of OpenSSL.

Alright. So just to get this right, this is up to linuxdeployqt, not the user of it, correct? Should there be a new issue for this then?

[probonopd]

I guess so. Ideally some Qt developers in the know would chime in on what to do. How is this handled by macdeployqt and windowsdeployqt?

[claell]

Hence, the private copy of OpenSSL needs to be patched so that it loads certificates from all those known locations.

@probonopd That does not seem to be true. The issue @nurupo referenced above bundled an unpatched version, which worked out of the box, afaik. So having linuxdeployqt ship a private version might not be too complicated. I will create an issue for that now.

[probonopd]

Or just use Qt 5.10 for applications that need OpenSSL?

[nurupo]

@claell btw, in that issue Qt 5.7.1 was being bundled into AppImage, the Qt from Debian Stretch packages.

[claell]

Or just use Qt 5.10 for applications that need OpenSSL?

I told that to the developer, until now I did not get a reply to that. Unfortunately there might be some things preventing a change right now.

@nurupo Thanks for the additional info.

Here is one specific comment from that issue about that topic. Later comments also show it working on other distributions: qTox/qTox#5747 (comment)

[probonopd]

Closing here since Arch is not one of the distributions we are targeting, OpenSSL is generally a [mess] (openssl/openssl#5042 least until 3.0), and updating to Qt 5.10 seems to resolve the issue.

Thanks for your investigations.