Fix bundling of cURL in AppImages

[TheAssassin]

Continuation of the discussion in #1.

[darealshinji]

I will take a look at it and try to enable https while still avoiding as much dependencies as possible.

[TheAssassin]

Thank you so much, @darealshinji.
mbedTLS is quite standalone, so you could try to link against it for SSL support, although I doubt that'd work when compiling using the Ubuntu shipped headers. Therefore I'd like to ask you, how much work would it be to make a package containing the headers/development files? That'd be a great improvement, as currently it is very likely that incompatible headers would be used.

[TheAssassin]

CC @probonopd.

[darealshinji]

cURL has an option to build it against mbedTLS which works well. I'll update the PPA soon.

[TheAssassin]

That's what I meant. Could you test whether the system CA certificates are still found?

[probonopd]

To complicate matters: If we could make it to work in a reliable way, wouldn't using the system's SSL implementation be generally preferable? We should be able to expect OpenSSL to be part of any target system, shouldn't we? Also I'd suspect distributions not to package zsync2 if it doesn't work with the system's SSL libraries.

[darealshinji]

There's a way to make gnutls look for all the default certificate paths: https://gist.github.com/darealshinji/5b1b74a4962717f6fa99976481e89c76
I'd like to know if mbedtls could be patched in a similar way.

We should be able to expect OpenSSL to be part of any target system, shouldn't we?

There's libssl.so.1.0.0 on Ubuntu but libssl.so.10/libssl.so.1.0.0e on CentOS. The old issue with inconsistent filenames.

[probonopd]

Argh. Desktop Linux Platform Issues strike again.

[TheAssassin]

./configure --without-ssl --with-polarssl=/usr/local --with-ca-bundle=/etc/ssl/certs/ca-bundle.crt

Source: https://tls.mbed.org/kb/how-to/compile-curl-with-mbedtls

[darealshinji]

@TheAssassin /etc/ssl/certs/ca-bundle.crt is not present on Ubuntu, that's why I wanted to know if it was possible to patch it to look for different files and pick the first one it finds. By the way, assuming libcurl to be part of the OS base may not be an option either, as the symbol prefixes are patched on Ubuntu, but they might not be on other distros.

[TheAssassin]

That was just an example how it could've been configured. I don't know if it's possible to specify multiple search directories. I'll have to look that up.

[TheAssassin]

Started a dialog with cURL about the lack of such a feature.

[TheAssassin]

Okay, so it appears crypto libraries don't support multiple search locations. I suggested that libcurl should get such a feature upstream, where it tests different locations and hands a working one to the crypto library.

The problem is, how to determine which location works? A directory might be there but empty, etc. How did you solve this in your patched library, @darealshinji?

[darealshinji]

This is a patch I used on gnuTLS: https://gist.github.com/darealshinji/5b1b74a4962717f6fa99976481e89c76

But I wasn't able to build a recent version of curl against a custom gnuTLS library and I don't remember if that patch actually worked or not.

Update: forgot about this PPA -> https://launchpad.net/~djcj/+archive/ubuntu/gnutls-patched

[TheAssassin]

I've created a patch for curl that decides which SSL CA chain should be used based on a search path here: https://github.com/TheAssassin/Pext/blob/master/travis/curl-ssl-searchpaths.patch

It isn't too efficient yet, as the loop runs on every request, but it shouldn't be too hard to cache the value somewhere. The best part is that this patch works with any SSL/TLS library used with curl.

Upstream said they'd be willing to merge such a patch if this behavior was configurable during build time (that's implemented in the CMake configuration already) and during runtime (I'll have to investigate how to implement that properly).

Pext is the first project using the patch, and after a few days of intensive testing, I can tell it works well, cross distro etc. (despite some issues with the libgit2 build in use, but that's another story, the curl CLI client for instance works fine).

TO DO:

• send a PR to curl and have it merged upstream
  • the CMake configuration isn't very well maintained and produces binaries different from the autotools built ones, therefore someone with autotools experience will have to integrate the feature there as well
• provide an easy method to use those libraries e.g., on Travis CI (I could imagine an OBS repository with packages forked from Debian's, removing some dependencies by disabling exotic protocols like LDAP, and implementing the final patch)

Thanks @darealshinji for the initial work on a solution for this. I'll keep you up to date about this.

[TheAssassin]

I've been working on packaging the patch for Debian on OBS: https://build.opensuse.org/package/show/home:TheAssassin:AppImageLibraries/curl-httponly

Turned out the biggest difficulty is to make Travis install my packages as a replacement for *curl* packages. Whatever I test locally, it always upgrades fine.

[TheAssassin]

My custom builds of cURL finally produce usable results. I've successfully integrated them into the AppImageUpdate build process. This repository will follow as soon as possible.

[TheAssassin]

Fixed in aeaf936.

Verified by hand that libcurl is bundled, and that it's the binary I build on OBS.