Use dlopen, dlsym, etc. to minimize depnendencies

[burdges]

It's worth mentioning that libextractor has a bunch of similar design consideraints to libcurl, both dynamically link against many external libraries, most of which are rarely used, both run on many platforms, etc. But libextractor.so and extract are linked directly against relatively few external dependencies.

Instead of directly linking to libraries it only rarely uses, libextractor builds a shared object plugin for the different filetypes it supports, and those shared objects link against the external libraries they need. See : https://www.gnu.org/software/libextractor/

I'd imagine doing this with libcurl would be an unpleasently large change, but if done well it'd buy libcurl users some additional flexibility. At minimum, it'd simplify packaging libcurl for distributions that usually do not want stuff like LDAP, SSH, etc. but cannot exclude them all outright. And importantly plugins would aid in tightening AppArmor and SELinux profiles for applicaitons that use libcurl. It might therefore be worth considering such a change for "some future version".

[kdudka]

8 years ago libcurl used dlopen() to load LDAP libraries. It did not work well in certain scenarios, so the code was changed not to use dlopen(). See d0edb47.

[burdges]

Appears libcurl was "manually" dynamically linking the LDAP libraries using dlopen, dlsym, etc., which yes might create a mess for portability.

In libextractor, there are modules for each filetype that live under /usr/lib/x86_64-linux-gnu/libextractor/ on Debian and export a single symbol of the form EXTRACTOR_[filetype]_[blah]. All these modules are dynamically linked against any external libraries they need.

It's therefore ld.so that resolves all the real library dependencies, while libextractor only needs to know it's plugin directory. It's easier to verify that buffer overrun should not alter the symbol name passed to dlsym too.

As an aside, there is a libextractor_common.so library that 5 plugins link against too. Afaik dlopen makes symbols from the caller available, so presumably this contains stuff that repeated across plugins, but not overly common.

[bagder]

I think this is a pretty interesting idea. I would say that a primary obstacle would be that it would need to be configurable since we have lots of users who want everything build statically and libcurl builds and runs on many systems that don't load stuff at run-time like normal *nix systems. This is not a priority feature for me personally so I'm not likely to work on it anytime soon. I'd be willing to help out if anyone else is willing to experiment.

[burdges]

I understand libltdl from libtool has a config option link everything statically while faking the use of dynamic plugins, again that's what libextractor does. There are still platform specific bits in doing so it seems.

[bagder]

I'm sure there are several possible ways to implement it, I'm just saying it needs to be taken into account when/if someone decides to work on this.