New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use dlopen, dlsym, etc. to minimize depnendencies #349

Closed
burdges opened this Issue Jul 16, 2015 · 5 comments

Comments

Projects
None yet
3 participants
@burdges

burdges commented Jul 16, 2015

It's worth mentioning that libextractor has a bunch of similar design consideraints to libcurl, both dynamically link against many external libraries, most of which are rarely used, both run on many platforms, etc. But libextractor.so and extract are linked directly against relatively few external dependencies.

Instead of directly linking to libraries it only rarely uses, libextractor builds a shared object plugin for the different filetypes it supports, and those shared objects link against the external libraries they need. See : https://www.gnu.org/software/libextractor/

I'd imagine doing this with libcurl would be an unpleasently large change, but if done well it'd buy libcurl users some additional flexibility. At minimum, it'd simplify packaging libcurl for distributions that usually do not want stuff like LDAP, SSH, etc. but cannot exclude them all outright. And importantly plugins would aid in tightening AppArmor and SELinux profiles for applicaitons that use libcurl. It might therefore be worth considering such a change for "some future version".

@kdudka

This comment has been minimized.

Collaborator

kdudka commented Jul 16, 2015

8 years ago libcurl used dlopen() to load LDAP libraries. It did not work well in certain scenarios, so the code was changed not to use dlopen(). See d0edb47.

@burdges

This comment has been minimized.

burdges commented Jul 17, 2015

Appears libcurl was "manually" dynamically linking the LDAP libraries using dlopen, dlsym, etc., which yes might create a mess for portability.

In libextractor, there are modules for each filetype that live under /usr/lib/x86_64-linux-gnu/libextractor/ on Debian and export a single symbol of the form EXTRACTOR_[filetype]_[blah]. All these modules are dynamically linked against any external libraries they need.

It's therefore ld.so that resolves all the real library dependencies, while libextractor only needs to know it's plugin directory. It's easier to verify that buffer overrun should not alter the symbol name passed to dlsym too.

As an aside, there is a libextractor_common.so library that 5 plugins link against too. Afaik dlopen makes symbols from the caller available, so presumably this contains stuff that repeated across plugins, but not overly common.

@bagder

This comment has been minimized.

Member

bagder commented Jul 19, 2015

I think this is a pretty interesting idea. I would say that a primary obstacle would be that it would need to be configurable since we have lots of users who want everything build statically and libcurl builds and runs on many systems that don't load stuff at run-time like normal *nix systems. This is not a priority feature for me personally so I'm not likely to work on it anytime soon. I'd be willing to help out if anyone else is willing to experiment.

@bagder bagder added the enhancement label Jul 19, 2015

@burdges

This comment has been minimized.

burdges commented Jul 20, 2015

I understand libltdl from libtool has a config option link everything statically while faking the use of dynamic plugins, again that's what libextractor does. There are still platform specific bits in doing so it seems.

@bagder

This comment has been minimized.

Member

bagder commented Jul 20, 2015

I'm sure there are several possible ways to implement it, I'm just saying it needs to be taken into account when/if someone decides to work on this.

@bagder bagder closed this in 8204844 Jul 25, 2015

jgsogo added a commit to jgsogo/curl that referenced this issue Oct 19, 2015

@lock lock bot locked as resolved and limited conversation to collaborators May 7, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.