Try non-generic shared lib names

[renchap]

Hi, and thank you (again) for vips, and ruby-vips!

I am using ruby-vips on Debian, using the setup I described here: https://renchap.com/blog/post/building_recent_libvips_on_debian/

As said in my blog post, we currently need to install libvips-dev on Debian for ruby-vips to work. This is due to Debian packages not including symlinks for libraries in the non-dev packages.

For example, without glib-dev, I dont have a /usr/lib/x86_64-linux-gnu/libglib-2.0.so -> /lib/x86_64-linux-gnu/libglib-2.0.so.0 symlink, thus ffi_lib glib_libname fails. This is the same for gobject and vips.

Installinglibvips-dev is adding ~400 MB on a ruby:2.5-slim docker image compared to only installing libvips.

Would it be possible to try to load the full libs name, including the version number, like for Windows? If using libvips.so.42, libgobject-2.0.so.0 and libglib-2.0.so.0 it should work:

# ldconfig -p | egrep "vips|gobject|glib"
        libvipsCC.so.42 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libvipsCC.so.42
        libvips.so.42 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libvips.so.42
        libvips-cpp.so.42 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libvips-cpp.so.42
        libpoppler-glib.so.8 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpoppler-glib.so.8
        libgobject-2.0.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
        libglib-2.0.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libglib-2.0.so.0

[jcupitt]

Hi @renchap, I like your blog post!

I agree, the library loading situation is not very good at the moment. I'd like to just load libvips.so, if possible, since most of ruby-vips is done with introspection. It ought to work with almost any libvips post-8.2. The Debian symlink policy is unfortunate -- it doesn't seem to take account of this case.

I suppose we could try loading libvips.so, then fall back to libvips.so.42 if that fails. Would that solve your problem? And the same for the other libraries, of course.

[jcupitt]

Off-topic, but I think I would not build libvips with all dependencies. Some of them (eg. cfitsio, or matio) are very specialized and will not be useful on the web. They are also (as far as I know) mostly untested and I expect they are exploitable.

I would just include the file format libraries you know you will need, so png, jpg and perhaps gif. I'd also turn imagemagick off, I think. svg and pdf might be useful. Maybe tif too?

[jcupitt]

... and there are a few loaders built into libvips which will not be very useful on the web: csv, ppm, pbm, pgm. They can be disabled with configure flags.

[renchap]

I suppose we could try loading libvips.so, then fall back to libvips.so.42 if that fails. Would that solve your problem? And the same for the other libraries, of course.

Yes this would work, or the other way around, as you prefer. How often the version number are incremented? Can this become painful to manage in the future? I thought about providing it via configuration (env variable?), but it requires manual configuration for every users, so probably not ideal.

I agree with the dependencies, I should trim my packages. I started from the (outdated) official packages that include all these dependencies, but I might revise this and do my own with only the deps I need. But for now, those dont bring a significant overhead compared to all those -dev packages :)

[jcupitt]

You're right, in practice it doesn't change very often. It's just another thing that might need updating at some point in the future, will probably be forgotten about, and then cause mysterious bugs.

We could glob for libvips.so.* and pick the highest, but perhaps that would be even more dangerous!

If you are running libvips in some kind of sandbox then including weakly tested format loaders is probably OK. If you are using ruby-vips in your main thread, then I'd turn everything off I could. We've been though many rounds of AFL and valgrind with the popular format libraries, but we've not tested all the loaders :(

[renchap]

You can not really glob, as you do not know where the libraries will be installed. We could try to parse ldconfig's cache, but this becomes very implementation-specific.
If you want to be flexible, I think trying out the current known version and fallbacking on the versionless name (or the other way around) it the best. We could also introduce an (optional) env variable if/when you want to try with new libraries or unusual installs.

Regarding the loaders, this may be best to discuss it in a new issue (in the vips repo?). Ideally I would love to have them as separate binaries, so distros can install a "basic" libvips package, and then have additionnal packages (-extra?) with more exotic loaders. This would remove the need for custom-compilation if you want an optimised/well tested package. On debian, nginx is done this way, with different packages to add more modules.
I am not sure how it could work with vips, but it definitely can improve packaging and avoid less-tested modules to impact users that dont need them if they are not installed (or not loaded).

[Nakilon]

About splitting into popular and not-so-popular parts: I would say the jpg, png, bmp, gif, (webp?) are the important ones.

[jcupitt]

Good list @Nakilon, though BMP uses libmagick, so that pulls in many, many other formats, unfortunately.

libvips has a simple module system already, so it'd be quite easy to split off the less-used loaders to a separate binary. I think it wouldn't work with DLLs though, since they don't support back-linking, so there would be an extra complication there.

[jcupitt]

I had another look at this in a docker container:

https://github.com/jcupitt/docker-builds/blob/master/ruby-vips-ubuntu18.04/Dockerfile

It looks like changing ruby-vips ffi_lib from glib-2.0 to load glib-2.0.so.0 might fix it (and making the matching change for other libraries).

Does this sound reasonable? Would this break other linuxes?

[jcupitt]

I've made a branch that has this change. @ioquatix I think this might affect you, what do you think?

[renchap]

I just tested your branch on a ruby:2.5-slim docker image using my own vips debian package (but no -dev) and it seems to work 🎉 The resulting docker image is 400MB lighter.

Unfortunately, it no longers works on my mac with this branch:

» ruby example2.rb
Traceback (most recent call last):
	6: from example1.rb:4:in `<main>'
	5: from example1.rb:4:in `require'
	4: from /Users/renchap/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bundler/gems/ruby-vips-a4f606d615b9/lib/vips.rb:13:in `<top (required)>'
	3: from /Users/renchap/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/bundler/gems/ruby-vips-a4f606d615b9/lib/vips.rb:28:in `<module:GLib>'
	2: from /Users/renchap/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ffi-1.9.25/lib/ffi/library.rb:99:in `ffi_lib'
	1: from /Users/renchap/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ffi-1.9.25/lib/ffi/library.rb:99:in `map'
/Users/renchap/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/ffi-1.9.25/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library 'glib-2.0.so.0': dlopen(glib-2.0.so.0, 5): image not found. (LoadError)
Could not open library 'libglib-2.0.so.0.dylib': dlopen(libglib-2.0.so.0.dylib, 5): image not found

As a side note, we discussed above about a way to provide a modular libvips package, allowing distros to provide less common (and tested) loaders in a separate package (libvips-extra) and not require people to recompile their own libvips if they want to avoid those loaders. Do you want me to open an issue in the main VIPS repo for this? As an example, installing libvips on Debian (without -dev packages) currently pulls Firefox and 700MB of disk usage.

[ioquatix]

My fork https://github.com/ioquatix/vips builds libvips directly in the gem. These changes won't affect that gem since it builds and uses the shared libraries directly. Thanks for the ping though, I do appreciate it.

[jcupitt]

Good news @renchap !

That's annoying about the mac, I'll check tonight. I guess the .so should not be there, is that right?

Yes, the Debian package was completely insane. They had libvips depending on nip2 (the libvips GUI!), so libvips pulled in a good chunk of the desktop. It's trimmed down now, I believe.

As I remember, splitting off the loaders was tricky on Windows. DLLs don't support back-linking, so a DLL can't call functions in the library it is loaded from. I believe the usual trick in the Windows world is to make a big struct with pointers to all the functions in your API, and then pass that into the DLL startup function.

I suppose it wouldn't be that much work to add, but it's been a low priority :(