Error from Python bindings: undefined symbol: g_variant_dict_ref

[kylemacfarlane]

I'm trying to use the libs you have available with pyvips but I always get this error:

cannot load library libgobject-2.0.so: /var/task/libgobject-2.0.so: undefined symbol: g_variant_dict_ref.  Additionally, ctypes.util.find_library() did not manage to locate a library called 'libgobject-2.0.so': OSError
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 2, in lambda_handler
    import pyvips
  File "/var/task/pyvips/__init__.py", line 39, in <module>
    gobject_lib = ffi.dlopen(_gobject_libname)
  File "/var/task/cffi/api.py", line 140, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/var/task/cffi/api.py", line 786, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/var/task/cffi/api.py", line 781, in _load_backend_lib
    raise OSError(msg)
OSError: cannot load library libgobject-2.0.so: /var/task/libgobject-2.0.so: undefined symbol: g_variant_dict_ref.  Additionally, ctypes.util.find_library() did not manage to locate a library called 'libgobject-2.0.so'

This errors means that the .so was found but failed to load.

Here's a copy of the packaged project (literally all it does is import pyvips): https://drive.google.com/file/d/0BzTQ-TolcOhSTkZrUk9KQ01qS3M/view?usp=sharing

My guess would be it's something to do with how the cffi Python module has been built. Amazon requires Python 3.6 on lambda but only has 3.5 on their AMIs. So Python 3.6 has to be built from source and then cffi has to be built from source.

[kylemacfarlane]

From what I understand it shouldn't matter that libvips was built without the Python dependencies stuff because the newer pyvips bindings don't use it: https://jcupitt.github.io/libvips/2017/09/01/libvips-for-Python.html

Also this error isn't lambda specific. I built everything with docker on EC2 and the same error happens there too.

One strange thing I noticed is that if I try to load the .so a second time it appears to work. But then later on I hit this error:

OSError: cannot load library libvips.so: /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by /var/task/libpng16.so.16)

Notice that it's looking for the system libz.

Multiple people are reporting the exact same bug with sharp: lovell/sharp#843 This kind of suggests it's not a Python problem but rather an issue with how libvips is built.

The problem actually shows up by merely using ldd on libpng from an EC2 instance.

> wget https://github.com/stechstudio/libvips-lambda/releases/download/v1.0.1/libvips-8.5.5-lambda.tar.gz
> tar xzf libvips-8.5.5-lambda.tar.gz
> ldd lib/libpng.so
lib/libpng.so: /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by lib/libpng.so)
        linux-vdso.so.1 =>  (0x00007ffd59ce9000)
        libz.so.1 => /lib64/libz.so.1 (0x00007ff82e0b4000)
        libm.so.6 => /lib64/libm.so.6 (0x00007ff82ddb2000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ff82d9ed000)
        /lib64/ld-linux-x86-64.so.2 (0x000056156d60c000)

This is probably because libvips-lamda builds zlib 1.2.11 but libpng seems to be pinned to 1.2.9? Some references to libpng/zlib version hackery here: pnggroup/libpng#155

But I tried both updating libpng to 1.6.32 and downgrading zlib to 1.2.9 and neither worked. It's still picking the system version of libz and then complaining that it's not 1.2.9.

And I still don't even know if this is the cause of my initial pyvips problem.

[kylemacfarlane]

I managed to fix the zlib problem by doing a simple build with only jpeg and png support and using the mostly preinstalled libraries instead of building every dependency from scratch. However in the end it doesn't fix my g_variant_dict_ref problem. At least now that I have a simple build going I can narrow it down more easily.

sudo yum -y group install "Development Tools"
sudo yum install -y zlib-devel libffi-devel expat-devel libpng-devel libjpeg-devel

wget https://download.gnome.org/sources/glib/2.52/glib-2.52.0.tar.xz
cd glib-2.52.0
./configure --with-pcre=internal --enable-shared --disable-libmount
make
sudo make install
cd ..

wget https://github.com/jcupitt/libvips/releases/download/v8.5.8/vips-8.5.8.tar.gz
tar xzvf vips-8.5.8.tar.gz
cd vips-8.5.8
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:~/glib-2.52.0
./configure
make

cd ..
mkdir libvips-lambda
cp -a vips-8.5.8/libvips/.libs/libvips.so* libvips-lambda/
cp -a /usr/local/lib/libgobject-2.0.so* libvips-lambda/
cp -a /usr/local/lib/libglib-2.0.so* libvips-lambda/
tar zcvf libvips-lambda.tar.gz libvips-lambda/

1. The libvips.so generated by this is 9mb vs the 3mb you guys made. Probably some options need to be turned off in configure.
2. Building it this way we only need the libvips and glib shared libraries as everything else is preinstalled on lambda. That would change as more image formats are enabled.

[kylemacfarlane]

Oh I see what the problem is. There's already a libgobject-2.0 installed on lambda but it's very old and g_variant_dict_ref requires >= 2.40.

The problem is that LD_PRELOAD doesn't work under Python on lambda. That was the first thing I tried for zlib and it would presumably fix both these problems.

[kylemacfarlane]

Some progress.

1. All these problems can be fixed on EC2 by setting LD_PRELOAD. However for whatever reason it has no affect on lambda.
2. g_variant_dict_ref can be solved by setting LD_LIBRARY_PATH=/var/task:LD_LIBRARY_PATH on lambda.
3. But then you run into version 'ZLIB_1.2.9' not found again which can't be fixed on lambda due to LD_PRELOAD not working.
4. The png/jpeg only version of libvips I built to fix zlib doesn't throw any errors but instead the lambda function times out suggesting that something else is wrong with my build. The png/jpeg only version of libvips I built to fix zlib appears to run fine once LD_LIBRARY_PATH is set on lambda.

[kylemacfarlane]

Success I think! I edited my previous comment.

The libvips I built times out when I set the lambda timeout to 3s but if I set the timeout to 10s it runs successfully in <100ms. I'm confused.

But anyway there's definitely a problem with zlib that needs to be fixed both in this project and sharp.

[kylemacfarlane]

The fix seems to be to simply downgrade to zlib 1.2.8 by setting VERSION_ZLIB=1.2.8. Even if ldd still complains on EC2 just ignore it and the libs will work on lambda.

The reason it has to be 1.2.8 is because Lamda pre-imports a bunch of Python modules which you can see by doing print(sys.modules.keys()) and one of them is zlib. Additionally it seems like environment variables are set late on Lambda (presumably for security reasons) so it pays no attention to LD_LIBRARY_PATH/LD_PRELOAD and there's nothing we can do to prevent it from loading /lib64/libz.so.1.2.8.

[bubba-h57]

Huh. I had not run into that. Thanks for the rundown!

[oak-tree]

I get the same error on ubuntu 16. Do I must built from source in order to solve this issue?