linux builds are not fully static

[lynxlynxlynx]

A user reported this for the 0.8.6 binary:

error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory

Maybe the .a file isn't present when building. Either way, it shouldn't happen.

[lynxlynxlynx]

Looks like the file is there, so the question is why isn't it used or does something else pull in also the dynamic version:
https://packages.ubuntu.com/xenial/amd64/libpng12-dev/filelist

[lynxlynxlynx]

I don't understand something about linking apparently. With our static linking enabled, gemrb_core gets linked statically and ldd doesn't show it any more. Good. Of course it shows all the external deps still, since static linking requires that, eg.:

$ pkg-config sdl --libs
-lSDL
$ pkg-config sdl --libs --static
-lSDL -lpthread -lm -ldl -lasound -lm -ldl -lpthread -lpulse-simple -lpulse -lX11 -lXext -lcaca -lpthread

This leaves us with 70 entries, from Xorg stuff to various codecs and other crap we don't need. Do we need to install tens and tens of dev packages, just to get more .a files in the VM? Sounds extreme, but the fact remains that something else is pulling in libpng16.

[MarcelHB]

Isn't the correct setting of setting missing at all? I did two quick things that surprised me.

1.) I did a fresh cmake .. -DSTATIC_LINK=1 and did objdump -p ./gemrb/gemrb | grep 'NEEDED'. This now lists all basic (libm, libgcc, ...) and specific stuff (libsdl2, libpng1x, ...) with the shared object files. This should not be at all I guess?

2.) I put TARGET_LINK_LIBRARIES(gemrb ... -static) into the respective CMakeLists.txt. Now something interesting happens:

PluginLoader.cpp:(.text+0x1ea): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

[lynxlynxlynx]

Point 1 is what I was talking about when mentioning ldd output. I only expect to see the glibc stuff, per the warning you quote. The whole reason we want static builds is for them to be as standalone as possible.

Isn't adding -static something cmake does for us? But good thought, we tell it to do it for the plugins and core, while we perhaps need to be explicit for the binary? Though I don't see anything in the add_executable docs.

(and -Wl,--as-needed only works for dynamic linking)

[MarcelHB]

Just to get the overall idea:

1. Static builds are supposed to have their bunch of plugin shared objects?
2. Static builds are also supposed to have the core shared object?
3. Each plugin is supposed to be statically built, e. g. SDL2Video incorporates libsdl2 and so on?

Given that, we maybe have some error in the settings propagation, i. e. the executable should not list any specific dependency at all, not even shared objects on non-static builds? Because this is what I guess is subject to the individual plugins.

[lynxlynxlynx]

1&2. If you mean our plugins, then no. We load them at runtime, so in this sense, only gemrb_core is there or missing depending on whether the build is dynamic or static.
3. That's the goal, their add_library uses the static keyword. Not sure how much it matters though, since we load them dynamically, so I guess that just proves the problem is when linking the gemrb binary itself — like you say.

Time to play around, thanks.

[lynxlynxlynx]

That was fast, just omitting plugins and moving -ldl is enough. :)

[lynxlynxlynx]

Too fast, didn't clean the build dir properly.

[lynxlynxlynx]

Found a thread with the gist of things:
https://stackoverflow.com/questions/3762057/cmake-how-to-produce-binaries-as-static-as-possible

Couldn't test it fully, since adding set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static") breaks the build. It can't find openal, which is correct, since the ubuntu package only installs the shared library. So it's not easily available on travis either. Perhaps we could -Wl,-Bdynamic -lopenal -Wl,-Bstatic it, but I already lost interest.

Mistake from above: in the static build all plugins are linked statically, so dynamic loading never comes into play, d-oh.

[MarcelHB]

Have you had a look at AppImage? I've been given a recommendation to that for scenarios like these. I never tried it myself, but it somehow sounds like an option here.

[lynxlynxlynx]

That, flatpacks and similar always seemed like an overkill. But it would be interesting to see how big the produced files are.

[MarcelHB]

I gave it very short try:

$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release
$ make -j4
$ make install DESTDIR=./AppDir
$ LD_LIBRARY_PATH=./AppDir/usr/lib/gemrb linuxdeploy-x86_64.AppImage --appdir=./AppDir --output=appimage

Gives me about 8MB AppImage file. Contains about 40 shared lib copies, including Wayland, X, PulseAudio and stuff. It then complains it will not find plugins in /usr/lib/gemrb/plugins but you probably need to do some minor resource tweaks first. Anyway, does not sound too bad.

Edit: Yes, such paths must be relative to the binary file location inside of the AppDir.

[lynxlynxlynx]

40 is still better than 70, so not bad at all. Current builds are 7.4M, so that's fine too.

Still, it'd be nice to fix the static build as well. It looks like opensuse doesn't provide a static openal either. Should just try building it manually; who knows how many packages more would be required.

Also here's how ja2s integrated it with cmake & ci: ja2-stracciatella/ja2-stracciatella#1215

[lynxlynxlynx]

Ok, now this works fine. I'll migrate it to github first, try appimagelint again and then we can lay it to rest.

Unfortunately everyone is sunsetting xenial, so we could only get it in for a few months.

[lynxlynxlynx]

It's less than 2M of overhead, we never have to worry about libpng again, but unfortunately we can't use automated tests. As soon as the video driver tries starting up, gh just craps out. Maybe a thing to retry once actions/runner#241 is fixed.