The Holy Grail of Linux Binary Compatibility: musl + dlopen

[Splizard]

I guess using Go + Godot to build native & installable Android & iOS binaries (without any proprietary SDKs) was too easy. So it's time for a real challenge...

Linux Binary Compatibility

(some background reading: https://jangafx.com/insights/linux-binary-compatibility)

For a while now, it's been very easy to reliably ship command line software & servers for Linux, just run go build and out pops a single static binary that will run on any Linux distribution running kernel 3.2 or later (which was released in 2012, so there's plenty of room for backwards compatibility).

The problems begin to creep in when you want access to hardware accelerated graphics. All the GPU drivers on Linux require accessing dynamic libraries via the C ABI. These C libraries are built against a particular libc, which is most commonly glibc but there are also a selection of musl-based distributions. If you compile a glibc library or executable, it won't run on a musl system and vice-versa. That's a big incompatibility right there!

In fact, I've directly experienced this, as I recently replaced the OS on my personal computer with the musl edition of Void Linux. Compiling the Zed editor with musl for example, was quite the challenge. It turns out that building graphics.gd projects on musl was also very broken. Go doesn't properly support c-shared or c-archive when building against musl.

That's a problem, firstly because this is my distro now, I need to be able to build graphics.gd projects! Secondly, in theory, musl has better support for static linking than glibc; so if there's any solution to this Linux Binary Compatibility mess, it's probably going to have something to do with musl.

Supporting musl

To work around these musl issues with Go, I had to patch the runtime with a build-overlay that applies when building for GOOS=musl. This is a new GOOS that I've introduced to graphics.gd, specifically to make musl builds possible.

Next up, I decided to ditch c-shared builds for musl, these were only convenient because you could easily plug and play Go into the official Godot binaries. The Godot Foundation doesn't provide official musl builds, so instead, I'm linking the Go code directly with Godot c-archive to end up with a single binary. Amazing, graphics.gd supports musl now!

There's just one issue, this means whenever somebody wants to release their project for Linux, they would have to create two builds, a Linux glibc build + a musl build and somehow communicate to their users, to pick the correct binary. Hell, before I installed Void Linux I didn't even fully comprehend the differences between musl and glibc, this feels like I'm simply contributing to the problem!

Single Static Binaries + Graphics

Hold up! Earlier I reported that a key benefit of musl was better static library support. There should be a way to build a graphics.gd project into a single static binary. Well, here's the thing. Yes, you can totally do this. Godot includes all of it's dependencies on Linux, everything else is dynamically loaded at runtime, so just add the -static command and...

ERROR Dynamic loading not supported

Ouch, Godot wants to use dlopen to interface with X11, Wayland, OpenGL, Vulkan etc. As it turns out, musl refuses to implement dlopen for static binaries. They don't want anyone to load a glibc library from musl because there are fundamental incompatibilities between how they implement TLS (thread-local-storage).

Don't worry though! As dlopen is compiled as a weak symbol, this means, that as long as graphics.gd implements it, there's still a chance to get a single static binary that can execute on any Linux system 3.2 onwards.

The Holy Grail

There's some precedent for this, there's the detour technique in C which will let you dlopen SDL and show graphics when running without a standard library. There's also Cosmopolitan's dlopen which uses a similar technique. So the solution here is to extend this for musl.

The way this works, is by including (or compiling) a small C program for the target machine. We load the program and execute into it from the same process. This program brings in the host's dynamic linker so that we can steal the system's dlopen and longjmp back into graphics.gd. We wrap any dynamically loaded functions with an assembly trampoline that switches to the system's libc TLS for the duration of the call. It all starts looking a lot like cgo.

So after much hair pulling and LLM wrangling, it turns out that musl + dlopen is all we need to produce single static binaries + graphics for Linux. Everyone can now enjoy the Go single-static-binary experience on Linux with full support for hardware accelerated graphics.

Try it!

Here's a build of the graphics.gd Dodge The Creeps sample project that should execute (and hopefully render graphics) on any Linux system with gcc installed (we don't embed the helper binaries yet).

https://release.graphics.gd/dodge_the_creeps.static

You can also cross-compile your own project (on any supported platform)

GOOS=musl GOARCH=amd64 gd build

Note you may need to delete your export_presets.cfg so that the new musl export preset is added to your project

[phreda4]

I use the dlopen/dlsym technique in Linux and loadlibrary/getprocaddress in my language's VM.
Loading, sdl2, the same code (r3forth) works in both operating systems.
Assembly isn't necessary for the calls (of course, the language is specific).
Although I don't use floating-point numbers, these types of calls are very complicated (in fact, I don't use them) because the assignment depends on the parameter's position (poor design?).
I have the problem of doing the same for Android and the web; I think I'll have to simulate this mechanism. What do you recommend I research to do this? Is the interface with Godot similar ? I'd like to see how to add my language to this engine too.

[Splizard]

Does your language use TLS and/or brk? if not, then you won't have any issues. Creating GDExtension bindings for Godot is a bit involved, you need to parse a JSON file and then use the functions exposed in their C headers to generate an interface against the engine.

[X-Ryl669]

Doesn't work for me:

Thread 1 "dodge_the_creep" received signal SIGSEGV, Segmentation fault.
0x0000000002b13d16 in foreign_tramp ()
(gdb) bt
#0  0x0000000002b13d16 in foreign_tramp ()
#1  0x00007fffaa55a5a0 in ?? ()
#2  0x00007ffff75e4558 in ?? ()
#3  0x00007fff5ffec436 in ?? ()
#4  0x00007fffaa2ed710 in ?? ()
#5  0x00007fffffffc0d0 in ?? ()
#6  0x00007fff5ffe5ecc in ?? ()
#7  0x00007fffffffc0d0 in ?? ()
#8  0x00007fff5f9cc488 in ?? ()
#9  0x00007fffaa559a40 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)

[Splizard]

Thanks for the report, is there anything you can share about the system you are trying to run on?

[X-Ryl669]

Arch linux system, amd64 (using glibc here)

CPU: 8-core Intel Core i7-10875H (-MT MCP-) speed/min/max: 800/800/5100 MHz
Kernel: 6.17.13-1-MANJARO x86_64 
Shell: Bash inxi: 3.3.4

[nektro]

failed on nixos 25.11

Godot Engine v4.5.1.stable.custom_build.f62fdbde1 (2025-10-13 19:05:40 UTC) - https://godotengine.org
ERROR: Can't load Xlib dynamically.
   at: DisplayServerX11 (platform/linuxbsd/x11/display_server_x11.cpp:6747)
WARNING: Display driver x11 failed, falling back to wayland.
     at: setup2 (main/main.cpp:3220)
WARNING: Can't load the Wayland client library.
     at: init (platform/linuxbsd/wayland/wayland_thread.cpp:4307)
ERROR: Could not initialize the Wayland thread.
   at: DisplayServerWayland (platform/linuxbsd/wayland/display_server_wayland.cpp:1905)
ERROR: Can't create the Wayland display server.
   at: create_func (platform/linuxbsd/wayland/display_server_wayland.cpp:1875)
ERROR: Unable to create DisplayServer, all display drivers failed.
Use "--headless" command line argument to run the engine in headless mode if this is desired (e.g. for continuous integration).
   at: setup2 (main/main.cpp:3230)
WARNING: ObjectDB instances leaked at exit (run with --verbose for details).
     at: cleanup (core/object/object.cpp:2514)

[Splizard]

The binary executes! But it can't load X11 nor Wayland on your system, perhaps the libraries have different names on NixOS? If you are interested in helping here, I'm sure this can be resolved. As long as the binary actually launches (as it did here), these types of issues can be patched.

[nektro]

[meghan@nixos:~]$ nix-shell -p pkgconf xorg.libX11 xorg.libX11.dev xorg.libxcb xorg.libXau xorg.libXdmcp

[nix-shell:~]$ pkgconf --libs x11
-L/nix/store/ja09m2gfni7pqk4bsrhjb8y9cq74rspp-libx11-1.8.12/lib -lX11

but running Downloads/dodge_the_creeps.static runs and prints the same error as above

[ring-c]

This article was mentioned on hacker news https://news.ycombinator.com/item?id=46762882
Interesting comments out there.