[RFC/tracking issue] deprecating -lib32 multilib system

[q66]

So, right now we have a system in place that works like this:

• When building i686 packages, and i686 specifically and only, libraries are taken and special -32bit packages are also generated for x86_64 in the special multilib/ repository, whether it's just libraries or full package is then controlled via the mode and so on
• In order to ensure that these libraries can be installed together with native x86_64 ones and not conflict, i686 packages are usually configured to use /usr/lib32 and all other packages are configured to use /usr/lib; through hooks, we make sure everything ends up in /usr/lib in the end, except the -lib32 packages which are populated into lib32 (this is also needed in case programs, e.g. mesa, dynamically load library plugins from somewhere else)
• This is pretty clunky and hacky, and only works for i686 on glibc, and only works with x86_64 on glibc
• It is also potentially problematic when migrating off buildbot, as it means the x86_64 and i686 targets cannot be separated
• But mainly it's a problem for other architectures and musl
• It also used to be a problem that the lib32/lib64 symlinks did not exist in unconfigured chroots, so toolchains had to be configured to always use dynamic linkers from lib and that was terrible and ABI breaking, but has since been mitigated and will be fully put into place with gcc10 upgrade; we could fix this because /usr/lib64 is now a symlink to /usr/lib on 64-bit systems, and /usr/lib32 is a symlink to /usr/lib on 32-bit systems, and these symlinks are a part of base-files, always

Proposed new system

Therefore, I am proposing a new system. The goals of this system should be:

• It should not need any special handling in xbps-src
• It should not need any special treatment by xbps itself
• It should not need generation of any special multilib repos/packages
• It should work on any architecture, any libc with a 32-bit counterpart

There are distributions, mainly Debian, that solve this by having a complete multiarch system in place. That involves having support for handling multiple architectures at once in their package manager, and having everything built in a way that /usr/lib is never used directly, but rather libraries use a special prefix that contains the target triplet, so there are never conflicts. While I considered implementing such system in Void, it quickly turned out to be too large of a can of worms, without that much benefit, which would also require a considerable amount of effort put everywhere. So, instead I am proposing what I call multilib prefixes, which is a simplified, generalized system that only handles the common case of running 32-bit processes on 64-bit systems (or alternatively, 64-bit processes on systems running 64-bit kernel with 32-bit userland).

How do multilib prefixes work?

Before explaining how this is going to work, I'm going to describe the steps needed to get there, it will make it easier. The ones marked so are already done.

• First, we should fix our toolchains to use proper dynamic linker paths again
• Then, we should update the toolchains to use lib32 on 32-bit systems and lib64 on 64-bit systems, unconditionally (this is actually not truly important, as the real libdir is handled separately by build systems, but still, for correctness)
• Then, we should fix xbps-src to allow lib64 paths in addition to lib32, by fixing the xbps-src hooks
• Then, we should fix glibc (done) and other related base bits (crosstoolchains and so on) to use lib32 and lib64 instead of lib
• Similarly, all build-style need to be fixed to follow this (use lib${XBPS_TARGET_WORDSIZE}), and all templates that explicitly set lib32 on i686 need to be generalized (e.g. mesa, pulseaudio and so on; these generally load plugins at runtime, so the path is important there; for most programs, it's not, since library paths are not written in executables/shared libs, only names are, and the lookup paths are controlled by dynamic linker)
• We will also need to find every program/library that does dynamic loading like above and the search path is hardcoded and meaningful there; these templates need to have their revision bumped to rebuild, probably after gcc is updated to 10 and so on - we already did this for most known things, others will be done as they are found
• For musl, we will need to ship custom /etc/ld-musl-ARCHITECTURE.path which will override the default dynlinker search paths to use the appropriate lib32 or lib64 ones - musl by default always uses lib
• For glibc, this can be (and already is, though a bit by accident) handled by ld.so.conf.d
• Fix up locales on musl to always use /usr/share/locale
• Then we will need to introduce handling for the prefixes themselves. More about that later.

Now, to get to how it works, and also why it works.

For one, the /usr/lib path we currently use will be preserved. This is because xbps-src is already updated to intercept anything installing into lib32 or lib64 and properly migrating it to lib using symlinks. This is important.

On 64-bit systems, base-files contains symlinks so that e.g. /usr/lib64 points to /usr/lib. Similar thing happens for lib32 on 32-bit systems. However, since the build systems and so on will automatically default to lib32 or lib64, all library lookups will happen through those symlinks, and not through the actual path. What does this mean?

In order to deal with the case of 32-bit binaries on 64-bit systems, all we have to do is simply create a new Void "root". Let's say this root will be /usr/i686-linux-gnu. This root will contain all the usual files. So your 32-bit libraries will be located in /usr/i686-linux-gnu/usr/lib.

How to install things into this prefix? That is easy; you can just use xbps-install. For example, XBPS_ARCH=i686 xbps-install -r /usr/i686-linux-gnu .... That means you will be able to make use of the entire standard 32-bit repo without any special handling. This is in fact how xbps-src already manages makedepends when cross-compiling.

What is needed now is to hook this prefix into your system. Since 64-bit systems will use lib64 and 32-bit ones will use lib32 for lookups, they are non-conflicting. Therefore, what you can do is make your /usr/lib32 a symlink to /usr/i686-linux-gnu/usr/lib. And that's it; if you run any 32-bit process in your 64-bit system, it will look up its libraries through /usr/lib32, which will point to your custom prefix, containing all the dependencies.

If you think about it, that's not very different from how the current 32bit libs are handled. It's just generalized, and does not use any special tooling.

Of course, this is actually not going to work as is, out of box. There is also the dynamic linker to take care of. This dynamic linker does not always respect lib32 paths or whatever, so it's going to be slightly more involved. Notably on musl, dynamic linker always comes from /lib. E.g. /lib/ld-musl-i686.so.1.

However, the dynamic linkers are the one thing where we are sure there will never be file conflicts. Therefore, you can just symlink the dynamic linker from your multilib prefix to the path where it's expected to be, and everything will just work.

And that is basically the gist of it; there is nothing much more complicated about it conceptually. Of course, there are issues we need to work out, because in practice it's not as smooth.

Problems

• If we choose /usr/<target-triplet> as the default multilib prefix, it will conflict with cross-toolchains. What we could do is integrate this with managing of cross-toolchains on your host system and make them a lot more useful. However, cross-toolchains also provide the libc and other packages that would otherwise be installed. In xbps-src, cross-vpkg-dummy is installed.
• We will need tooling for managing these multilib prefixes. Using just xbps as it is is fairly clunky, because by default these prefixes don't have repos set up in them and so on. Especially initial setup is relatively involved. Another problem is making sure the symlinks are properly set up, for lib32 (or lib64) as well as the dynamic linker symlinks if necessary. The hypothetical management tool could deal with this automatically, besides providing functionality for installing the packages into the prefix.
• Right now, /usr/lib32 is actually not a symlink on 64-bit systems, but a directory. This is because /usr/lib32/locale is a symlink, to your native locale data. We would need to replace that. This is actually a glibc-only problem; application locale data go in /usr/share/locale but glibc puts some autogenerated files in /usr/lib/locale which need to be shared.
• There might be packages that require wordsize-specific data files in /usr/share. These need to be found, and such files need to go to /usr/lib(32|64). This is the right thing to do either way, since they're no different from binary files, /usr/share should only contain architecture-independent data.
• There might be templates where it would be useful to split the libraries manually into a subpackage (foo-libs, for example). This is not a hard requirement, but since multilib prefixes should generally only contain libraries and data files are useless, it might reduce how much disk space we waste.
• There are some "full" 32-bit packages which also contain executable programs. Those would need to be handled specially. Perhaps the hypothetical "prefix management tool" could also take care of installing appropriate symlinks in /usr/bin, probably upon user request.

There might be more issues to work out, these are just ones I can think of right now. I think all of them are solvable, so we should be able to make a concept like this into reality sometime in near future, and deprecate/remove the current multilib repos.

@void-linux/pkg-committers

[ahesford]

In general this seems like a great idea. Are there any issues with rpaths in this scheme? I believe there are at least a few packages that install executables or shared libs that rely on rpaths to find some dependencies. Of course, if they are hard coded in a way that breaks your proposal, they would probably be broken under the current i686 multilib anyway...

[q66]

if there are they are already broken in the current scheme, but things with rpaths don't need to concern us much since the whole 32-bit multilib thing is largely for compatibility libraries, it is rare that you'd use it for 32-bit programs (and most of these should still be okay even then)

[Skirmisher]

Seems pretty good all around. Standardizing the existing lib32/64 setup while pointing the multilib component to a separate prefix is a pragmatic approach that still leaves room for flexibility.

I do want to address the prefix location thing: cross toolchain prefixes are weird because they have libraries for the prefix arch in usr/lib, but have host arch binaries in usr/bin, making it inconsistent with a real install (especially since in a multiarch situation, you would be able to run the "native" versions of those tools anyway). I'm not sure if there's a way to work around the host-arch binaries, or if it would be straightforward to move the cross prefixes somewhere else, e.g. /usr/cross/<prefix>... If nothing else, though, I would find it reasonable to put multiarch prefixes in their own subdirectory like /usr/multiarch or /usr/arch or whatever else.

[q66]

yeah the cross prefix unification thing is just an idea but not something we necessarily have to do

it also has other problems, e.g. gccgo's libgo in crossprefix is built with a dummy static libucontext on musl so it probably does not work, and the libc is built with a partial compiler and sometimes lower optimization level so we likely do not actually want to run anything relying on the crosstoolchain's target libs :P

[ericonr]

We will need tooling for managing these multilib prefixes. Using just xbps as it is is fairly clunky, because by default these prefixes don't have repos set up in them and so on. Especially initial setup is relatively involved.

This initial setup will be rather clunky, since it's not really possible to derive the repo path from whatever is configured in /etc/xbps.d; aarch64 has a completely different path from what arm uses, for example.

Just as a note for something that may not be immediately obvious, the reason we need the /usr/lib32 symlink as well as the dynamic linker configuration, is that the latter affects how programs are loaded (where the linker looks for the libraries your program is linked against), while the former is a fix that will allow libraries like mesa and pulse to simply work, since they will search for things to dlopen in /usr/lib32.

A few concerns that came to mind now were:

• the case where a library looks for binary helpers (like PAM does, but you shouldn't have a copy of PAM in the alternate root, so it isn't a good example) in /usr/libexec will be kind of broken, unless one has the "native" counterparts installed and fully compatible.
• our wine-common package, specifically the /usr/bin/wine launcher, will need to be thoroughly fixed, but I'm not entirely sure how.
• and

There might be packages that require wordsize-specific data files in /usr/share. These need to be found, and such files need to go to /usr/lib(32|64). This is the right thing to do either way, since they're no different from binary files, /usr/share should only contain architecture-independent data.

At least one package that I know of is aspell. EDIT: I was mistaken about aspell, it did put the files in /usr/lib.

[ericonr]

Packages that receive special treatment for i686 (due to multilib) and should be fixed (like #27291) for this project to work (I will be updating this list as I find them; feel free to edit. checks will be added when the package has been fixed and rebuilt):

• attr
• MangoHud
• pulseaudio
• glibc
• acl
• sqlite-replication
• common/environment/configure/gnu-configure-args.sh itself
• libhugetlbfs (LIB64=lib64)
• avahi-discover (does some weird stuff with tmpinstall)
• cegui (might just allow removal of a sed block)
• steam INSTALL hook? not sure
• python(3)-tkinter
• amdvlk
• alsa

[ericonr]

Our 32bit nvidia packages are created manually, we'd have to figure out how to make them work.

For this, we could try to build the nvidia stuff inside a i686 masterdir and make packages only with the libraries, without the DKMS module.

[ahesford]

I just did this on an aarch64 installation on a Raspberry Pi, when I realized I couldn't run retroarch because all of the cores precompiled by that project are 32-bit only. Enter multilib forr ARM. Except for the repository path differences (an easy fix), everything worked beautifully.

mkdir -p /usr/armv7l/etc/xbps.d
echo "repository=https://mirrors.servercentral.com/voidlinux/current" > /usr/armv7l/etc/xbps.d/00-repository-main.conf
XBPS_ARCH=armv7l xbps-install -S -C /usr/armv7l/etc/xbps.d retroarch mesa-dri
rm -rf /usr/lib32 && ln -s armv7l/usr/lib /usr/lib32
ln -s /usr/armv7l/usr/lib/ld-2.30.so /usr/lib/ld-linux-armhf.so.3
/usr/armv7l/usr/bin/retroarch

We could wrap this in a simple script that derives XBPS_ARCH from the executable name (e.g., xbps-install-<arch>) which can check for /some/path/to/<arch>, complain if it finds no etc/xbps.d/00-repository-main.conf (or maybe prepopulate the file) and make the necessary links after installation if they are missing.

[github-actions]

Issues become stale 90 days after last activity and are closed 14 days after that. If this issue is still relevant bump it or assign it.

[the-maldridge]

We'd certainly like to keep this going, I'm going to exempt this from bot actions with the request label, its not quite the right label, but its the quick fix I have right now.