JCuda 11.5.2 binary incompatibility

[corepointer]

Hi!

The Linux binaries of JCuda 11.5.2 seem to be incompatible with at least Ubuntu 20.04 (latest LTS) and Debian 11. They seem to have been compiled on a more recent system with a newer glibc. If I compile from source, everything works fine. If I use the ones downloaded by maven, I get the error:

java.lang.UnsatisfiedLinkError: /tmp/libJCublas2-11.5.2-linux-x86_64.so: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.3
4' not found (required by /tmp/libJCublas2-11.5.2-linux-x86_64.so) 

Glibc version in Ubuntu20/Debian11 is 2.31:

/lib/x86_64-linux-gnu/libc.so.6 
GNU C Library (Debian GLIBC 2.31-12) stable release version 2.31.
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 10.2.1 20210110.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.

[jcuda]

I have to ask @blueberry here: Is there any information about the Linux version that you are using, and the installed GLIBC version? (There had been a similar issue a few years ago ( #10 ), but it's probably just a general version mismatch...)

[blueberry]

I'm using Arch linux with glibc 2.35. (I guess that's the default there, since I don't remember explicitly changing it).

[jcuda]

As usual, I'm not sure about common workflows on Linux: Is it reasonable for people to update their installed GLIBC, or is this so much a "core" library that it should usually not be touched by users?

[blueberry]

As with any other library, you can use newer libc in a specific process, by setting up one or two environment variables (for your process only or whatever you like).

[corepointer]

The C library (glibc in most cases for Linux) is a core component of any system, as all programs rely on in in some form. If you mess with it your system is gone (been there, done that). Requiring to install another glibc and set up the right environment is not something I would ask from an ordinary user.
Now the question is if the newer can run the older - compiling for glibc 2.31 and running on 2.34+.

[blueberry]

That is correct, but I'm not talking of installing glibc anywhere where it could be mistaken for your original version of glibc. I'm simply talking about putting glibc.so (or whatever binary file is appropriate) in a directory specific to a project and including it in LD_LIBRARY_PATH (or, again, whatever else is appropriate) for that process only, not globally. I don't see how that mess up the system.

An ordinary user of JCuda is, I believe, a programmer, which should be able to handle something like this. An ordinary user of that programmer's program is not a programmer and is probably not able to do this, but, OTOH, that user is probably not able to use even java -jar theprogram.jar, but relies on running a shell script, which could set up the process to use the appropriate glibc shipped inside the program's directory?

I agree that it would be best if this was compiled with a glibc version that would work on all systems. There's only a question then which versions to support, as not every LTS of every distribution is at the same version of glibc.

[blueberry]

Maybe the best solution would be to build the awaited version 11.6.0 on Ubuntu LTS?

[corepointer]

I agree, setting this up can be asked of a programmer who uses jcuda. But the situation is far from ideal. It unnecessarily blows up package size and what if other native shared libraries that are used concurrently in some java program require the original glibc?
Trying this out on Ubuntu20/Debian11 (same glibc version) seems reasonable imho. A majority of desktop users will use this or something newer. Server users tend to be conservative with upgrades but can be expected to handle containerized setups (docker/singularity/etc). Using singularity myself on some hpc systems. That works quite well 👍

[jcuda]

From my understanding (and please correct me if I'm wrong), there are roughly these options;

• Require the user of JCuda to update their GLIBC. This would be "dangerous", because they could mess up their system
• Require the user of JCuda to manually move a newer version of the library into a certain path. That would be cumbersome, and error-prone

(And both of these options are not something that a user should have to do. A user should just declare the Maven dependency and be done...)

• Re-Compile the binaries of 11.5.2 with some older GLIBC dependency, and publish it as 11.5.2b
• Or finally: Accept that 11.5.2 has this issue, and make sure that this does not happen for 11.6.0

I'm not sure what would be necessary for the latter. I could imagine that ( @blueberry ) it could be technically possible to create a 11.5.2b release with the older dependency. You'd probably have to manually unpack this older GLIBC version somewhere and declare it as dependency in CMake or so. Whether or not that's worth the effort is hard to say. But... I assume that something similar would have to be done to avoid the problem in 11.6.0 as well.

compiling for glibc 2.31 and running on 2.34+.

Iff these version numbers roughly follow 'semantic versioning', then this should be fine, and one could probably say that the dependency to GLIBC should be ~"the oldest version that works". If it was possible to compile this with, say, version 2.01 and this version still worked for someone who had installed 2.34, the problem could be solved.

Still, I'm not sure how exactly someone with a "modern" Linux could configure the build so that it only requires an "old" GLIBC version...

[blueberry]

Yes, if necessary I'll compile it with older glibc that I can put at the appropriate directory inside a jcuda build infrastructure, but I don't know how to set cmake to do that, as I find it utterly confusing.

[blueberry]

OTOH, the next Ubuntu LTS 22.04 is due in April, I guess it'll use current glibc, so I think this issue already solved itself...

[corepointer]

That might be true for desktop systems. On cluster environments I have to put up with Debian11, Ubuntu20, RHEL8, CentOS7. But as discussed, server users should know how to set up their environment. On these systems the bigger problem is to get recent drivers installed (which the admins of these systems are very reluctant to do, so I'm stuck with CUDA 11.2 - problem solved) 😮‍💨

[jcuda]

(Sorry for the delay).

Admittedly, I'm not entirely sure how to proceed here.

To my understanding, it would be preferable to have the library compiled with a GLIBC that is ~"as old as possible to work, and in a version that common Linux distributions are downward compatible with". But I don't know the details about how that could be achieved technically.

If somebody can provide JCuda binaries for 11.5.2 that are compiled with such an "older" GLIBC version, then I could try to bundle this and publish it as 11.5.2b.

The same question will come up in view of #44 - maybe compiling the 11.6.1 binaries with such an "old" GLIBC could solve the issue for most users.

Any ideas?

(In doubt, I could also try to revive my Linux VM and see whether I can compile the binaries with an older version there, but that tends to be a pain in the back, and I'll never be able to test these binaries in a VM...)

[blueberry]

@jcuda I can at least test these binaries...

[corepointer]

As I've really become a fan of singularity lately, I can compile & test different versions quite flexibly. I'll be in need of binaries for CUDA 11.2 and 11.3 as well, because some cluster environments I am using have older divers that don't support newer CUDA SDKs. Haven't tested yet, if the current binaries work there.
I can help out with compile & test if you want (potentially with some delay though). How would these binaries be delivered/published?

[jcuda]

@corepointer The binaries for 11.2 should already be available, as of #39 (comment) (11.3 was essentially skipped - it's sometimes hard to allocate the time to make reqular, quick and frequent updates...). I don't know whether the binaries for 11.2 have the same GLIBC issue, but it would be good to know.

How would these binaries be delivered/published?

The build instructions at https://github.com/jcuda/jcuda-main/blob/master/BUILDING.md describe the overall process. The result of that process should be a bunch of JAR files in the jcuda-main/output directory that include the JARs with the Linux-specific native libraries in them - these JARs are all that I need in order to create the Maven release.

[jcuda]

I'll close that for now. We know the reason for the incompatibility, and how to 'solve' it. (Essentially: Build JCuda with the 'oldest' version of GLIBC that can be used). One compatibility release for 16.1 was released in #52 . I'll probably not create compatibility releases for older versions, but we can keep an eye on the GLIBC version for future releases. Thanks again @corepointer for the binaries provided in #51 (comment)