`libopenblas` unusable when built with binutils 2.38 on Ubuntu 18.04 and 20.04

[carlocab]

Running a small test program (see below) that is linked to libopenblas results in the following error:

./test: error while loading shared libraries: libopenblas.so.0: ELF load command address/offset not properly aligned

This does not occur with binutils 2.37 or on Ubuntu 22.04.

This exercise was done with OpenBLAS v0.3.20. I haven't yet had the opportunity to try other versions.

The test program is

/* test.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include "cblas.h"
                                                       
int main(void) {
  int i;
  double A[6] = {1.0, 2.0, 1.0, -3.0, 4.0, -1.0};
  double B[6] = {1.0, 2.0, 1.0, -3.0, 4.0, -1.0};
  double C[9] = {.5, .5, .5, .5, .5, .5, .5, .5, .5};
  cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,
              3, 3, 2, 1, A, 3, B, 3, 2, C, 3);
  for (i = 0; i < 9; i++)
    printf("%lf ", C[i]);
  printf("\\n");
  if (fabs(C[0]-11) > 1.e-5) abort();
  if (fabs(C[4]-21) > 1.e-5) abort();
  return 0;
}

Compiled and executed with

gcc-12 test.c \
  -I/home/linuxbrew/.linuxbrew/opt/openblas/include \
  -L/home/linuxbrew/.linuxbrew/opt/openblas/lib \
  -lopenblas \
  -o test

./test

The GCC version does not seem important, since I see this bug with GCC 7+. (GCC 6 fails to build.)

This was discovered with a Homebrew-built OpenBLAS, but I can reproduce it by compiling OpenBLAS v0.3.20 without using Homebrew.

This may well be a bug in binutils. However, we've compiled a lot of software with binutils 2.38 at Homebrew, and only OpenBLAS seems to produce this problem.

More details at Homebrew/homebrew-core#106978.

[carlocab]

Oh, and for reference, OpenBLAS was configured with:

DYNAMIC_ARCH=1
USE_OPENMP=1
NUM_THREADS=56
TARGET=CORE2

I initially discovered this problem with a Homebrew-built GCC, but I can reproduce it with an Ubuntu-built toolchain.

[martin-frbg]

I do not think this has anything to do with openblas in particular - indeed a bug in binutils (e.g. historically there was such a problem with strip) or even the kernel on your machine (using host pagesize to align ELF sections) seems more likely

[fxcoudert]

Tried to report to binutils at https://sourceware.org/bugzilla/show_bug.cgi?id=29435

[carlocab]

Dropping the extra configuration (DYNAMIC_ARCH, USE_OPENMP, NUM_THREADS, and TARGET) makes the problem go away, in case that helps.

[martin-frbg]

Some of these just add more object modules (and consequently symbols, perhaps also library sections ?), NUM_THREADS influences static data,TARGET is probably the least likely to have any bearing on this problem. Unfortunately, searching for this libc error message turns up a mix of weird problems for/in various libraries (and with older binutils as well) that mostly seemed to just mysteriously go away on recompile/reinstall.

[fxcoudert]

At least in this case it's very reproducible, we've tracked it to the specific binutils version across three different systems/OS/compilers/etc.

[carlocab]

Yes, I've built OpenBLAS possibly over two hundred times today (thanks to GitHub for the free infra), and the result is consistent.

Dropping only DYNAMIC_ARCH (keeping the rest) also fixed things. Dropping only USE_OPENMP did not. I did not change NUM_THREADS (yet?).

Edit: Tweaking NUM_THREADS by setting it to 2, 4, 8, 16, 32, or 64, while keeping the rest of the configuration the same, does not help.

[martin-frbg]

As far as I understood (while working on something completely different and unrelated), the error message is related to segment load addresses within the library, so having ten times as many kernels (for DYNAMIC_ARCH vs build host) might simply increase the chance that a segment with a bad address, an offset beyond the range of the library, or whatever is generated. I am not aware of the build system doing any funky things (like using strip or patchelf that were implicated in some of the other bug reports I found), just FC/CC with -shared --Wl,whole_archive ... --Wl,-no_whole_archive to create the shared library from the ar archive of objects.

[fxcoudert]

I can confirm linking against the static library avoids the issue, so the object files themselves are fine.

I have put a copy of the faulty library (built on x86_64 linux) as well as the same one (built without DYNAMIC_ARCH) at this URL if someone wants to debug this issue further: https://www.dropbox.com/s/yth3avnycvomv4d/binutils-debug.tar.gz?dl=0

[carlocab]

Tested the binutils patch. It works. Happy for this to be closed, but I'll leave that up to you, @martin-frbg.

[martin-frbg]

Thanks for the feedback - I had seen HJ Lu's patch but not gotten around to trying it yet. (Interestingly the bug is not reproducible in my usual work environment that is based on opensuse - probably needs particular combinations of kernel and glibc to trigger it).
Hopefully that fix puts an end to all the similar heisenbugs that have sporadically been reported to other projects over the years, often in the context of complex python codes but at least once even while loading libstdc++
Guess I'll add an entry to the FAQ in the wiki to keep it somewhat visible.