borg2 b24: precompiled binaries don't work on cpus < x86-64-v3

[Ratio2]

$ ./borg-linux-glibc243-x86_64-gh
Illegal instruction

Debian 13 has:
openssl (3.5)
libc6 (2.41).

There is no reason for it not to work, other than the build environment

[ThomasWaldmann]

Your glibc is older than the one in the build system, that can be an issue, but iirc would lead to a different error msg.

Need more infos:

• you did not say which borg version you downloaded, was it 2.0.0b24?
• what is the sha256sum of your download?
• what cpu exactly is in your machine?

[Ratio2]

Yes, version is latest 2.0.0b24, 2.0.0b23 works fine:

$ ./borg-linux-glibc239-x86_64-gh --version
borg-linux-glibc239-x86_64-gh 2.0.0b23
$ ./borg-linux-glibc243-x86_64-gh --version
Illegal instruction

$ sha256sum borg-linux-glibc243-x86_64-gh
5fa90f4eac2b6ea4847848c24f3f812926cae117a7a444ee10a2301b9ec6aba7  borg-linux-glibc243-x86_64-gh

$ cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 58
model name	: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
stepping	: 9
microcode	: 0x21
cpu MHz		: 1527.322
cache size	: 6144 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
vmx flags	: vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_unknown vmscape
bogomips	: 4788.79
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

[ThomasWaldmann]

Thanks, that explains it. The i7-3630QM is Ivy Bridge: it has AVX, but none of the x86-64-v3 features (AVX2, BMI1/2, FMA, MOVBE, LZCNT).

Diagnosis: the 2.0.0b24 borg-linux-glibc243-x86_64-gh binary is compiled for x86-64-v3, so it needs a Haswell-or-newer CPU (or an equivalent VM CPU model). Reproduced under qemu-user with the IvyBridge CPU model (same feature set as your flags line): the bundled libpython3.14.so.1.0 dies with SIGILL on a BMI2 rorx instruction inside mimalloc's random init (ChaCha qround) during Py_Initialize, i.e. before any borg code runs, which is why there is no output at all. CPUs without AVX fail even earlier, on a VEX vpxor in mi_random_init_ex. Every model older than Haswell fails (qemu64, Conroe, Penryn, Nehalem, Westmere, SandyBridge, IvyBridge); Haswell, Skylake, Icelake, EPYC and newer pass a full create/extract/check/compact workload with all chunkers and compressors.

Root cause: the GitHub ubuntu-26.04 x64 runner image is Ubuntu's opt-in amd64v3 architecture variant, and its gcc defaults to -march=x86-64-v3. The DWARF producer strings of the bundled libpython (the actions/setup-python 3.14.7 build) and of borg's own extension modules all say -march=x86-64-v3, although neither the python-versions build nor our CI compile lines pass any -march flag. The bundled Ubuntu system libraries (libbz2, libuuid, libtinfo, ...) contain AVX code as well, so this cannot be fixed by compiler flags on our side; anything built on that image needs AVX2. (The runner-images repo even carries a workaround for the "amd64v3" apt index on 26.04 x64.)

glibc is not involved: the same binary runs fine on Debian 13 (glibc 2.41) on a newer CPU. Two side findings though: the bundled termios module and libtinfo.so.6 need GLIBC_2.42, so on Debian 13 the interactive passphrase prompt falls back to echoing the input (GetPassWarning) and readline cannot be imported. The arm64 binary (ubuntu-26.04-arm) is not affected by any of this; it is generic armv8-a code and runs under qemu with a Cortex-A53 model.

[Ratio2]

Yes, you are right:

sudo debootstrap --cache-dir="$PWD/cache" --variant=minbase resolute resolute
sudo arch-chroot resolute
apt install --no-install-recommends wget
cd /tmp
wget https://github.com/borgbackup/borg/releases/download/2.0.0b24/borg-linux-glibc243-x86_64-gh
chmod +x borg-linux-glibc243-x86_64-gh
./borg-linux-glibc243-x86_64-gh --version
Illegal instruction        ./borg-linux-glibc243-x86_64-gh --version
apt list --installed | grep libc6
libc6/resolute,now 2.43-2ubuntu2 amd64 [installed]

[ThomasWaldmann]

Potential solution: #10344

[ThomasWaldmann]

@Ratio2 can you test this binary (straight from the github-based build in pr #10344):

https://paste.thinkmo.de/cVc4KHnA#borg-linux-glibc239-x86_64-gh

[Ratio2]

Works:

$ ./borg-linux-glibc239-x86_64-gh\(1\) --version
borg-linux-glibc239-x86_64-gh(1) 2.0.0b25.dev49+g88bf6916e

[ThomasWaldmann]

OK, this is 49 commits after b24, but guess it is ok for testing. Thanks for the bug report!