Linux native-built with aggressive CPU optimisations that prevents loading on some VMMs

[SunandPeter]

Up to this point, I've always built signal-cli myself with gradlew/GraalVM.

Today I figured I'd try a pre-built version, so I downloaded the latest signal-cli native build for Linux

After decompressing it, I gave it a quick check (my usual ./signal-cli --help), whereupon I was greatly surprised to see that it failed to run at all!

I am running this on a Virtual Private Server (VPS) which advertises a Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz to its guests.

The CPU Flags are:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust smep erms xsaveopt arat md_clear

The output of the binary is:

./signal-cli --help The current machine does not support all of the following CPU features that are required by the image: [CX8, CMOV, FXSR, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, LZCNT, AVX, AVX2, BMI1, BMI2, FMA].

It looks like LZCNT, AVX2, BMI1, BMI2, and FMA are specified in the build scripts - most likely by the CPU "level" with the code generation tool.

Is this intentional or have some code generators' started defaulting to "same-as-build-host-level " CPU output settings?

Cheers,
Lorham

[AsamK]

signal-cli is built with the graalvm native default options, which should be x86-64-v3.

-march: generate instructions for a specific machine type. Defaults to x86-64-v3 on AMD64 and armv8-a on AArch64. Use -march=compatibility for best compatibility, or -march=native for best performance if a native executable is deployed on the same machine or on a machine with the same CPU features. To list all available machine types, use -march=list.

https://www.graalvm.org/latest/reference-manual/native-image/overview/BuildOptions/

[m-ueberall]

@SunandPeter: If you're using a Linux system with glibc 2.28 or newer, you could give signal-cli_ubuntu2004_amd64.gz a try, which is referenced on this project's "Binary distributions" wiki page.
The x86_86/aarch64 binaries are built using GraalVM CE (which currently does not support the -march option). According to elfx86exts, they should only require the following extensions:

% elfx86exts ./signal-cli0124_ubuntu2004_amd64 | grep -E "CPU architecture|Instruction set|CPU Generation"
File format and CPU architecture: Elf, X86_64
Instruction set extensions used: CMOV, MMX, MODE64, NOT64BITMODE, SSE1, SSE2
CPU Generation: Unknown
% elfx86exts ./signal-cli0124_ubuntu2004_arm64 | grep -E "CPU architecture|Instruction set|CPU Generation"
File format and CPU architecture: Elf, Aarch64
Instruction set extensions used: FPARMV8, NEON

[SunandPeter]

I'm on Ubuntu GLIBC 2.35-0ubuntu3.4) that's part of the Ubuntu "Server" 22.04.3 LTS, which I'm sure is fine. It just seems that the CPU target is "very modern" by default. I would image many VMM image deployments would run across things like this, or am I the odd freak out?

Thanks for the rapid replies. And yes, signal-cli_ubuntu2004_amd64 binary works correctly.

./signal-cli --version
signal-cli 0.12.4

Strangely (to me), the one that works is considerably smaller than the "hyper-optimised" build.

131,395,312 (x86-64-v3) versus 80,792,560 ("universal x86_64").

When I've done GraalVM native builds with native Rust/etc libraries, I get binaries closer to 94MB.

I'm sure this is all documented somewhere. I apologise for being so lazy.

Thanks, everyone.
=L=

[m-ueberall]

Strangely (to me), the one that works is considerably smaller than the "hyper-optimised" build.
131,395,312 (x86-64-v3) versus 80,792,560 ("universal x86_64").
When I've done GraalVM native builds with native Rust/etc libraries, I get binaries closer to 94MB.

This is because the GraalVM optimisations address memory footprint and performance, not the size of the binary (see this comment). The remaining differences in filesize can be explained by (1) the use of the backend (gcc vs. rustc/clang) and (2) whether or not the JNI libary – which is included in the binary and will be extracted at runtime – has been stripped of debugging information beforehand or not.

When using the LLVM backend along with GraalVM EE/Oracle GraalVM, the resulting binaries are indeed much larger; from an older test (with default settings):

-rw-r----- 1 sys-maint adm 116338288 2023-08-23 15:00:47 signal-cli0120_ubuntu2004_arm64__graalvm-jdk-17.0.8+9.1_stripped
-rw-r----- 1 sys-maint adm 149356440 2023-08-23 15:00:37 signal-cli0120_ubuntu2004_arm64__graalvm-jdk-17.0.8+9.1_unstripped

[SunandPeter]

What a strange new world this all is for me. I get that the performance goals might mean larger binaries, but I am puzzled by the notion that memory footprints might be improved by a larger binary. I operated from old school notions that statically loaded binaries that execute within a W^X context only have the code you load off-disk to execute in-core (read-only-execute). If you can write to it, you can't (shouldn't) be able to execute it, etc. I realise most operating systems probably don't work that way, but the ones I choose do.

Are you saying that these native builds embed self-extracting "libraries" that are decompressed and run at load-time? Oh dear. So which "native" builds are going to give me pure "no-extraction"/run-as-it-lays-on-disk runtimes?

Wow.
=L=

PS: Yes, I understand the nature of dynamic link libraries, but those are well-defined constructs that don't violate W^X restrictions for executables.

[m-ueberall]

Are you saying that these native builds embed self-extracting "libraries" that are decompressed and run at load-time? Oh dear. So which "native" builds are going to give me pure "no-extraction"/run-as-it-lays-on-disk runtimes?

I would have to check whether the libraries are compressed, but they're definitely being extracted and then loaded using the default mechanisms of the operating system in question. As long as an instance is running, you'll see two copies of the dynamic libraries in your default temporary folder:

2023-11-04T14:56:21+0100] sys-maint@desktop01:~% systemctl status signal-cli
● signal-cli.service - Send secure messages to Signal clients
     Loaded: loaded (/etc/systemd/system/signal-cli.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/signal-cli.service.d
             └─override.conf
     Active: active (running) since Sat 2023-11-04 14:55:43 CET; 42s ago
   Main PID: 2431632 (signal-cli_ubun)
      Tasks: 80 (limit: 76944)
     Memory: 106.3M
     CGroup: /system.slice/signal-cli.service
             └─2431632 /opt/signal-cli/bin/signal-cli_ubuntu2004_amd64 --config /var/lib/signal-cli daemon --system

Nov 04 14:55:39 desktop01 systemd[1]: Starting Send secure messages to Signal clients...
Nov 04 14:55:43 desktop01 signal-cli_ubuntu2004_amd64[2431632]: INFO  DaemonCommand - Starting daemon in multi-account mode
Nov 04 14:55:43 desktop01 systemd[1]: Started Send secure messages to Signal clients.
Nov 04 14:55:43 desktop01 signal-cli_ubuntu2004_amd64[2431632]: INFO  DaemonCommand - DBus daemon running on SYSTEM bus: org.asamk.Signal
[2023-11-04T14:56:26+0100] sys-maint@desktop01:~% ll -rt /tmp/*.so | tail -2
-rw------- 1 signal-cli signal-cli  5086360 2023-11-04 14:55:39 /tmp/7420734240291654823libsignal_jni.so
-rwxr--r-- 1 signal-cli signal-cli  1022176 2023-11-04 14:55:39 /tmp/sqlite-3.43.0.0-af074721-7617-431e-8c14-925495695756-libsqlitejdbc.so*
[2023-11-04T14:56:40+0100] sys-maint@desktop01:~%

(Don't ask me why they have non-unified file attributes, these are probably identical to those of the referenced copies at compile time; .so libraries do not necessarily require an executable flag, so this should even work on file systems like ZFS where you're able to prevent the execution of binaries located on certain datasets. (Maybe I'll explicitly test this one day.))

The only – but in this case unfortunately not so simple – W^X compliant solution is to provide GraalVM with static libraries instead of their dynamic counterparts shown above. However, modifying the upstream source for libsignal to produce a libsignal_jni.a isn't trivial but on the contrary would require quite a bit of expertise/effort (as can be seen from issues/discussions related to the aforementioned project) and is also not considered a typical use case upstream, see this issue.

[m-ueberall]

Addendum: Having written the above, I just stumbled upon option -H:+StaticExecutableWithDynamicLibC; I'll retrigger a local test build to ensure that the above is still valid (hopefully it isn't anymore 🤞).

EDIT: Looks like this doesn't change a thing, the dynamic libraries are still being extracted/loaded at run time.

[jpr105]

@m-ueberall thank you for your suggestion.

In fact I made all the main job on another more recent laptop (VM Parallels Desktop + Linux Mint) and finished associating Signal app using this release: signal-cli_ubuntu2004_amd64.gz

Perfect, thanks guys

[disaster123]

this is not really the same but may be related. I'm no longer able to run the signal-cli on Debian bullseye.

I get:

./signal-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./signal-cli)
./signal-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./signal-cli)

[m-ueberall]

@disaster123: It is definitely related; as mentioned above, you'll need to switch to one of the third-party native builds that only require glibc 2.28 or newer which are referenced here as well.

[morph027]

@disaster123

got a similar issue in my packaging repo, downgraded the build toolchain to buster to only require glibc 2.28 - https://gitlab.com/packaging/signal-cli/

[disaster123]

Thanks but not sure about your reply should I try or test anything?

[morph027]

Just a hint that you might try the native build from this project as it should work in bullseye 😉

https://packaging.gitlab.io/signal-cli/installation/standalone/index.html