Building libjxl dependency Highway 0.14.0 halfway results in "Error running executable: Illegal instruction"

[kylxbn]

Hello. Hopefully I'm doing this right.

Describe the bug

I am trying to build libjxl (0.5) from the Arch Linux User Repository (AUR) and halfway while building its dependency Highway (0.14.0) from the AUR also , I get this error when I am using the GCC:

[ 56%] Building CXX object CMakeFiles/highway_test.dir/hwy/highway_test.cc.o
[ 58%] Linking CXX executable tests/highway_test
CMake Error at /usr/share/cmake-3.21/Modules/GoogleTestAddTests.cmake:77 (message):
  Error running test executable.                                                                                                           

    Path: '/home/kylxbn/.cache/paru/clone/highway/src/build/tests/highway_test'                                                            
    Result: Illegal instruction                                                                                                            
    Output:                                                                                                                                
                                                                                                                                           

Call Stack (most recent call first):                                                                                                       
  /usr/share/cmake-3.21/Modules/GoogleTestAddTests.cmake:173 (gtest_discover_tests_impl)                                                   


make[2]: *** [CMakeFiles/highway_test.dir/build.make:102: tests/highway_test] Error 1
make[2]: *** Deleting file 'tests/highway_test'
make[2]: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
make[1]: *** [CMakeFiles/Makefile2:368: CMakeFiles/highway_test.dir/all] Error 2
make[1]: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
make: *** [Makefile:146: all] Error 2
make: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
==> ERROR: A failure occurred in build().
    Aborting...
error: failed to build 'highway-0.14.0-1': 

When I try using the clang compiler, I get this:

[ 56%] Building CXX object CMakeFiles/highway_test.dir/hwy/highway_test.cc.o
[ 58%] Linking CXX executable tests/highway_test
CMake Error at /usr/share/cmake-3.21/Modules/GoogleTestAddTests.cmake:77 (message):
  Error running test executable.                                                                                                           

    Path: '/home/kylxbn/.cache/paru/clone/highway/src/build/tests/highway_test'                                                            
    Result: Illegal instruction                                                                                                            
    Output:                                                                                                                                
                                                                                                                                           

Call Stack (most recent call first):                                                                                                       
  /usr/share/cmake-3.21/Modules/GoogleTestAddTests.cmake:173 (gtest_discover_tests_impl)                                                   


make[2]: *** [CMakeFiles/highway_test.dir/build.make:102: tests/highway_test] Error 1
make[2]: *** Deleting file 'tests/highway_test'
make[2]: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
make[1]: *** [CMakeFiles/Makefile2:368: CMakeFiles/highway_test.dir/all] Error 2
make[1]: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
make: *** [Makefile:146: all] Error 2
make: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'

This has been happening since libjxl 0.3.x days (not sure if even older versions are affected, since I only tried compiling 0.3.x), and I was thinking that it was just that my CPU is really old (since it doesn't support SSE 4.2). But I guess I better ask for confirmation regarding this, too.

To Reproduce

I have Paru as my AUR helper, and just trying to install libjxl with it will trigger the error.

paru -S libjxl

It seems to be failing while building the test binaries for Highway. I'm not really familiar with how Highway works, but it looks like it will build test executables, then try to execute it. However, maybe because of that missing SSE 4.2 support on my CPU, some inline assembly code fails to run, resulting in that "Illegal instruction" error and the build failing.

Expected behavior

I was hoping that an Illegal instruction error would not occur.

Environment

• OS: Arch Linux with GNU/Linux 5.13.7-zen1-1-zen (Zen kernel)
• Desktop: KDE Plasma 5.22.4
• Multimedia: PipeWire 0.3.32 and Wayland 1.19 (with xorg-xwayland 21.1.2)
• Compiler version: GNU GCC 11.1.0 and Clang 12.0.1
• Compile flags: Please see below
• Terminal: yakuake
• Shell: bash 5.1.008
• CPU type: Intel Core 2 Duo P8400 (x86_64) @ 2.26GHz
• CPU flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm pti tpr_shadow vnmi flexpriority vpid dtherm ida
• Hardware: 4GB RAM, 140GB HDD, Dell Latitude E5400
• cjxl/djxl version string: cjxl v0.5 (not sure if that's how it's shown--I wasn't able to compile cjxl due to dependency problems)

Compile flags:

CARCH="x86_64"
CHOST="x86_64-pc-linux-gnu"
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \
        -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security \
-fstack-clash-protection -fcf-protection"
CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,zelro,-z,now"

Additional context

This is an old laptop and it probably does not support SSE 4.2. While using SIMD instructions is cool and all, maybe a fallback to good old x86_64 ISA would be good if the CPU does not support SIMD? I'm honestly not even sure if the compiler can detect it and if it can be done with stuff like preprocessor commands, but... Just a suggestion? I'm honestly not that familiar with C++.

This was how the build files were generated when I am using GCC

-- The C compiler identification is GNU 11.1.0
-- The CXX compiler identification is GNU 11.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test HWY_EMSCRIPTEN
-- Performing Test HWY_EMSCRIPTEN - Failed
-- Found GTest: /usr/lib64/cmake/GTest/GTestConfig.cmake (found version "1.11.0")  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kylxbn/.cache/paru/clone/highway/src/build

and this is for clang:

-- The C compiler identification is Clang 12.0.1
-- The CXX compiler identification is Clang 12.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test HWY_EMSCRIPTEN
-- Performing Test HWY_EMSCRIPTEN - Failed
-- Found GTest: /usr/lib64/cmake/GTest/GTestConfig.cmake (found version "1.11.0")  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kylxbn/.cache/paru/clone/highway/src/build

Hopefully this is useful! Please tell me if I can provide any more details and I would happily help.

[jan-wassenberg]

Hi @kylxbn , thanks for this extensive report :)

gtest does indeed run tests during the build to "discover" what tests it should run.

I think the libjxl version is less important than which version of Highway we are using. 0.14.0 should have support for SSSE3, which your CPU supports.
There is even a fallback to no SIMD at all, but this is governed by compiler flags. If the "baseline" (what the build flags tell the compiler and Highway to assume) include SSE4.2 and AES, or even AVX2, then it will crash like this.

To debug this, we can watch for build output like:
Compiled HWY_TARGETS: AVX3 AVX2 SSE4 SSSE3 Scalar
HWY_BASELINE_TARGETS: Scalar

If we're not getting to that point because the Highway tests are failing first, we can set the
CMake variable BUILD_TESTING to OFF. The binary that prints the above output is build/third_party/highway/hwy_list_targets -
are you able to build and run that? If so, what's the output?

[kylxbn]

Thank you very much for your assistance regarding this matter!

I tried building Highway again, and I was able to see such an output!

[  6%] Building CXX object CMakeFiles/hwy_list_targets.dir/hwy/tests/list_targets.cc.o
[  8%] Linking CXX executable hwy_list_targets
make[2]: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
[  8%] Built target hwy_list_targets
make[2]: Entering directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
make[2]: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
make[2]: Entering directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
[ 10%] Building CXX object CMakeFiles/hwy.dir/hwy/aligned_allocator.cc.o
[ 13%] Building CXX object CMakeFiles/hwy.dir/hwy/nanobenchmark.cc.o
[ 15%] Building CXX object CMakeFiles/hwy.dir/hwy/targets.cc.o
[ 17%] Linking CXX static library libhwy.a
Compiled HWY_TARGETS: AVX3 AVX2 SSE4 SSSE3 Scalar
HWY_BASELINE_TARGETS: Scalar
make[2]: Leaving directory '/home/kylxbn/.cache/paru/clone/highway/src/build'
[ 17%] Built target hwy

Oops! It looks like AVX2/AVX3 is indeed being used, but unfortunately, my CPU looks like it does not support it!

In that case, I guess disabling the tests would be useless, since Highway itself is built with AVX support and so, would probably just crash once libjxl uses it?

I will be very willing to provide more information regarding this issue if needed! It might be hard to find such an old CPU like this, anyway :D

[jan-wassenberg]

@kylxbn thank you for sharing the output!
Unfortunately this rules out one simple explanation (the baseline being set too high).
This output is as it should be: we are compiling for various instruction sets, but they should only be used if actually supported by the CPU.

Can you help gather more information by running gdb highway_test? (that is the one that failed in your initial report, so it was built, but you can also use other tests if they were built in your second build attempt)

It would be helpful to know which instruction is failing (r to run, then disassemble /m 0x1234, where the latter is the address of the crash), and hopefully also a stack trace to show how we got there (bt full).

[kylxbn]

Sure! I have some learning to do (I know what GDB, disassembly, stack traces, etc. are, but I have never used GDB before so this will be a new experience :P )

I'll get back as soon as I have the data to report!

Thank you very much for being patient with me! I sincerely appreciate it.

[jan-wassenberg]

I think this may be a manifestation of google/highway#318 - we had two "enums" mistakenly set to the same value. I am changing it to an enum class to prevent such errors.

[kylxbn]

I tried to build google/highway again using the latest master branch and I am sorry to inform that the same error still persists.

I tried to run the test executable with GDB but unfortunately, it seems that cmake (or the build system) deletes the executable after building because it is failing:

[ 54%] Linking CXX executable tests/highway_test
CMake Error at /usr/share/cmake-3.21/Modules/GoogleTestAddTests.cmake:77 (message):
  Error running test executable.

    Path: '/home/kylxbn/.cache/yay/highway-git/src/build/tests/highway_test'
    Result: Illegal instruction
    Output:
      

Call Stack (most recent call first):
  /usr/share/cmake-3.21/Modules/GoogleTestAddTests.cmake:173 (gtest_discover_tests_impl)


make[2]: *** [CMakeFiles/highway_test.dir/build.make:102: tests/highway_test] Error 1
make[2]: *** Deleting file 'tests/highway_test'
make[2]: Leaving directory '/home/kylxbn/.cache/yay/highway-git/src/build'
make[1]: *** [CMakeFiles/Makefile2:368: CMakeFiles/highway_test.dir/all] Error 2
make[1]: Leaving directory '/home/kylxbn/.cache/yay/highway-git/src/build'
make: *** [Makefile:146: all] Error 2
make: Leaving directory '/home/kylxbn/.cache/yay/highway-git/src/build'
==> ERROR: A failure occurred in build().
    Aborting...
error making: highway-git

with the significant line being

make[2]: *** [CMakeFiles/highway_test.dir/build.make:102: tests/highway_test] Error 1
make[2]: *** Deleting file 'tests/highway_test'

By the way, this is in no way hurrying people or anything! Just asking for guidance.
Is there a way to prevent cmake from deleting the failing executable? Or if google/highway#318 is still not finished, of course I'd be very willing to wait!

Just thought I'd try to be helpful in any way I can.

Again, thank you very much for your patience regarding this issue!

[error256]

I've got the same issue building the main branch on Debian using "ci.sh", the file wasn't deleted, gdb says

Program received signal SIGILL, Illegal instruction.
_GLOBAL__sub_I_nanobenchmark.cc () at ../third_party/highway/hwy/nanobenchmark.cc:175
175	  asm volatile(

[jan-wassenberg]

@kylxbn thank you for checking this again! hm, that's surprising. We do have a new tool to help diagnose this. Can you share the three lines in the build output starting with "Compiled HWY_TARGETS:"?

@error256 Thanks for letting us know. The issue appears to be that your CPU does not support rdtscp. I believe that is supported since Intel Nehalem. Out of curiosity, are you running AMD or something else?
This instruction is actually important and hard to replace, but what we can do is check for it at the start of nanobenchmark and fail gracefully; I will do that.

[error256]

Out of curiosity, are you running AMD or something else?

An old Pentium D, it's more than enough for most tasks, so I still use it sometimes, but usually not for anything like building libraries, I just happened to.

[jan-wassenberg]

@error256 got it, thanks for sharing :)

[kylxbn]

@kylxbn thank you for checking this again! hm, that's surprising. We do have a new tool to help diagnose this. Can you share the three lines in the build output starting with "Compiled HWY_TARGETS:"?

Hello! Thank you for your assistance.

I tried compiling the latest Git again, but I’m not sure if it even reaches the point that all hwy targets are compiled. I’m not sure what I should show here so I just copied all output and uploaded it here privately. I hope it would be helpful!

[kylxbn]

@jan-wassenberg

But I did try going into the source directory and running run_tests.sh from there and this appears:

Compiled HWY_TARGETS: AVX3 AVX2 SSE4 SSSE3 Scalar
HWY_BASELINE_TARGETS: Scalar
Current CPU supports: SSSE3 Scalar

The complete output is available here. I had to interrupt the build because it’s using up all 4GB of my RAM and it’s causing my system to run on swap. I’d try again later, though!

[risicle]

FWIW I also experience this error when building 0.5 on a virtualized macos 10.15. Though not on bare hardware - guessing the virtualized cpu confuses the detection.

[jonsneyers]

Any update on this?

[mo271]

If there's still an issue, please re-open on their github.