vectorscan/5.4.7 fails to enable SVE2 with gcc 10.3.0

[mddirba]

The main issue is that CMakeLists.txt is appending flags for march that are already in the flag list (namely +sve2 and +sve2-bitperm). The second issue is the attempt to set mtune to be the same as march which does not work for aarch64.

git status

HEAD detached at vectorscan/5.4.7
nothing to commit, working tree clean

/usr/bin/cmake --prefix=/usr/local -DBUILD_SVE2=1 -DBUILD_SVE2_BITPERM=1 -DCMAKE_INSTALL_PREFIX=/usr/local -DFAT_RUNTIME=off -DBOOST_ROOT=/bmark/snort/0/source/boost/ -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="-mcpu=native" -DCMAKE_CXX_FLAGS="-mcpu=native" /bmark/snort/0/source/hyperscan

-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test ARCH_X86_64
-- Performing Test ARCH_X86_64 - Failed
-- Performing Test ARCH_IA32
-- Performing Test ARCH_IA32 - Failed
-- Performing Test ARCH_AARCH64
-- Performing Test ARCH_AARCH64 - Success
-- Performing Test ARCH_ARM32
-- Performing Test ARCH_ARM32 - Failed
-- Performing Test ARCH_PPC64EL
-- Performing Test ARCH_PPC64EL - Failed
-- Build type RELEASE
-- using release build
-- Boost version: 1.77.0
-- Found PythonInterp: /usr/bin/python (found version "2.7.18")
-- Build date: 2022-05-06
-- Building static libraries
-- gcc version 10.3.0
CMake Warning at CMakeLists.txt:187 (message):
Something went wrong determining gcc tune:
-mtune=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres
not valid, falling back to -mtune=native

-- ARCH_C_FLAGS :
-- ARCH_CXX_FLAGS :
-- g++ version 10.3.0
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for C++ include arm_neon.h
-- Looking for C++ include arm_neon.h - found
-- Looking for C++ include arm_sve.h
-- Looking for C++ include arm_sve.h - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for _aligned_malloc
-- Looking for _aligned_malloc - not found
-- Performing Test HAS_C_HIDDEN
-- Performing Test HAS_C_HIDDEN - Success
-- Performing Test HAS_CXX_HIDDEN
-- Performing Test HAS_CXX_HIDDEN - Success
-- Looking for _LIBCPP_VERSION
-- Looking for _LIBCPP_VERSION - not found
-- Performing Test HAVE_NEON
-- Performing Test HAVE_NEON - Success
-- Performing Test HAVE_SVE2_BITPERM
-- Performing Test HAVE_SVE2_BITPERM - Failed
-- Performing Test HAVE_SVE2
-- Performing Test HAVE_SVE2 - Failed
-- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED
-- Performing Test HAVE_CC_BUILTIN_ASSUME_ALIGNED - Success
-- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED
-- Performing Test HAVE_CXX_BUILTIN_ASSUME_ALIGNED - Success
-- Performing Test HAVE__BUILTIN_CONSTANT_P
-- Performing Test HAVE__BUILTIN_CONSTANT_P - Success
-- Performing Test C_FLAG_Wvla
-- Performing Test C_FLAG_Wvla - Success
-- Performing Test C_FLAG_Wpointer_arith
-- Performing Test C_FLAG_Wpointer_arith - Success
-- Performing Test C_FLAG_Wstrict_prototypes
-- Performing Test C_FLAG_Wstrict_prototypes - Success
-- Performing Test C_FLAG_Wmissing_prototypes
-- Performing Test C_FLAG_Wmissing_prototypes - Success
-- Performing Test CXX_FLAG_Wvla
-- Performing Test CXX_FLAG_Wvla - Success
-- Performing Test CXX_FLAG_Wpointer_arith
-- Performing Test CXX_FLAG_Wpointer_arith - Success
-- Performing Test CC_SELF_ASSIGN
-- Performing Test CC_SELF_ASSIGN - Failed
-- Performing Test CXX_SELF_ASSIGN
-- Performing Test CXX_SELF_ASSIGN - Failed
-- Performing Test CC_PAREN_EQUALITY
-- Performing Test CC_PAREN_EQUALITY - Failed
-- Performing Test CXX_UNUSED_CONST_VAR
-- Performing Test CXX_UNUSED_CONST_VAR - Success
-- Performing Test CXX_IGNORED_ATTR
-- Performing Test CXX_IGNORED_ATTR - Success
-- Performing Test CXX_REDUNDANT_MOVE
-- Performing Test CXX_REDUNDANT_MOVE - Success
-- Performing Test CXX_WEAK_VTABLES
-- Performing Test CXX_WEAK_VTABLES - Failed
-- Performing Test CXX_MISSING_DECLARATIONS
-- Performing Test CXX_MISSING_DECLARATIONS - Success
-- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS
-- Performing Test CXX_UNUSED_LOCAL_TYPEDEFS - Success
-- Performing Test CXX_WUNUSED_VARIABLE
-- Performing Test CXX_WUNUSED_VARIABLE - Success
-- Performing Test CC_STRINGOP_OVERFLOW
-- Performing Test CC_STRINGOP_OVERFLOW - Success
-- Building for current host CPU: -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm -mtune=native
-- Looking for mmap
-- Looking for mmap - not found
-- Doxygen not found, unable to generate API reference
-- Sphinx not found, unable to generate developer reference
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'libpcre>=8.41'
-- Found libpcre, version 8.45
-- PCRE version 8.41 or above
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Could NOT find Threads (missing: Threads_FOUND)
-- Checking for module 'sqlite3'
-- Found sqlite3, version 3.31.1
-- Performing Test SQLITE_VERSION_OK
-- Performing Test SQLITE_VERSION_OK - Failed
CMake Error at cmake/sqlite3.cmake:34 (message):
sqlite3 is broken from 3.8.7 to 3.8.10 - please find a working version
Call Stack (most recent call first):
tools/hsbench/CMakeLists.txt:1 (include)

-- Configuring incomplete, errors occurred!
See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeOutput.log".
See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeError.log".

tail -n 20 CMakeFiles/CMakeError.log

Change Dir: /bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_4ea79/fast && /usr/bin/make -f CMakeFiles/cmTC_4ea79.dir/build.make CMakeFiles/cmTC_4ea79.dir/build
make[1]: Entering directory '/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_4ea79.dir/src.c.o
/usr/bin/cc -mcpu=native -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm -mtune=native -DSQLITE_VERSION_OK -o CMakeFiles/cmTC_4ea79.dir/src.c.o -c /bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp/src.c
Assembler messages:
Error: must specify extensions to add before specifying those to remove
Error: unrecognized option -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm
make[1]: *** [CMakeFiles/cmTC_4ea79.dir/build.make:66: CMakeFiles/cmTC_4ea79.dir/src.c.o] Error 1
make[1]: Leaving directory '/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_4ea79/fast] Error 2

Source file was:
#include <sqlite3.h>
#if SQLITE_VERSION_NUMBER >= 3008007 && SQLITE_VERSION_NUMBER < 3008010
#error broken sqlite
#endif
int main() {return 0;}

[mddirba]

My naive recommendation is just to remove the code adding the flags, as well as failing the build if SVE2 was requested but it is not available.

git diff

diff --git a/CMakeLists.txt b/CMakeLists.txt
index cb4ba80..30a4321 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -228,16 +228,6 @@ if (ARCH_IA32 OR ARCH_X86_64)
     endif()
 endif()

-if (ARCH_AARCH64)
-    if (BUILD_SVE2_BITPERM)
-        set(GNUCC_ARCH "${GNUCC_ARCH}+sve2-bitperm")
-    elseif (BUILD_SVE2)
-        set(GNUCC_ARCH "${GNUCC_ARCH}+sve2")
-    elseif (BUILD_SVE)
-        set(GNUCC_ARCH "${GNUCC_ARCH}+sve")
-    endif ()
-endif(ARCH_AARCH64)
-


 message(STATUS "ARCH_C_FLAGS   : ${ARCH_C_FLAGS}")
 message(STATUS "ARCH_CXX_FLAGS : ${ARCH_CXX_FLAGS}")
 
diff --git a/cmake/arch.cmake b/cmake/arch.cmake
index 29c39b4..02fc8d4 100644
--- a/cmake/arch.cmake
+++ b/cmake/arch.cmake
@@ -188,6 +188,15 @@ else (NOT FAT_RUNTIME)
     if (ARCH_PPPC64EL AND NOT HAVE_VSX)
         message(FATAL_ERROR "VSX support required for Power support")
     endif ()
+    if (BUILD_SVE1 AND NOT HAVE_SVE1)
+           message(FATAL_ERROR "SVE1 support not avaliable on this system")
+    endif ()
+    if (BUILD_SVE2 AND NOT HAVE_SVE2)
+           message(FATAL_ERROR "SVE2 support not avaliable on this system")
+    endif ()
+    if (BUILD_SVE2_BITPERM AND NOT HAVE_SVE2_BITPERM)
+           message(FATAL_ERROR "SVE2_BITPERM support not avaliable on this system")
+    endif ()

 endif ()

[mddirba]

The third issue is that the march flag is not checked after the new flags are appended and the failure appears to be from the sqlite version check.

[markos]

@mddirba right, this must be on actual hardware, my tests have been only on qemu, where SVE2 autodetection works as I'm building it regularly. I'm not entirely happy with features autodetection in general and I would be more than happy to replace it with something more robust. I will try your suggestion and will probably have a new release soon.

[abondarev84]

any updates?

[markos]

Hi, unfortunately SVE2 support will not change soon, but we will get a Graviton3 system soon so such problems should be fixed and SVE support should be improved.

[abondarev84]

I've been trying on graviton3 ec2 instance and it seems like sve is also not being utilized.
from running perf and disassembly, doesn't look like sve is being utilized when building with -DBUILD_SVE=1.
Is this what you're referring to in the "SVE support should be improved" part of your reply?

[markos]

keep in mind that SVE support has only been added in a couple specific modules as a PoC and not everywhere. I have done multiple tests on qemu and SVE does work there. Most likely SVE detection and enablement is done incorrectly on actual hardware, hence the problems.

[markos]

Hi @mddirba, SVE support has just been enabled in recent merged PR #125. Would you mind testing current develop branch on SVE2 hardware, as I'm preparing a new release and it would be best to include any minor fixes for SVE2 now, otherwise it will have to be fixed afterwards.

[mddirba]

CMakeLists.txt still attempts to append to -march so this code will not compile on N2.

if (ARCH_AARCH64)
    if (BUILD_SVE2_BITPERM AND NOT SVE2_BITPERM_FOUND)
        set(GNUCC_ARCH "${GNUCC_ARCH}+sve2-bitperm")
    elseif (BUILD_SVE2 AND NOT SVE2_FOUND)
        set(GNUCC_ARCH "${GNUCC_ARCH}+sve2")
    elseif (BUILD_SVE AND NOT SVE_FOUND)
        set(GNUCC_ARCH "${GNUCC_ARCH}+sve")
    endif ()
endif(ARCH_AARCH64)

/usr/bin/cmake --prefix=/usr/local -DBUILD_SVE2=1 -DBUILD_SVE2_BITPERM=1 -DCMAKE_INSTALL_PREFIX=/usr/local -DFAT_RUNTIME=off -DBOOST_ROOT=/bmark/snort/0/source/boost/ -DCMAKE_BUILD_TYPE=Release /bmark/snort/0/source/hyperscan '-DCMAKE_C_FLAGS=-m

arch=native -mtune=native' '-DCMAKE_CXX_FLAGS=-march=native -mtune=native'
-- Build type RELEASE
-- using release build
-- Boost version: 1.77.0
-- Build date: 2022-09-19
-- Building static libraries
-- gcc version 10.3.0
-- ARCH_FLAG 'march' '', TUNE_FLAG 'mtune' ''
-- ARCH_FLAG 'march' 'armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres', TUNE_FLAG 'mtune' 'neoverse-n2'
-- gcc will tune for armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres, neoverse-n2
-- ARCH_C_FLAGS :
-- ARCH_CXX_FLAGS :
-- g++ version 10.3.0
CMake Error at CMakeLists.txt:371 (message):
arm_sve.h is required to build for SVE.

-- Configuring incomplete, errors occurred!
See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeOutput.log".
See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeError.log".
root@cn106a0crbr1-438:~# FLAG="-mcpu=neoverse-n2" /usr/bin/cmake --prefix=/usr/local -DBUILD_SVE2=1 -DBUILD_SVE2_BITPERM=1 -DCMAKE_INSTALL_PREFIX=/usr/local -DFAT_RUNTIME=off -DBOOST_ROOT=/bmark/snort/0/source/boost/ -DCMAKE_BUILD_TYPE=Release /bmark/snort/0/source/hype
rscan "-DCMAKE_C_FLAGS=${FLAG}" "-DCMAKE_CXX_FLAGS=${FLAG}"
-- Build type RELEASE
-- using release build
-- Boost version: 1.77.0
-- Build date: 2022-09-19
-- Building static libraries
-- gcc version 10.3.0
-- ARCH_FLAG 'march' '', TUNE_FLAG 'mtune' ''
-- ARCH_FLAG 'march' 'armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres', TUNE_FLAG 'mtune' 'neoverse-n2'
-- gcc will tune for armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres, neoverse-n2
-- ARCH_C_FLAGS :
-- ARCH_CXX_FLAGS :
-- g++ version 10.3.0
CMake Error at CMakeLists.txt:371 (message):
arm_sve.h is required to build for SVE.

-- Configuring incomplete, errors occurred!
See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeOutput.log".
See also "/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeError.log".

tail -n21 /bmark/snort/0/source/hyperscan/CMakeFiles/CMakeError.log

Source file was:
#if !defined(PPC64) && !(defined(LITTLE_ENDIAN) && defined(VSX))
#error not ppc64el
#endif
int main(void) { return 0; }
Determining if the include file arm_sve.h exists failed with the following output:
Change Dir: /bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_cc705/fast && /usr/bin/make -f CMakeFiles/cmTC_cc705.dir/build.make CMakeFiles/cmTC_cc705.dir/build
make[1]: Entering directory '/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_cc705.dir/CheckIncludeFile.cxx.o
/usr/bin/c++ -march=native -mtune=native -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm -mtune=neoverse-n2 -o CMakeFiles/cmTC_cc705.dir/CheckIncludeFile.cxx.o -c /bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp/CheckIncludeFile.cxx
Assembler messages:
Error: must specify extensions to add before specifying those to remove
Error: unrecognized option -march=armv8.5-a+crypto+rcpc+sve2-sm4+sve2-aes+sve2-sha3+sve2-bitperm+nopredres+sve2-bitperm
make[1]: *** [CMakeFiles/cmTC_cc705.dir/build.make:66: CMakeFiles/cmTC_cc705.dir/CheckIncludeFile.cxx.o] Error 1
make[1]: Leaving directory '/bmark/snort/0/source/hyperscan/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_cc705/fast] Error 2

[markos]

Right, I'm afraid I cannot guess what the problem is, I expected it to behave similarly to SVE, but apparently there is a catch. One suggestion, could you please try replacing that block with this:

if (ARCH_AARCH64)
    if (BUILD_SVE2_BITPERM AND NOT SVE2_BITPERM_FOUND)
        set(GNUCC_ARCH "${GNUCC_ARCH}+sve2-bitperm")
    endif()
    if (BUILD_SVE2 AND NOT SVE2_FOUND)
        set(GNUCC_ARCH "${GNUCC_ARCH}+sve2")
    endif()
    if (BUILD_SVE AND NOT SVE_FOUND)
        set(GNUCC_ARCH "${GNUCC_ARCH}+sve")
    endif ()
endif(ARCH_AARCH64)```

and retry?

[abondarev84]

It is missing the fix that was included into the merged pr in the develop branch.

It won't have the additional code that searches for the flags in the auto detected arch flag so the failure is the same as previously I believe.
Just that section alone won't fix it.

[markos]

This has been fixed in #131 . Enabling SVE2 on eg. Yitian:

$ cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SVE=on -DBUILD_SVE2=on -DBUILD_SVE2_BITPERM=on ..