Android x86 build failed due to _mm_crc32_u8 requires sse4.1

[chongchaoyu]

Crypto++ Issue Report

I use ndkbuild to build Crypto++ 6.1.0 and Crypto++ 7.0.0 to a static lib, but both of them failed, the error is '_mm_crc32_u8' requires target feature 'sse4.1', but would be inlined into function 'CRC32C_Update_SSE42' that is compiled without support for 'sse4.1', and I have uploaded Android.mk and Application.mk.mks.zip, the detail build log is
ndkbuild.txt

The detail information is:

• macOS High Sierra, 10.13.2(71c88)

• cpu:
  machdep.cpu.brand_string: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz
  machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC POPCNT AES PCID XSAVE OSXSAVE TSCTMR AVX1.0 RDRAND F16C

• Crypto++ 6.1.0 and Crypto++ 7.0.0

• Use ndkbuild, Android.mk and Application.mk

• ndk version is r16b, GNU Make 3.81

• Error message is:
  /Users/mikeyu/work/projects/vega/source/yecha/client/frameworks/runtime-src/Classes/thirdparty/pre/jni/cryptopp700/crc-simd.cpp:148:13: error: always_inline function '_mm_crc32_u8' requires target feature 'sse4.1', but would be inlined into
  function 'CRC32C_Update_SSE42' that is compiled without support for 'sse4.1'
  c = _mm_crc32_u8(c, *s);
  ^
  /Users/mikeyu/work/projects/vega/source/yecha/client/frameworks/runtime-src/Classes/thirdparty/pre/jni/cryptopp700/crc-simd.cpp:151:13: error: always_inline function '_mm_crc32_u32' requires target feature 'sse4.1', but would be inlined
  into function 'CRC32C_Update_SSE42' that is compiled without support for 'sse4.1'
  c = _mm_crc32_u32(c, *(const word32 )(void)s);
  ^
  /Users/mikeyu/work/projects/vega/source/yecha/client/frameworks/runtime-src/Classes/thirdparty/pre/jni/cryptopp700/crc-simd.cpp:154:13: error: always_inline function '_mm_crc32_u8' requires target feature 'sse4.1', but would be inlined into
  function 'CRC32C_Update_SSE42' that is compiled without support for 'sse4.1'
  c = _mm_crc32_u8(c, *s);
  ^
  3 errors generated.
  make: *** [/Users/mikeyu/work/projects/vega/source/yecha/client/frameworks/runtime-src/Classes/thirdparty/pre/obj/local/x86/objs/cryptopp_static/cryptopp700/crc-simd.o] Error 1

[noloader]

Thanks @chongchaoyu.

Yeah, that sounds about right. Here are some additional resources that may help you with Android.mk. Item (3) is one of the more relevant topics.

1. PR #3, Add Android.mk to build using android NDK by @cawka and @Deadpikle. It is a little dated now. We held off on the PR because of the unusual directory structure required by Android. I would have merged it years ago if Android.mk could reside in root with the makefile.

2. Running make sources will list the files needed by the library proper (libcryptopp.a) and the test suite (cryptest.exe). It should allow you to group files properly and remove test suite files if unneeded.

3. BASE+SIMD wiki page list the files that need arch flags and the arch flags. It was updated recently for the upcoming release.

4. Android (Command Line) wiki page which provides make-based instructions. The procedures use GNUmakefile-cross because we need extra flags for the platforms. We use this to test Android builds on Travis CI.

5. Android Activity wiki page which provides information on using the shared object in an Android Java program.

6. cryptopp-autotools and cryptopp-cmake GitHubs. They are unofficial but we help maintain them. I believe both have gaps for Android so they probably won't help you.

[chongchaoyu]

@noloader Thank you for your response, I will try it later with your resources.
BTW, I have downgraded to 5.6.5, and it works well.

[noloader]

@chongchaoyu

One other thing... The current config.h has the following. It will part of the upcoming 7.1 release.

// Fixup Android and SSE, Crypto. It may be enabled based on compiler version.
#if (defined(__ANDROID__) || defined(ANDROID))
# if (CRYPTOPP_BOOL_X86)
#  undef CRYPTOPP_SSE41_AVAILABLE
#  undef CRYPTOPP_SSE42_AVAILABLE
#  undef CRYPTOPP_CLMUL_AVAILABLE
#  undef CRYPTOPP_AESNI_AVAILABLE
#  undef CRYPTOPP_SHANI_AVAILABLE
# endif
# if (CRYPTOPP_BOOL_X64)
#  undef CRYPTOPP_CLMUL_AVAILABLE
#  undef CRYPTOPP_AESNI_AVAILABLE
#  undef CRYPTOPP_SHANI_AVAILABLE
# endif
#endif

Older versions of the library don't have the block above. Older versions required a user add the define on the command line. That's a pain in the butt. Since Android has a stable hardware requirements list, we now do it for users.

[noloader]

@chongchaoyu, @cawka, @Deadpikle,

What do you guys think about setting up cryptoppp-android on my GitHub like cryptoppp-autotools and cryptoppp-cmake? The GitHub will include the Android.mk file, and we can run it like cryptoppp-autotools and cryptoppp-cmake.

The model seems to be working well with cryptoppp-cmake. We setup the GitHub and others contribute to it. Folks with Cmake expertise are made collaborators so we don't slow the process down. In practice folks like @abdes and @jcfr just check-in when they see fit. There is no patch review process.

We help maintain it like when we add crc-simd.cpp with special flags. When new source files hit with special needs we handle it or open a bug report so folks with more experience can tend to it.

I think it really benefits users because users have a centralized copy of the most up-to-date files without using random sources on the web.

[chongchaoyu]

@noloader @cawka @Deadpikle I think it is a good idea, and it can benefit users a lot and save much time. I am not a skillful c++ programmer, I spent almost two days to compile sources to static lib, suffering many problems, such as version, c++_static/gnustl_static, so painful:(

[noloader]

@chongchaoyu,

I believe the cryptopp-android project files are ready for consumption. I also added a wiki page at Android.mk (Command Line).

While you are working through the Android build system you might want to look at Wrapper DLL. It is probably the way I would go with design if I had to do something on Android.