Ask about Linux ARM64/aarch64 support

[liusheng]

Hi,

I am working on promoting some opensource projects running on ARM64/aarch64 Linux servers, I want to know whether projects using ThreadSantizer and AddressSanitizer can running on ARM64/aarch64 Linux servers, I have found that the support matrix of AddressSantizer for differents OS and arch here: https://github.com/google/sanitizers/wiki/AddressSanitizer#introduction
It looks like the AddressSantizer support Android on ARM64, but doesn't support Linux on ARM64, does that means AddressSantizer cannot run on ARM64 Linux server ?

Thanks a lot!

[kiwik]

I have same question, we want to enable ThreadSantizer and AddressSanitizer for big data project apache/kudu on ARM64 Linux server, but face lots of issues.

https://github.com/apache/kudu#running-tests-with-the-clang-addresssanitizer-enabled

[kcc]

AddressSantizer does work on Aarch64 Linux. Please try and report specific issues.

Please note that on Aarch64 Linux (with a new enough kernel) a better option might be the new tool call HWASAN (a.k.a. ASAN v2): https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html
The main benefit is drastically reduced memory overhead.

[liusheng]

Hi @kcc ,
Thanks for your clarification, we are working on building and testing Kudu on aarch64 server, now, we can sucessfully build the Kudu with TSAN enabled, but all the unittests cannot run.

I am testing on Ubuntu 18.04 server

root@kudu-asan2:/opt/kudu/build/asan# uname -a
Linux kudu-asan2 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:10 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

and use the llvm-9.0.0. all the compiled tests cannot run and fail with following error:

root@kudu-asan2:/opt/kudu/build/asan# bin/kudu-ts-cli-test
AddressSanitizer:DEADLYSIGNAL
=================================================================
==14360==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0xffffdfed61b0 sp 0xffffdfed61b0 T0)
==14360==Hint: pc points to the zero page.
==14360==The signal is caused by a READ memory access.
==14360==Hint: address points to the zero page.

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>) 
==14360==ABORTING

could you please give me any suggestion?

[kcc]

I would suggest you to start with small test programs.
Do they work on your machine with your version of clang?

[kiwik]

I test it in last week, this's more clear environment info.
We enable ASAN to build apache/kudu and run testing on ARM64 server.
Kudu build clang from source code, and version is 9.0.0

ubuntu@ubuntu:~/workspace/github.com/apache/kudu$ uname -a                                                                                                                                                         
Linux ubuntu 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:10:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux                                        
ubuntu@ubuntu:~/workspace/github.com/apache/kudu$ cat /etc/issue      
Ubuntu 18.04.4 LTS \n \l                                                                                                                                         
ubuntu@ubuntu:~/workspace/github.com/apache/kudu/build/llvmasan$ ../../thirdparty/clang-toolchain/bin/clang++ --version                          
clang version 9.0.0 (https://github.com/apache/kudu.git de3c338d0acedc2b3f7ee36b9e5d0c644daf7b9d)                                                                
Target: aarch64-unknown-linux-gnu                                                                                                                                                                                  
Thread model: posix                                                                                                                                                                                                
InstalledDir: /home/ubuntu/workspace/github.com/apache/kudu/build/llvmasan/../../thirdparty/clang-toolchain/bin

I find the issue is related with compile option and platform (x86_64 and ARM64).

If I enable dynamic linking -DKUDU_LINK=dynamic and ASAN -DKUDU_USE_ASAN=1 in Kudu building,
x86_64 is OK, but for ARM64, all of test cases throw the following error.

ubuntu@ubuntu:~/workspace/github.com/apache/kudu/build/asan$./bin/example-test 
AddressSanitizer:DEADLYSIGNAL 
================================================================= 
==20451==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0xffffd0853a20 sp 0xffffd0853a20 T0) 
==20451==Hint:pcpointstothezeropage. 
==20451==The signal is caused by a READ memoryaccess. 
==20451==Hint: address points to the zero page.
AddressSanitizercannotprovideadditionalinfo. 
SUMMARY: AddressSanitizer: SEGV (<unknown module>) 
==20451==ABORTING

If I enable static linking -DKUDU_LINK=static and ASAN -DKUDU_USE_ASAN=1 in Kudu building,
Both x86_64 and ARM64 are OK.

My example-test is a very simple test case, like this:

ubuntu@ubuntu:~/workspace/github.com/apache/kudu$ cat src/kudu/testing/example-test.cc                                                                                                                             
#include <gtest/gtest.h>                                                                                                                                                                                           
                                                                                                                                                                                                                   
TEST(TestExample, Test1) {                                                                                                                                                                                         
  ASSERT_TRUE(0 == 0);                                                                                                                                                                                             
}     

[kiwik]

I enable TSAN for Kudu on ARM64 server, face the same issue like ASAN in above comment, looks like ASAN and TSAN with dynamic linking on ARM64 will casue the following error, but with static linking is OK.

ubuntu@ubuntu:~/workspace/github.com/apache/kudu/build/asan$./bin/example-test 
AddressSanitizer:DEADLYSIGNAL 
================================================================= 
==20451==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0xffffd0853a20 sp 0xffffd0853a20 T0) 
==20451==Hint:pcpointstothezeropage. 
==20451==The signal is caused by a READ memoryaccess. 
==20451==Hint: address points to the zero page.
AddressSanitizercannotprovideadditionalinfo. 
SUMMARY: AddressSanitizer: SEGV (<unknown module>) 
==20451==ABORTING

[kcc]

Please provide a reproducer w/o kudu or gtest.

[kiwik]

NOTES: We are working on Kudu ARM64 support and face the issue, please reproduct issue on specifc branch of forked project (https://github.com/liusheng/kudu.git), some code have not yet been merged into Kudu master now

Reproduct steps for Kudu:

0. ARM server (4C 8G RAM 100G disk or more, I use an virtual machine on public cloud)
1. git clone -b build-on-aarch64 https://github.com/liusheng/kudu.git
2. cd {KUDU_SRC}
3. mkdir -p build/debug
4. ./thirdparty/build-if-necessary.sh
5. cd build/debug
6. use following command to enable ASAN or TSAN

    CC=../../thirdparty/clang-toolchain/bin/clang \
    CXX=../../thirdparty/clang-toolchain/bin/clang++ \
    cmake -DCMAKE_BUILD_TYPE=debug -DKUDU_USE_ASAN=1 ../..

7. make -j4 && ctest -j4

More details please check Kudu document https://github.com/apache/kudu#kudu-developer-documentation
Related bug at Kudu side https://issues.apache.org/jira/browse/KUDU-3101

[kiwik]

Kudu community have merged ARM supported into master, so you can try to reproduce the issue in Kudu master branch https://github.com/apache/kudu