llvm_mode and OpenBSD

[hexcoder-]

When trying to build llvm_mode in OpenBSD, it does not work right out of the box.

clang-local does not search for llvm libs in their installed directories by default (for security purposes).

So when trying to build some test program with afl-clang-fast it fails:

[*] Testing the CC wrapper and instrumentation output...
unset AFL_USE_ASAN AFL_USE_MSAN AFL_INST_RATIO; AFL_QUIET=1 AFL_PATH=. AFL_CC=clang ../afl-clang-fast -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH="/usr/local/lib/afl" -DBIN_PATH="/usr/local/bin" -DVERSION="2.52b" ../test-instr.c -o test-instr
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm10ModulePass17assignPassManagerERNS_7PMStackENS_15PassManagerTypeE'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm10ModulePassD2Ev'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm24DisableABIBreakingChecksE'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm4Pass13releaseMemoryEv'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm4Pass17dumpPassStructureEj'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm4Pass18getAsImmutablePassEv'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm4Pass18getAsPMDataManagerEv'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm4Pass18preparePassManagerERNS_7PMStackE'
clang:../afl-llvm-pass.so: undefined symbol '_ZN4llvm4Pass26getAdjustedAnalysisPointerEPKv'
clang:../afl-llvm-pass.so: undefined symbol '_ZNK4llvm10ModulePass17createPrinterPassERNS_11raw_ostreamERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE'
clang:../afl-llvm-pass.so: undefined symbol '_ZNK4llvm10ModulePass27getPotentialPassManagerTypeEv'
clang:../afl-llvm-pass.so: undefined symbol '_ZNK4llvm4Pass11getPassNameEv'
clang:../afl-llvm-pass.so: undefined symbol '_ZNK4llvm4Pass14verifyAnalysisEv'
clang:../afl-llvm-pass.so: undefined symbol '_ZNK4llvm4Pass16getAnalysisUsageERNS_13AnalysisUsageE'
clang:../afl-llvm-pass.so: undefined symbol '_ZNK4llvm4Pass5printERNS_11raw_ostreamEPKNS_6ModuleE'
error: unable to load plugin '../afl-llvm-pass.so': 'Cannot load specified object'
gmake: *** [Makefile:98: test_build] Error 1

This is from the old afl-2.52b package, but applies to our version as well.
So, for OpenBSD we have to set the llvm lib path explicitly.

I will investigate later and report back.

[hexcoder-]

Changing
CLANG_LFL = $(LLVM_CONFIG) --ldflags $(LDFLAGS)
to
CLANG_LFL = $(LLVM_CONFIG) --ldflags $(LLVM_CONFIG) --libdir/libLLVM.so.0.0 $(LDFLAGS)
avoids the linking error.

But instrumentation does not work yet.

[hexcoder-]

Hmm, it seems clang under OpenBSD 6.5 is restricted and does not support -Xclang options. Clang seems to accept them, but ignores them. There is no opt either :-(. That would mean stock afl would not work in standard llvm_mode. Can anybody using OpenBSD confirm?

[CaBeckmann]

Apparently there are two slightly different versions of LLVM on OpenBSD. One from the main tree in /usr/bin and /usr/lib and one from the ports tree in /usr/local/bin and /usr/local/lib. I tried the following changes to the Makefile on my machine:

diff --git a/llvm_mode/Makefile b/llvm_mode/Makefile
index 160a8fe..86aa085 100644
--- a/llvm_mode/Makefile
+++ b/llvm_mode/Makefile
@@ -25,7 +25,12 @@ BIN_PATH     = $(PREFIX)/bin
 
 VERSION     = $(shell grep '^\#define VERSION ' ../config.h | cut -d '"' -f2)
 
-LLVM_CONFIG ?= llvm-config
+ifeq "$(shell uname)" "OpenBSD"
+  LLVM_CONFIG ?= $(BIN_PATH)/llvm-config
+else
+  LLVM_CONFIG ?= llvm-config
+endif
+
 LLVMVER  = $(shell $(LLVM_CONFIG) --version)
 LLVM_UNSUPPORTED = $(shell $(LLVM_CONFIG) --version | egrep -q '^[12]|^3\.0|^1[0-9]' && echo 1 || echo 0 )
 LLVM_MAJOR = ($shell $(LLVM_CONFIG) --version | sed 's/\..*//')
@@ -61,7 +66,7 @@ ifeq "$(shell uname)" "Darwin"
 endif
 
 ifeq "$(shell uname)" "OpenBSD"
-  CLANG_LFL += `$(LLVM_CONFIG) --libdir`/libLLVM.so.0.0
+  CLANG_LFL += `$(LLVM_CONFIG) --libdir`/libLLVM.so
 endif
 
 # We were using llvm-config --bindir to get the location of clang, but
@@ -69,8 +74,13 @@ endif
 # probably better.
 
 ifeq "$(origin CC)" "default"
-  CC         = clang
-  CXX        = clang++
+  ifeq "$(shell uname)" "OpenBSD"
+    CC         = $(BIN_PATH)/clang
+    CXX        = $(BIN_PATH)/clang++
+  else
+    CC         = clang
+    CXX        = clang++
+  endif
 endif
 
 # sanity check.

After this you still get warnings during compilation, but the instrumentation seems to be working:

[*] Testing the CC wrapper and instrumentation output...
unset AFL_USE_ASAN AFL_USE_MSAN AFL_INST_RATIO; AFL_QUIET=1 AFL_PATH=. AFL_CC=/usr/local/bin/clang AFL_LLVM_LAF_SPLIT_SWITCHES=1 AFL_LLVM_LAF_TRANSFORM_COMPARES=1 AFL_LLVM_LAF_SPLIT_COM
PARES=1 ../afl-clang-fast -O3 -funroll-loops -Wall -D_FORTIFY_SOURCE=2 -g -Wno-pointer-sign -DAFL_PATH=\"/usr/local/lib/afl\" -DBIN_PATH=\"/usr/local/bin\" -DVERSION=\"++2.53d\" ../test
-instr.c -o test-instr 
clang-7:../split-switches-pass.so: /usr/local/bin/clang-7 : WARNING: symbol(_ZTVNSt3__110__function6__funcIPFvRKN4llvm18PassManagerBuilderERNS2_6legacy15PassManagerBaseEENS_9allocatorIS
A_EES9_EE) size mismatch, relink your program
clang-7:../compare-transform-pass.so: /usr/local/bin/clang-7 : WARNING: symbol(_ZTVNSt3__110__function6__funcIPFvRKN4llvm18PassManagerBuilderERNS2_6legacy15PassManagerBaseEENS_9allocato
rISA_EES9_EE) size mismatch, relink your program
clang-7:../split-compares-pass.so: /usr/local/bin/clang-7 : WARNING: symbol(_ZTVNSt3__110__function6__funcIPFvRKN4llvm18PassManagerBuilderERNS2_6legacy15PassManagerBaseEENS_9allocatorIS
A_EES9_EE) size mismatch, relink your program
clang-7:../afl-llvm-pass.so: /usr/local/bin/clang-7 : WARNING: symbol(_ZTVNSt3__110__function6__funcIPFvRKN4llvm18PassManagerBuilderERNS2_6legacy15PassManagerBaseEENS_9allocatorISA_EES9
_EE) size mismatch, relink your program
Rewriting 1 switch statements 
switch: 2 cases 8 bit
Running split-compare-pass 64
Running split-compare-pass 32
Running split-compare-pass 16
../afl-showmap -m none -q -o .test-instr0 ./test-instr < /dev/null
echo 1 | ../afl-showmap -m none -q -o .test-instr1 ./test-instr
[+] All right, the instrumentation seems to be working!

[hexcoder-]

Thanks a lot, I will have a look soon. Two versions of llvm was a bit confusing for me.
What version of OpenBSD are you running?

[CaBeckmann]

I am running OpenBSD 6.5 with the llvm-7.0.1p9 port/package installed.
I was confused by the different versions as well. A colleague told me that LLVM on OpenBSD started out as a port. Later they added LLVM to base OpenBSD, but the port still exists. And apparently it has more features enabled and is in part newer than the version in base.

[hexcoder-]

Thanks, I can reproduce it. The linker errors are mysterious, though. But it is certainly a progress.

[hexcoder-]

I adapted your solution and added a check for the required package with commit 3bfd88aa . Thanks.