Erros when generating distance file

[DigOrDog]

SETUP:

1. OS:ubuntu 20.04
2. RAM:250G
3. aflgo:commit 1b81e7c
4. Fuzz Taregt:llvm-dis(llvm-15.0.4)
5. BBtargets.txt

llvm/tools/llvm-dis/llvm-dis.cpp:108
llvm/tools/llvm-dis/llvm-dis.cpp:109
llvm/tools/llvm-dis/llvm-dis.cpp:110

Steps To Reproduce

1. run AFLGo building script to configure.
2. download llvm-15.0.4

# download and tar
wget https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-15.0.4.tar.gz
tar -zxvf llvmorg-15.0.4.tar.gz
mv llvm-project-llvmorg-15.0.4 llvm-project
export SUBJECT=$PWD/llvm-project

3.Set targets

# Setup directory containing all temporary files
mkdir temp
export TMP_DIR=$PWD/temp

BBtargets.txt
llvm/tools/llvm-dis/llvm-dis.cpp:108
llvm/tools/llvm-dis/llvm-dis.cpp:109
llvm/tools/llvm-dis/llvm-dis.cpp:110

# Print extracted targets. 
echo "Targets:"
cat $TMP_DIR/BBtargets.txt

4. generate CG and intra-procedural CFGs from the subject.

# Set aflgo-instrumenter
export CC=$AFLGO/instrument/aflgo-clang
export CXX=$AFLGO/instrument/aflgo-clang++

# Set aflgo-instrumentation flags
export COPY_CFLAGS=$CFLAGS
export COPY_CXXFLAGS=$CXXFLAGS
export ADDITIONAL="-targets=$TMP_DIR/BBtargets.txt -outdir=$TMP_DIR -flto -fuse-ld=gold -Wl,-plugin-opt=save-temps"
export CFLAGS="$CFLAGS $ADDITIONAL"
export CXXFLAGS="$CXXFLAGS $ADDITIONAL"

export LDFLAGS=-lpthread
# Build only LLVM
pushd llvm-project
mkdir build
pushd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release 
-DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX"
-DCMAKE_CXX_FLAGS="$ADDITIONAL" -DCMAKE_C_FLAGS="$ADDITIONAL" 
-DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS" 
../llvm
ninja

5. Generate distance file. Firstly we need to clean up BBnames.txt and BBcalls.txt, otherwise distance_calculator may fail.

cat $TMP_DIR/BBnames.txt | grep -v "^$"| rev | cut -d: -f2- | rev | sort | uniq > $TMP_DIR/BBnames2.txt && mv $TMP_DIR/BBnames2.txt $TMP_DIR/BBnames.txt

cat $TMP_DIR/BBcalls.txt | grep -Ev "^[^,]*$|^([^,]*,){2,}[^,]*$"| sort | uniq > $TMP_DIR/BBcalls2.txt && mv $TMP_DIR/BBcalls2.txt $TMP_DIR/BBcalls.txt

$AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis

Expected Behavoirs:

$AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis perform well 

Actual Behavoirs:

(aflgo) h3d@h3d-AS-4124GS-TNR:~/HUCHENG/project/targets/llvm-project/build$ $AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis
(0) Constructing CG for /home/h3d/HUCHENG/project/targets/llvm-project/build/bin/llvm-dis.0.0.preopt.bc..
(1) Computing distance for callgraph
(2) Computing distance for control-flow graphs (this might take a while)
awk: fatal: cannot open file `/home/h3d/HUCHENG/project/targets/temp/dot-files/cfg._ZN4llvm12function_refIFPNS_14GlobalVariableEvEE11callback_fnIZN12_GLOBAL__N_117DataFlowSanitizer7runImplERNS_6ModuleEE3.dot' for reading (No such file or directory)
**awk: fatal**: cannot open file `/home/h3d/HUCHENG/project/targets/temp/dot-files/cfg._ZZL28predictValueUseListOrderImplPKN4llvm5ValueEPKNS_8FunctionEjRKN12_GLOBAL__N_18OrderMapERSt6vectorINS_12UseListOrderESaISB_EEENK3.dot' for reading (**No such file or directory**)
**mv**: cannot stat '/home/h3d/HUCHENG/project/targets/temp/dot-files/cfg._ZN4llvm19TargetTransformInfo5ModelIN12_GLOBAL__N_19NoTTIImplEE27preferPredicateOverEpilogueEPNS_4LoopEPNS_8LoopInfoERNS_15ScalarEvolutionERNS_15AssumptionCacheEPNS_17TargetLibraryInfoEPNS_13DominatorTreeEPNS_25LoopVectorizationLegalityE.dot.smaller.dot': **File name too long**

[strongcourage]

Hi @DigOrDog,

Thank you for reporting the issue. I've never fuzzed LLVM using AFLGo, so perhaps the target is too large. The first thing you should do is ensure that you can fuzz LLVM using AFL. Best.

[martinclauss]

Reading the error messages you get, the problem might just be the maximum allowed length of filenames. Check the limits with:

getconf -a | grep -iE '\b(name_max|path_max)\b'

on my machine:

NAME_MAX                           255
PATH_MAX                           4096

So in your concrete example:

echo -n 'cfg._ZN4llvm19TargetTransformInfo5ModelIN12_GLOBAL__N_19NoTTIImplEE27preferPredicateOverEpilogueEPNS_4LoopEPNS_8LoopInfoERNS_15ScalarEvolutionERNS_15AssumptionCacheEPNS_17TargetLibraryInfoEPNS_13DominatorTreeEPNS_25LoopVectorizationLegalityE.dot.smaller.dot' | wc -c
257

As far as I know, there is no easy way to increase this limit since it depends on the Linux kernel and file system driver.
I don't know the details of how important this filename is for AFLGo but maybe it can be shortened somehow (e. g. hash of the mangled symbol?)

[DigOrDog]

#138 (comment)
As mentioned above, I have indeed found that the issue is caused by the excessively long file name. It's easy to see that this file is generated by executing

$AFLGO/distance/gen_distance_fast.py $SUBJECT/build/bin $TMP_DIR llvm-dis. 

So, I'd like to ask if there is a solution to this problem.

Hi @DigOrDog,

Thank you for reporting the issue. I've never fuzzed LLVM using AFLGo, so perhaps the target is too large. The first thing you should do is ensure that you can fuzz LLVM using AFL. Best.

[DigOrDog]

Reading the error messages you get, the problem might just be the maximum allowed length of filenames. Check the limits with:

getconf -a | grep -iE '\b(name_max|path_max)\b'

on my machine:

NAME_MAX                           255
PATH_MAX                           4096

So in your concrete example:

echo -n 'cfg._ZN4llvm19TargetTransformInfo5ModelIN12_GLOBAL__N_19NoTTIImplEE27preferPredicateOverEpilogueEPNS_4LoopEPNS_8LoopInfoERNS_15ScalarEvolutionERNS_15AssumptionCacheEPNS_17TargetLibraryInfoEPNS_13DominatorTreeEPNS_25LoopVectorizationLegalityE.dot.smaller.dot' | wc -c
257

As far as I know, there is no easy way to increase this limit since it depends on the Linux kernel and file system driver. I don't know the details of how important this filename is for AFLGo but maybe it can be shortened somehow (e. g. hash of the mangled symbol?)

Thank you for pointing out the root of this issue. I also wonder if this file name will have any impact on the AFLGo process. Hashing does seem like a very good solution. I'm not sure if it's feasible.

[DigOrDog]

Hi @DigOrDog,

Thank you for reporting the issue. I've never fuzzed LLVM using AFLGo, so perhaps the target is too large. The first thing you should do is ensure that you can fuzz LLVM using AFL. Best.

Hi @strongcourage
Today I was able to fuzz LLVM with AFL, so using AFL to fuzz LLVM is not an issue. I'd like to use Hash with AFLGO to address the problem of long file names, but I'm not sure if this would affect the code logic. I hope you can provide a response, thank you.