Ankou fails during initialization

[Rumata888]

Hi. I'm using the latest version of AFLPlusPlus with afl-clang-fast. The compiled harness definitely accepts input. However, for some reason ankou gives the following error during initialization for each case:
Problem when writing in control pipe: invalid argument
I run it with the following command:
./Ankou -app ./fuzzed_app -threads 1 -dict my.dict -i fuzz/ankou_input -o fuzz/ankou_output
Any ideas what could be the issue here?
Running with regular afl-fuzz works

[Jiliac]

Hey @Rumata888 . Thanks for trying out Ankou!

This is a problem of communication between Ankou and the compiled binary. Ankou intends to reproduce the way AFL works but there may be issues. Could you give more details on how you compiled this binary so I can reproduce and debug the issue?

[vanhauser-thc]

afl++ has an advanced feature where there can be a back-and-forth communication between target and afl-fuzz.
likely it was just an unlucky value that afl-fuzz sent and with retrying it might just work.

But I just added a check to only accept an afl-fuzz transfer request if it was compiled with LTO.
you have to checkout the dev branch though and recompile your target.

[Rumata888]

HI. I don't see a dev branch @vanhauser-thc . It seems you haven't pushed the commit.
I had problems with compiling with lto, so I just used regular afl-clang-fast.
@Jiliac Tested with an example vulnerable app: https://raw.githubusercontent.com/mykter/afl-training/master/quickstart/vulnerable.c
Compiled it with afl. Same probllem.
And thank you. I really like the idea behind Ankou. That's why I wanted to try it out. Hope, I will be able to.

[Rumata888]

Ah, @vanhauser-thc, sorry. You meant in afl++. Will try that.

[Rumata888]

Still getting the same problem. Tried with afl-gcc and everything worked. Any ideas?

[Jiliac]

Just to confirm my understanding: Using vanilla AFL (not AFL++), you compiled with afl-gcc and it worked, but using afl-clang-fast Ankou crashed with the control pipe error mentioned above? afl-gcc was mostly used during the development of Ankou, so it's possible there is a problem with the clang version of the compiler. I will look at this more closely over the weekend.

[vanhauser-thc]

I tried a compiled binary with afl++ 3.00a with vanilla afl-fuzz from google and as expected this works fine.
seems to be an ankou issue

[Rumata888]

I was using AFL++, not vanilla AFL (wanted to try ankou with laf-intel). afl-clang-fast failed on me, afl-gcc worked.

[Jiliac]

Actually, the whole communication with the binary is broken. The fork server doesn't even start correctly. I think we never tried to use afl-clang-fast as a compiler actually, so this might not be due to a specific feature of afl++.

2020/09/12 11:52:32 Fork server did not start correctly: signal 47.                                            
2020/09/12 11:52:32 Problem in starting frk srv                                                                
2020/09/12 11:52:32 Problem when writing in control pipe: invalid argument  

This is the function doing the initialization in Ankou. As I remember, it was just a translation of the equivalent init_forkserver of vanilla AFL. I don't see that there is a special case of afl-clang-fast in AFL code. Do you know what I am missing @vanhauser-thc ?

[vanhauser-thc]

it looks to me like the forkserver control in fuzz-loop.go is implemented wrong.
intially there must be a read from afl-fuzz once the target is started (so a write happens in the target to the pipe).
then comes the endless loop afl-fuzz -> target (ready to send testcase), afl-fuzz <- target (sends pid), afl-fuzz <- target (done, return code)

the initial first read is not made and that might be the reason.

[Jiliac]

Thanks a lot for looking at it ❤️ ! Will try to get this fix before next week.

[Jiliac]

I got back in the code. Actually the read is done, but in the initForkserver function. With the gcc binary, this read goes along correctly; The status returned is 0. But in the case of the clang binary, the status returned is 0xc000002f, which as far as I can figure out with the man 2 wait page, means the fork server was killed with signal 47? But then, there is no signal 47 defined AFAIK?

[vanhauser-thc]

Ah that is the issue.
you interpret the value of the first received int. that int has no sensible value. it can be unassigned/random or just a zero.
afl++ uses this to inform afl-fuzz about activated features.
vanilla afl and spin-offs ignore the value of the byte so they dont have an issue.
your implementation assess the value it seems it checks it if it is 0 which is unnecessary.

[Jiliac]

Ok thanks a ton for the help! Pushed a commit which removes this check. @Rumata888 tell me if now it works for you 😀.

[vanhauser-thc]

If you removed the check just for the first read on the forkserver pipe, but all other reads do the crash check - then it is the correct fix :)

[vanhauser-thc]

off-topic @Jiliac if you want to improve your benchmarks on fuzzbench then just use afl++ as the compiler for the targets. I recommend CLASSIC + CTX + laf-intel for that

export AFL_LLVM_LAF_ALL=1
export AFL_LLVM_INSTRUMENT=CLASSIC,CTX

[Jiliac]

Very good point on fuzzbench settings! I should include them. But fuzzbench doesn't accept Ankou in its benchmarks so far because Ankou gets too memory hungry when the PCA mode starts. I think it might be doable to optimize, but it'd be a lot of work.

[Rumata888]

Ok thanks a ton for the help! Pushed a commit which removes this check. @Rumata888 tell me if now it works for you .

Yes, it works, thanks.

[Rumata888]

off-topic @Jiliac if you want to improve your benchmarks on fuzzbench then just use afl++ as the compiler for the targets. I recommend CLASSIC + CTX + laf-intel for that

export AFL_LLVM_LAF_ALL=1
export AFL_LLVM_INSTRUMENT=CLASSIC,CTX

Why not LTO? Isn't it faster supposedly?

[vanhauser-thc]

LTO and pcguard in afl++ are collision free, which is an amazing thing, however requires that the fuzzer knows how large the map of the target is. that is why we transfer that special value in the beginning that made ankou crash.
no other fuzzer supports that currently so LTO and afl++'s pcguard compiled binaries can only be used with afl++'s afl-fuzz

(actually not fully true, if the target has less than 64k edges then they work fine with any afl spin-off. however there will be no speed increase. although the coverage will not be colliding anymore)

[Rumata888]

LTO and pcguard in afl++ are collision free, which is an amazing thing, however requires that the fuzzer knows how large the map of the target is. that is why we transfer that special value in the beginning that made ankou crash.
no other fuzzer supports that currently so LTO and afl++'s pcguard compiled binaries can only be used with afl++'s afl-fuzz

(actually not fully true, if the target has less than 64k edges then they work fine with any afl spin-off. however there will be no speed increase. although the coverage will not be colliding anymore)

Thanks for the explanation.
Btw @Jiliac have you tested ANKOU in a multithreaded setting? In the paper only 1-core setting is mentioned. I seem to be having a problem with CPU utilization. It's at 70%. Does ANKOU analyze the results of each test 1 by 1?

[Jiliac]

Yes Ankou was developped first for multicore, and then adapted to run on only 1 core to be able to benchmark the same way other fuzzing paper did. Can you give more details on your setting? How many cores you are using? All of them are at 70% for long period of time? Is the memory full?

[Rumata888]

I'm using it in a Linux Virtual machine with 8 cores dedicated to it out of 8 cores on the host. When I'm running 8 instances of afl or libfuzzer in fork mode with 8 threads, the CPU utilization is close to 100%. When I'm running ANKOU with 8 threads it averages around 70%. 7 GB of 32 GB of memory dedicated to the virtual machine are in use. It seems that there is a bottleneck.
What makes it worse is that Ankou has a tendency to increase test case length. As a result throughput suffers.

[Jiliac]

Very weird, I never had this problem before. What's your target? The one you mentioned above? Is it from the beginning, or after some time, like 5 to 10 minutes, when the PCA mode activates?

[Rumata888]

A mail parsing library. When the PCA mode activates. It's been fuzzing for more than 30h now, but CPU usage rarely goes higher than 80%. Also, a large chunk of that usage is ANKOU itself, not the fuzzed application

[Jiliac]

Then probably the PCA process is the bottleneck. It can potentially happen for programs with too many branches but I have never seen it happen in practice until now. Would have to experiment directly on the same target you are to be sure.

[Rumata888]

Well, it's a pity. I think it could be enhanced by using SIMD on x64 targets. Also, it seems, that for test cases where token repetitions result in loops it quickly becomes ineffective because of test case growth. So while the idea is nice, you can't use it everywhere.

[Jiliac]

Yes I looked into SIMD, but I didn't see a simple way to benefit from it for Go code. Seems like all the operations should be written manually :/

The "loop problem" should be handle by the "log transformation" used by AFL, shouldn't it? From AFL doc:

In addition to detecting new tuples, the fuzzer also considers coarse tuple
hit counts. These are divided into several buckets:

  1, 2, 3, 4-7, 8-15, 16-31, 32-127, 128+

This way each new iteration in a loop is less meaningful than the previous ones. Did you see this problem in practice?

[Rumata888]

Log buckets prevent AFL from adding test-cases with slight increases in particular edge count. Ankou determines whether a test-case should be added by how different it is from the current basis. The problem with that is that it will be much more likely that a test-case is distant enough from the current basis if one of its independent edges has a maximized hit count. But the hit count is logarithmic, so the test-case grows larger.
EDIT: I think, it could actually help a little, if you translated them back to linear scale.

[Jiliac]

Ah very good point! Actually, for the implementation, we used the AFL insight in the PCA. When we create the vector on which is used in Ankou algorithm (PCA + fitness function), we actually use the logarithm of the hit count. You can see this in the code (tr is the hit count).

[Rumata888]

Yup. I looked at it once again and it seems the problem was that initially aflplusplus generated several long test cases. Since ANKOU added them and they had unique coverage, it made it harder to add smaller inputs (they have to trigger very different edges to be inserted into the corpus when they don't have loops). I think Ankou would benefit from an initial check and a disclaimer, that its performance can degrade when presented with such initial corpus.