Add sanitizer coverage feedback evolution support part2

[anestisb]

Overview

This a follow up for #44. Clang's sancov data parsing component has been upgraded with runtime coverage bitmaps aiming to identify newly discovered execution paths. Initial sancov feedback parsing was relying solely on maximum global coverage counters for seeds evolution without considering newly discovered basic blocks that don't improve global counters. This PR is introducing a series of upgrades into that direction by implementing fundamental sancov runtime data parsing procedures & book keeping runtime structures.

Following paragraphs provide an executive summary of involved commits. Additional, details are available at each commit's description and accompanied source code comments.

SanCov upgrades

Supported methods

Clang sanitizers' coverage data can be exported with two methods based on environment variable flags:

1. As individual files per instrumented executable/DSO
2. Unified into a single raw packed file

Both methods are supported from sancov parsing component, although the later is preferred for performance (reduce File I/O & discovery logic) and efficiency (if sanitizer unhandled signal, such as SIGKILL, raised coverage data are lost - huge problem for Android) reasons. It should be also noted that coverage bitmaps are supported only for the later due to huge performance overhead when parsing data produced with first method. As such, when individual files are selected (coverage_direct flag disabled) only global coverage counters are supported without using BB bitmaps.

Parsing coverage data

When sanitizer coverage is enabled, compiler is injecting a call for the __sanitizer_cov procedure at every basic block identified. Runtime collected coverage data are then stored into generated files from __sanitizer_cov_dump procedure invoked as part of sanitizer's Die steps. Detected hit BB addresses are absolute, thus they need to be adjusted to relative before updating fuzzer's coverage bitmaps (that randomization bits again). This is accomplished by parsing the pid.sancov.map file which contains a minimized version of ProcFS maps that can be used to calculate BB relative addresses using matching map's base address.

When coverage feedback is enabled with raw unpack method, a digital tree (trie) is generated whenever a new input seed is selected for evolution. Loaded executable/DSO map names (as parsed from pid.sancov.map) are used as keys for the trie node actions (insert, search). Additionally, for each node where the matching exec/DSO has coverage instrumentation enabled, a coverage bitmap is allocated and maintained in parallel with trie lifetime. Trie and bitmaps are shared across the fuzzer threads, with each thread (roughly) executing the following steps when parsing coverage data:

• Parse pid.sancov.map file
  • Identify PC register size (required for raw file unpack offsets calculation)
  • Parse map entries and cache them in memory
  • Update global trie if new exec/DSO detected
• Parse pid.sancov.raw file
  • Unpack BB absoute addressess
  • Find matching map entry from cached maps file
  • Calculate relative address from map's base address
  • Find matching trie node using map name as search key
  • Check node's coverage bitmap if BB address has been hit before
  • If new hit update bitmap and associated counters
• Update thread's summary counters (coverage summary, instrumented #DSOs, etc.)
• Delete matching pid sancov files

Parsing component takes advantage of addresses locality (next BB addr most probably fits into same map entry) to minimize search overhead by caching last used map entry & trie node. Additionally, a small sorted lookup array of maps' base addresses is generated to speed up the maps index search step when parsing raw files.

Interaction with global data structures is mutex protected to avoid races. Bitmap updates occur in a first-come-first-serve nature ensuring that new BB hits are measured only once from actively running workers.

Evolution metrics

For evolution decisions (discard/keep mutated test cases) fuzz_sanCovFeedback is evaluating worker thread's coverage based on:

1. Newly discovered (not met before) BBs && total hit BBs not significantly dropped
2. Improved maximum coverage (total BB hits improved)
3. Additional instrumented DSOs loaded (dlopen() paths discovered)

For rule 1. the second part of the expression has been temporarily introduced until test case queues are implemented. Currently honggfuzz is using absolute elitism, promoting only one test case (the best) to next iterations. In that case if newly discovered paths are blindly followed without considering global coverage metrics, there's a higher propability that fuzzer enters a dead-end state where target always early aborts. This is a very common case for media decoders/parsers where a potentially malformed chunk triggers additional evaluation/error-handling exec paths, which effectively results into new BB hits. However, such exec path is most probably triggering an early abort call due to malformed data trapping the fuzzer into a dead-end that might never escape until initial seed is replaced due to expiration. When evolution seed queues are implemented previous metrics will be revised.

First execution of new seed selection

The 1st iteration of a newly selected seed from input corpus is perfomed without any content mangling taking place. This will effectively set the initial coverage data with base metrics required to compare against at following evolution. Additionally, to avoid races between worker threads that might result into inaccurate base data creation, only one thread is executing this first round blocking the other threads from continuing before all initial data structures updates are completed.

SanCov display sample

============================== STAT ==============================
Iterations: 9700
Start time: 2016-01-13 03:02:56 (6926 seconds elapsed)
Input file/dir: 'seeds'
Fuzzed cmd: 'target'
Fuzzing threads: 2
Execs per second: 2 (avg: 1)
Crashes: 7165 (unique: 2, blacklist: 590, verified: 2)
Timeouts: 97
Dynamic file size: 59100 (max: 10485760)
Dynamic file max iterations keep for chosen seed (22/512)
Coverage (max):
  - total hit #bb:  6990 (coverage 8%)
  - total #dso:     5 (instrumented only)
  - discovered #bb: 1246 (new from input seed)
  - crashes:        0
============================== LOGS ==============================
[I][31688] fuzz_sanCovFeedback():480 SanCov Update: file size (Cur,New): 59090,59090, newBBs:22, counters (Cur,New): 6987/5,6990/5

Linux: fuzzing with sanitizers

exitcode/SIGABRT monitoring

A common practise when fuzzing with sanitizers enabled for target application, is to set the abort_or_error flag. This will effectively result into a SIGABRT being raised when sanitizer detects an error. If SIGABRT is a monitored signal from fuzzer's ptrace API, crash detection logic will identify the interesting signal and proceed with unwinding & post crash actions.

For some targets SIGABRT monitoring is not desired resulting into abort_or_error flag being useless since some detected errors might get lost. In these case a custom exitcode is registered and monitored per enabled sanitizer (ASan, MSan, UBSan). If ptrace API detects a PID exit with monitored exitcodes, additional crash processing actions are triggered that parse the sanitizer saved report file. linux/arch.c initialization routines detect if SIGABRT is a monitored signal based on common header config and adjust sanitizer flags and reports generation accordingly. Crash counters, uniqueness decissions (including blacklist) and crashes verifier actions have been updated to support for this new crash monitor method.

Additionally, for sanitizers that include unwinded stack traces into reports (such as ASan), crashing thread's trace is parsed to calculate stack hashes benefiting when detecting duplicates and verifying crashes behavior.

Since sanitizer flags are now defined dynamically based on input arguments, linux/arch.c init procedure is generating the appropriate flags once and stores them in global buffers that can be instantly accessed from worker threads when spawning new processes.

Number of major frames

Originally top 7 frames from call stack where used to calculate hash signature. When ASan fuzzing enabled and abort_or_error flag set, the top 6-9 frames are usually occupied with sanitizer internal procs, resulting into crashes being wrongly marked as duplicates and thus being lost. As such when these two conditions are met, arch init procedure is increasing the default 7 major frames to 14, mitigating the problem.

exitcode detected crash report sample

=====================================================================
FUZZER ARGS:
 flipRate     : 0.000500
 externalCmd  : NULL
 fuzzStdin    : FALSE
 timeout      : 30 (sec)
 ignoreAddr   : 0xff
 memoryLimit  : 0 (MiB)
 targetPid    : 0
 wordlistFile : NULL
 dynFileMethod: NONE
 fuzzTarget   : /system/bin/target -i ___FILE___
ORIG_FNAME: 32f4ea6b94720bf2ee7d7adca4228d25
FUZZ_FNAME: ./ASAN.PC.a999d5a3.STACK.1764a787cf.CODE.WRITE.ADDR.0xb29a6ee4.INSTR.[UNKNOWN].fuzz
PID: 20278
EXIT CODE: 104 (ASAN)
OPERATION: WRITE
FAULT ADDRESS: 0xb29a6ee4
STACK HASH: 0000001764a787cf
STACK:
 <0xa999d5a3> [/system/lib/target2.so + 0xd35a]
 <0xa995a1cb> [/system/lib/target2.so + 0x901c]
 <0xa9a093e7> [/system/lib/target2.so + 0x13f3e]
 <0xa999bff3> [/system/lib/target2.so + 0xd1ff]
 <0xa98f34bf> [/system/lib/target2.so + 0x294b]
 <0xa990973f> [/system/lib/target2.so + 0x3f73]
 <0xa98ddc1f> [/system/lib/target2.so + 0x13c1]
 <0xb580bf7f> [/system/lib/target1.so + 0xb6f7]
 <0xb5819f37> [/system/lib/target1.so  + 0xc4f3]
 <0xb46e74c7> [/system/lib/target0.so  + 0x194c]
 <0xb470f84f> [/system/lib/target0.so + 0x4184]
 <0xb46f221b> [/system/lib/target0.so + 0x2421]
 <0xb59e0333> [/system/lib/libutils.so + 0x1033]
 <0xb5a36aa7> [/system/lib/libc.so + 0x49aa]
 <0xb5a06751> [/system/lib/libc.so + 0x1975]

Misc improvements

• Fixed getdelim & getline memory leaks at files component (detected with valgrind)
• Updated display messages with more accurate context for sancov data
• Updated report writing with appropriate entries when crashes detected via exitcode and not raised signal
• Comments upgrades/clean-ups
• Check if target exec timeout is big enough to tollerate coverage overhead. Users should monitor for excessive #timeouts and adjust if required.
• Enable compatibility with OSX indent
• Remove dead helper static functions

[anestisb]

Even the smallest comment that might have some performance improvement at sancov parsing or book keeping is super welcome. Evolution seed queues are in the todo list for the near future to get the best out of the coverage bitmaps.

[robertswiecki]

Amazing work, thanks!