Coverage Evaluation

[maxammann]

Targets:

• OpenSSL 1.1.1 (one version e.g. f/k)
• wolfSSL 5.3.0

Fuzzers/Testers:

• AFLNet/StateAFL/AFLNew (through profuzzbench)
• TLSAnvil

Methodology:

Option A)

• Let the fuzzer run for 1h or 24h and collect corpus
• Instrument a plain statically compiled server executable with coverage
• Execute the corpus over TCP against the server (In AFLNet this is done through afl-replay https://github.com/aflnet/aflnet/blob/master/afl-replay.c)

Issues:

• ⚠️ In tlspuffin we use a custom setup of the server (certificate, options)

Option B)

• Use Option A) for AFLNet, TLSAnvil
  • ...
• Let the fuzzer run for 1h or 24h and collect corpus
• Compile a libssl.a static library just like the coverage binary from Option A)
  • Do not use any fuzzing specific flags for this binary! No -fsanitze=coverage! Use -fprofile-arcs -ftest-coverage!
  • Requires a build of tlspuffin which is designed for coverage evaluation.
• Use libssl.a in tlspuffin and execute the seed corpus

TODO

• Adjust coverage filters for profuzzbench

• Compile without optimizations

• Verify and check ASAN usage (max 1 campaign in parallel with ASAN!) #244

• Enable 1.2 and 1.3 (wolfssl: -v d -7 3)

• profuzzbench

  • Create a coverage.csv from profuzzbench with OpenSSL
    • StateAFL
    • AFLNwe
    • AFLNet
  • Create a coverage.csv from profuzzbench with wolfSSL
    • StateAFL
    • AFLNwe
    • AFLNet

• tlspuffin

  • Create a coverage.csv from tlspuffin with OpenSSL
  • Create a coverage.csv from tlspuffin with wolfSSL

• TLSAnvil

  • Generate max. coverage from TLS-Anvil with OpenSSL
    Coverage with TLS-Anvil when testing an OpenSSL 1.1.1j server:

  • Generate max. coverage from TLS-Anvil with wolfSSL

• Precompiled (Instead of precompiled I validated that the max. line count matches across all campaign evaluations)

  • Use precompiled OpenSSL with TLS-Anvil with OpenSSL
  • Use precompiled OpenSSL with profuzzbench
  • Use precompiled OpenSSL with TLS-Anvil with wolfSSL
  • Use precompiled wolfSSL with profuzzbench
  • Use precompiled OpenssL with tlspuffin
  • Use precompiled wolfSSL with tlspuffin

[maxammann]

Related:

• Test failures tls-attacker/TLS-Anvil#11
• libfaketime required? stateafl/stateafl#10

StateAFL -K bug:

• https://github.com/stateafl/stateafl/blob/d923e22f7b2688db45b08f3fa3a29a566e7ff3a4/llvm_mode/afl-llvm-rt-state-tracer.o.c#L1879
• https://github.com/stateafl/stateafl/blob/d923e22f7b2688db45b08f3fa3a29a566e7ff3a4/README-AFLnet.md?plain=1#L248

[maxammann]

AFLnwe: -f .tmp-file to workaround short read race condition.

-> Does not work, fix is to just use a malloc buffer

race condition probably happend because openssl was exiting directly

[maxammann]

Execution plan:

• Task A: Profuzzbench fuzzers (aflnet, aflnwe, stateafl): 10 trials, 24h each
  • Fuzzes wolfSSL and OpenSSL TLS 1.3/1.2 servers
  • 10 (trials) * 2 (targets) * 3 (fuzzers) = 60 campaigns
  • -> Coverage plot
• Task B: tlspuffin: 10 trials, 24h, each one core
  • Restrict to TLS 1.3/1.2 servers (only TLS 1.3/1.2 server seeds)
  • 10 (trials) * 2 (targets) = 20 campaigns
  • -> Coverage plot

---

• Task C: tlspuffin 32 cores, 24h, 1 trial
  • (all seeds)
  • 1 (trials) * 2 (targets) = 2 campaigns
  • -> Single coverage number
• Task D: TLS Anvil
  • Test first clients and then servers
  • 1 (trials) * 2 (targets) = 2 campaigns
  • -> Single coverage number

The final result of this evaluation will be:

• a large coverage plot, which includes tlspuffin, StateAFL, AFLNet and AFLnwe
• Single number coverage comparision between TLS Anvil and tlspuffin and something like tlspuffin reached 95% the coverage of TLS Anvil after x hours.

---

Rediscover vulns:

• Task E: Profuzzbench fuzzers (aflnet, aflnwe, stateafl): 3 trials, 24h each
  • Fuzzes wolfSSL 1.3/1.2 servers
  • ASAN
  • 3 (trials) * 1 (targets) * 3 (fuzzers) = 3 campaigns
  • -> # of crashes

---

(optional)

• Task A.1: Profuzzbench fuzzers (aflnet, aflnwe, stateafl): 10 trials, 24h each
  • Fuzzes OpenSSL TLS 1.2 server
  • With null cipther
  • 10 (trials) * 1 (targets) * 1(fuzzers) = 10 campaigns
  • -> coverage plot (+ # of crashes)

[maxammann]

Pain points during evaluation

• Getting StateAFL running (-K flag)
• Getting AFLnwe runnign (-f flag)
• tlspuffin uses Cargo for linking. This makes tlspuffin reproducible, but it is very hard to compare with others, because linking is done by rustc instead of having control
• Environment differences: like clang/gcc versions, gcov versions, optimization levels, different compiler options chosen in build.rs/container/pesto.

[maxammann]

• CVE-2022-38152 (SDOS2): Not reachable because wolfSSL_clear is not called
• CVE-2022-38153 (CDOS): Not reachable because not client is fuzzed
• CVE-2022-39173 (BUF): Not reachable because too hard.
• CVE-2022-42905 (HEADP): ??

[maxammann]

Running s_server with --rand-state=0 does not work -> crash

[maxammann]

GCC and Clang calculate coverage differently, some seeses case XYZ as a line, the other doesn't.

[maxammann]

Q: TLS-Anvil client versions? wolfssl client 1.2/1.3, openssl client 1.2/1.3
A: Both: https://github.com/tls-attacker/TLS-Anvil-Large-Scale-Evaluator/blob/187193ab62a5473c9486dcaa2d3ec79f66fe34a0/src/main/java/de/rub/nds/tlstest/evaluator/evaluationtasks/TestsuiteClientEvaluationTask.java#L96-L98

Let TLS-Anvil choose.

[maxammann]

Renegotiatoin Info in 5.3.0 is incompatible with TLS-Anvil: tls-attacker/TLS-Attacker#152

[maxammann]

Had to fight this one: profuzzbench/profuzzbench#6

[maxammann]

Fixed coverage misses when generating it. flush on exit

[maxammann]

read_from_terminal was the issue with aflnwe stuck coverage on OpenSSL

[maxammann]

Observed issues during this evaluation:

• AFLnwe: OpenSSL exits immediately because stdin is not connected thuanpv/aflnwe#2
• AFLnwe: Race condition when target is very fast thuanpv/aflnwe#1
• AFLnwe: Timestamp of queued test cases are non sequential thuanpv/aflnwe#3
• AFLnwe: Copy seeds instead of hard linking thuanpv/aflnwe#4
• TLS-Anvil: Crash during wolfSSL 5.3.0 client exploration tls-attacker/TLS-Attacker#152
• StateAFL: StateAFL can not be used with -K stateafl/stateafl#11