DynamoRIO "Incompatable API Version"

[wessupermare]

Running a test application works fine with instrumented build but fails under dynamic instrumentation with recommended version of DynamoRIO (7.0.1 release).

Linux

root@computer:/<path>/manul/manul# python3 manul.py -i in -o out --debug "./test.out @@"
> [WARNING] Output directory is not empty, creating backup of output folder
> [INFO] Done
> [INFO] 1 fuzzer instances sucessfully launched
> [INFO] Starting fuzzer 0
> [INFO] Setting up shared mem 2 for fuzzer:0
> [INFO] Initializing mutators
> [INFO] Performing dry run
> [INFO] Launching /<path>/manul/DynamoRIO-Linux-7.1.0-1/bin64/drrun -c /<path>/manul/manul/linux/dbi_64/libbinafl.so -coverage_module test.out -debug -- ./test.out in/inp.txt
> /<path>/manul/DynamoRIO-Linux-7.1.0-1/bin64/drrun -c /<path>/manul/manul/linux/dbi_64/libbinafl.so -coverage_module test.out -debug -- ./test.out in/inp.txt
> [INFO] Target started, waiting for return
> [INFO] Output from target <Application /<path>/manul/manul/test.out (79). Client library targets an incompatible API version and should be re-compiled.>
> [ERROR] inp.txt doesn't cover any path in the target, Make sure the binary is actually instrumented
> [WARNING] Fuzzer 0 unexpectedly terminated
> ^C[INFO] Stopping all fuzzers and threads
> Killed

Windows doesn't even get to that point due to #28, but is included for completeness.

Windows

C:\<path>\manul\manul>python3 manul.py -i in -o out --debug "<different_path>\Test.exe @@"
> [INFO] 1 fuzzer instances sucessfully launched
> [INFO] Starting fuzzer 0
> [INFO] Setting up shared mem 1565800978_0 for fuzzer:0
> [INFO] Initializing mutators
> [INFO] Performing dry run
> Process Process-1:
> Traceback (most recent call last):
>   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1264.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 297, in _bootstrap
>     self.run()
>   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1264.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 99, in run
>     self._target(*self._args, **self._kwargs)
>   File "C:\<path>\manul\manul\manul.py", line 895, in run_fuzzer_instance
>     fuzzer_instance.run() # never return
>   File "C:\<path>\manul\manul\manul.py", line 753, in run
>     self.dry_run()
>   File "C:\<path>\manul\manul\manul.py", line 495, in dry_run
>     memset(self.trace_bits, 0x0, SHM_SIZE)
> OSError: exception: access violation writing 0x0000000062420000
> [WARNING] Fuzzer 0 unexpectedly terminated
> [INFO] Stopping all fuzzers and threads

[mxmssh]

Could you provide more details about your environment? Like OS version and etc. Are you trying to use 64-bit Dynamorio with 64-bit client against 64-bit binary?

[wessupermare]

Environment:
Windows 10 build 18956.1000 & Ubuntu 19.10 (current as of 2019/08/19).
Python 3.7.4.
Manul up to date from source as of this morning (2019/08/13).
Everything system-wise up to date.

Yes, the architectures do line up. I built everything locally. I'm using the Linux DynamoRIO for Ubuntu, and the Windows DynamoRIO for Windows. All of the paths check out, and running the drrun command manually gives the same error. I think you might need to add the files for building the afl DynamoRIO module locally so that everything will line up system by system. Not sure. Thanks!

[mxmssh]

Does drrun work with any other client(not related to Manul) in your environment?

[wessupermare]

No, it doesn't even get to the binary stage. It looks like a compatibility issue between DynamoRIO and libbinafl.so.

[mxmssh]

Can you try any client from .. \samples\bin64\ ?

[wessupermare]

Could I get a sample invocation? Not sure quite what you mean.

[killyp]

Having this issue as well on Win10 1803.

DynamoRIO Version - 7.1.0
Manul Version - 0.3
Python Version - 3.7.4

[killyp]

Fixed by downgrading DynamoRIO Version to 7.0.0-RC1

[symeonp]

Hey,

I did encounter same issue on windows, I fixed that by recompiling the coverage dll files with version 7.91.18103. It looks like Maksim compiled and shipped the dll/so files from an earlier version as KillyP commented.

[mxmssh]

That's right, I will update readme and provide instruction on how to compile clients. Thank you folks for triaging that.

[mxmssh]

So, the general advice is to downgrade DynamoRIO as KillyP mentioned. I updated the README in case if the newest version of DynamoRIO is required. Closing the issue.

[wessupermare]

I'm still having trouble, but admittedly different trouble. Now I get [ERROR] 1.txt doesn't cover any path in the target, Make sure the binary is actually instrumented followed by a series of [WARNING] Fuzzer 0 unexpectedly terminated.
Yes, my config file points to the correct locations and yes, the architectures line up.

[mxmssh]

Could you run it with --debug option?

[wessupermare]

Sure!
Output:

[INFO] 1 fuzzer instances sucessfully launched                                                                                                                                                                                                                                  [INFO] Starting fuzzer 0
[INFO] Setting up shared mem 1566332221_0 for fuzzer:0
[INFO] Initializing mutators
[INFO] Performing dry run
[INFO] Launching C:\Users\Wes\source\repos\AutoFuzz\manul\DynamoRIO-Windows-7.0.0-RC1\bin64\drrun.exe -c C:\Users\Wes\source\repos\AutoFuzz\manul\manul\win\dbi_64\binafl.dll -coverage_module Test.exe -debug -- ..\..\Test\bin\Debug\netcoreapp3.0\Test.exe in/1.txt
[INFO] Target started, waiting for return
[INFO] Initial input file: 1.txt triggers an exception in the target
[WARNING] Initial input 1.txt leads target to crash (did you disable leak sanitizer?). Enable --debug to check actual output
[WARNING] Fuzzer 0 unexpectedly terminated
[WARNING] Fuzzer 0 unexpectedly terminated
[WARNING] Fuzzer 0 unexpectedly terminated
[INFO] Stopping all fuzzers and threads

[mxmssh]

Two possible options here:

1. There is a bug in the Manul instrumentation client
2. There is some problem with the target application itself...

[wessupermare]

The target runs fine when I run it on its own. The provided DRRUN command exits silently but %ERRORLEVEL% is set to -1073741819.

[mxmssh]

is it .NET application?

[wessupermare]

I'm testing one written in C compiled with stock GCC, and one written in C# compiled with preview Roslyn. So one of each.

[mxmssh]

TBH, I don't know why it doesn't work. The first thing that we should exclude is the problem with DynamoRIO. There are test clients supplied with DynamoRIO, can you try to run one of them?

[mxmssh]

They are located in the \samples\bin64\ folder

[wessupermare]

Okay. What binary should I run them against?

[mxmssh]

Test\bin\Debug\netcoreapp3.0\Test.exe

[wessupermare]

Will do.

[wessupermare]

drrun.exe -c ..\samples\bin64\memtrace_simple.dll -- ..\..\..\Test\bin\Debug\netcoreapp3.0\Test.exe ..\..\manul\in\1.txt

Same %ERRORLEVEL%

[mxmssh]

Most likely there is some problem with DBI instrumentation of your binary. DynamoRIO maintainers can provide more details for you. You can open an issue here: https://github.com/DynamoRIO/dynamorio/issues

[wessupermare]

Thanks!

[wessupermare]

Quick question: does DynamoRIO DBI instrumentation work on your end? I've tried it on a separate machine and it still isn't working.

[mxmssh]

Yes, I've done 2 fuzzing campaigns in the past on Windows using DynamoRIO DBI instrumentation and Manul.

[wessupermare]

Do you have a config and test application I can run to see if it's my environment?

[mxmssh]

There is actually one test application in win/test/test64. You just need to specify paths (to drrun and win/dbi_64/binaf.dll in the manul.config and run it.

[wessupermare]

[WARNING] Initial input 1.txt leads target to crash (did you disable leak sanitizer?). Enable --debug to check actual output
That was run with --debug, by the way.
After that it runs into #28 .

[mxmssh]

By the way, when you run it with --debug it should create a log file in the same folder where you run the binary. Could you copy-paste it here?

[wessupermare]

I don't see a log file. What would it be called?

[mxmssh]

Could you copy-paste your config file here? The file should be called afl.*.proc.log

[wessupermare]

#   Manul - configuration file
#   -------------------------------------
#   Maksim Shudrak <mshudrak@salesforce.com> <mxmssh@gmail.com>
#
#   Copyright 2019 Salesforce.com, inc. All rights reserved.
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at:
#     http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

# Manul config file
# Format: <option_name> = <value>
# -----------------

# absolute path to dictionary with usefull tokens
#dict = dictionaries/test.dict

# Mutator weights (should be 10 in total). Use my_mutator:x,my_mutator_2:x to define and use your own
# custom mutator (should be 10 in total). Specify 0 to disable certain mutators. Weights basically are
# used to tell manul how many mutations per 10 executions should be performed by certain fuzzer.
# example afl:5,radamsa:2,my_awesome_fuzzer:3
# afl will be used to mutate 5 out of 10 cases, 2 out of 10 for radamsa and 3 out of 10 for my_awesome_fuzzer
# Your custom mutator's main file should be located in the same folder as manul.py.
# Two default mutators should always be defined (afl, radamsa)
mutator_weights=afl:10,radamsa:0
#mutator_weights=afl:6,radamsa:0,example_mutator:4

# Use determenistic seed for test cases generation (only radamsa option).
determenistic_seed = False

# Print fuzzing summary per thread instead of total summary
print_per_thread = False

# disable volatile bytes suppression algorithm
#disable_volatile_bytes = True

dbi = dynamorio
# If dbi param is not None the path to dbi engine launcher and dbi client should be specified.
dbi_root = C:\Users\Wes\source\repos\AutoFuzz\manul\DynamoRIO-Windows-7.0.0-RC1\bin64\drrun.exe
dbi_client_root = C:\Users\Wes\source\repos\AutoFuzz\manul\manul\win\dbi_64\binafl.dll
#dbi_client_libs = None

# Timeout for target binary
timeout = 10

# net_config_master and net_config_slave below are used to share manul instances over network.
# Path to network configuration file with a list of IP:port slave addresses. Specified for master instance.
net_config_master = None

# IP and port to listen for connections from master (e.g. net_config_slave = 0.0.0.0:1337)
net_config_slave = None

# Run in debug mode, print details in console
debug = False

# Print Manul ASCII logo at the beginning
manul_logo = False

# Disable stats saving in the manul working dir
#no_stats = True

# Save debug messages to log files (one per thread)
logging_enable = False

# Bitmap sync frequency (5000 recommended for DBI mode)
sync_freq = 10000

# Custom path to save input file
#custom_path = test_path

# Command line fuzzing (experimental)
#cmd_fuzzing = True

# define signals to be ignored by manul
user_signals = 6

# Network fuzzing. Target IP address
#target_ip_port = 127.0.0.1:7715
# tcp | udp
#target_protocol = tcp
# wait time before actually start sending test cases in the socket
#net_init_wait = 1
# wait time between test cases
#net_sleep_between_cases = 0.0

[wessupermare]

That log file doesn't exist. I've even done a recursive ls and grepped through it with no success.

[mxmssh]

Weird, there is something wrong with DynamoRIO instrumentation...

[wessupermare]

Rebuilding both DynamoRIO and the client lib from source fixed it on Linux. Thanks for your help! How does a Windows build work for that? Do I do it the same way with CMake for Windows?