call the python3 binary. needed on OSX, and tested working on Debian

[martinlindhe]

I've only had time to try out afl-collect yet, but these changes are needed for proper usage on OSX.

2 tests are failing on OSX because the setsid binary is not available.

[coveralls]

Coverage remained the same at 90.868% when pulling 2010609 on martinlindhe:python3 into c80ea37 on rc0r:master.

[coveralls]

Coverage remained the same at 90.868% when pulling 2010609 on martinlindhe:python3 into c80ea37 on rc0r:master.

[coveralls]

Coverage remained the same at 90.868% when pulling 2010609 on martinlindhe:python3 into c80ea37 on rc0r:master.

[martinlindhe]

A second issue on OSX is the gdb+exploitable step isn't working. I'll need to dig a little deeper to figure out what's going on there (it is working fine on my debian box).

Envoking like this:

afl-collect --execute-gdb-script gdb-script --remove-invalid . coll -- ./target_bin @@
afl-collect 1.28a by rc0r <hlt99@blinkenshell.org> # @_rc0r
Crash sample collection and processing utility for afl-fuzz.

[*] Going to collect crash samples from '/Users/m/XXX'.
[*] Found 1 fuzzers, collecting crash samples.
[*] Successfully indexed 9 crash samples.
[*] Saving invalid sample info to database.
[!] Removed 0 invalid crash samples from index.
[!] Removed 0 timed out samples from index.
[*] Generating intermediate gdb+exploitable script '/Users/m/XXX/coll/gdb-script.0' for 9 samples...
[*] Executing gdb+exploitable script 'gdb-script.0'...
*** GDB+EXPLOITABLE SCRIPT OUTPUT ***
*** ***************************** ***
[*] Saving sample classification info to database.
[!] Removed 0 duplicate samples from index. Will continue with 9 remaining samples.
[*] Generating final gdb+exploitable script '/Users/m/XXX/coll/gdb-script' for 9 samples...
[*] Copying 9 samples into output directory...

gdb is configured and has the "exploitable" command available. no idea how i could invoke the command in gdb though, any clues?

$ cat ~/.gdbinit
source /usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/exploitable-1.32_rcor-py3.5.egg/exploitable/exploitable.py

$ ls -alh /usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/exploitable-1.32_rcor-py3.5.egg/exploitable/exploitable.py
-rw-r--r--  1 m  admin    10K Jun 29 18:23 /usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/exploitable-1.32_rcor-py3.5.egg/exploitable/exploitable.py

$ gdb
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin15.4.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) exploitable
__main__:99: UserWarning: GDB v7.11 may not support required Python API
Description: The target process exited normally.
Short description: GracefulExit (0/0)
Hash: 0.0
Exploitability Classification: NOT_EXPLOITABLE
Explanation: The target process exited normally.
(gdb)```

[martinlindhe]

Hm, I think this is the issue, investigating...

brew install gdb gives me a gdb that uses "system python" (ie python 2),

reinstalling with brew install gdb --with-python (which should use python 3)

$ brew info gdb
gdb: stable 7.11 (bottled)
GNU debugger
https://www.gnu.org/software/gdb/
[... snip ...]
==> Options
--with-python
    Use the Homebrew version of Python; by default system Python is used

[rc0r]

Regarding your second issue:

Where is your gdb binary? Path to gdb is currently hardcoded in afl_collect.py.

[martinlindhe]

Ah, binary is /usr/local/bin/gdb. I will look at afl_collect.py too

on my previous note, recompiling gdb didnt seem to have much effect, will investigate some more.

$ gdb
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin15.5.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) python-interactive
>>> sys.version_info
sys.version_info(major=2, minor=7, micro=11, releaselevel='final', serial=0)

(it still uses python 2.7)

[rc0r]

Regarding your PR:

I'd prefer to keep the code as it is for two reasons:

1. I'm not sure that changing all refs from python to python3 in the code doesn't break things for systems where Python 3 is the default and installed as python. (This might not be the case for any system out there, but I simply don't know.)
2. Because of its rather exotic dependencies afl-utils is not very suitable for distribution via PyPI (etc.) atm. So the recommended way of installling it is by cloning from the GH repo and installing from source (aka running setup.py). In order to not cause conflicts with Python packages installed by the system package manager it is highly recommended to install into a virtualenv. I think leaving the code as it is further encourages the use of virtualenv on systems that default to Python 2, which is the case for widely used Ubuntu and Debian, f.e.

Using virtualenv the setup process would be as follows:

virtualenv -p python3 my_venv
source my_venv/bin/activate
cd <path-to-afl-utils>
python setup.py install
python setup.py test
afl-collect <opts>

That's the way I'm using afl-utils all the time. In case of a python package fuckup I simply do

rm -rf ./my_venv

and start from scratch.

[rc0r]

Ah, binary is /usr/local/bin/gdb. I will look at afl_collect.py too
on my previous note, recompiling gdb didnt seem to have much effect, will investigate some more.

[...]

(it still uses python 2.7)

Adapt the path to gdb in afl_collect.py:43 first. IIRC exploitable does not require Python 3 to run. As can been seen in a gdb output you posted earlier issuing exploitable only gives a warning...

If this still doesn't fix your problems, may I ask you to open a separate issue summarizing the problem you've encountered? Let's keep the PR separate from other issues. ;)

[martinlindhe]

Path to gdb is currently hardcoded in afl_collect.py.

Ah, I see!. now at least gdb is executed (can tell because osx gives notice about a process tries to control another one), however still stuck on gdb having python 2.7 so the script wont run.

Python 3.3+ has shutil.which() that solves this in a clean way, instead of hardcoding paths; https://docs.python.org/3.3/library/shutil.html#shutil.which

Regarding to not change python to python3:

1. Okay, makes sense to me. I also don't know if some systems don't provide python3 binary (i only have limited experience with the language).
   I've hit this issue with python3 tools on OSX several times before though. I will give this virtualenv a go instead, maybe I'm just trying to solve a non-issue.
2. Yea, I would also expect users of this package to be able to git clone.

Let me poke some more, maybe we can turn this PR into a OSX caution in the INSTALL.md instead.

I'm more interested in using these tools as-is, than developing them further. (found this project by the instinct of not writing my own afl triager) so I just want them to work, as I am working on other tooling for afl atm.

[martinlindhe]

If this still doesn't fix your problems, may I ask you to open a separate issue summarizing the problem you've encountered? Let's keep the PR separate from other issues. ;)

Sure, and thanks for your patient help!

[rc0r]

Python 3.3+ has shutil.which() that solves this in a clean way, instead of hardcoding paths; https://docs.python.org/3.3/library/shutil.html#shutil.which

Thanks for the hint! You might want to check the updated experimental branch.

Let me poke some more, maybe we can turn this PR into a OSX caution in the INSTALL.md instead.

Yep, updating INSTALL.md with a more verbose description of the process of using virtualenv and thus addressing potential issues regarding Python 2 vs Python 3 on different systems is a good idea.