Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tidy up, and add harness for examples #27

Merged
merged 4 commits into from Jan 13, 2020

Conversation

@gerph
Copy link
Contributor

gerph commented Jan 12, 2020

This change adds a harness for the examples so that they can be run as part of automation, or to check what the performance is in a simple way. The changes are:

  • Rename the csv example to zipfile, as that's what it tests
  • Update the examples so that they can run on Python 2 (when the rest of the library is capable of doing so).
  • Create a harness that allows the examples to be run through the fuzzer. This harness reports on what it found from the results, which can be useful in particular to report on the speed of all of the examples if performance is a concern.

The harness output looks like this:

charles@laputa ~/external/pythonfuzz (add-harness-for-examples)> python3 examples/run_all_examples.py
Looking in examples for examples
Example: codeop
  Executions       : 1000
  Corpus           : 7
  Coverage         : 360
  Final speed      : 653/s
  Memory           : 36.47 MB
  Runtime          : 1.99 s
  Overall speed    : 502.66/s
  Return code      : 0
Example: zipfile
  Executions       : 1000
  Corpus           : 2
  Coverage         : 106
  Final speed      : 452/s
  Memory           : 34.92 MB
  Runtime          : 2.10 s
  Overall speed    : 476.58/s
  Return code      : 0
Example: htmlparser
  Executions       : 1000
  Corpus           : 7
  Coverage         : 442
  Final speed      : 695/s
  Memory           : 35.75 MB
  Runtime          : 1.77 s
  Overall speed    : 566.00/s
  Return code      : 0
Example: zlib
  Executions       : 1000
  Corpus           : 2
  Coverage         : 35
  Final speed      : 385/s
  Memory           : 34.83 MB
  Runtime          : 1.33 s
  Overall speed    : 752.01/s
  Return code      : 0
Example: xml
  Executions       : 1000
  Corpus           : 4
  Coverage         : 69
  Final speed      : 852/s
  Memory           : 35.21 MB
  Runtime          : 1.47 s
  Overall speed    : 679.20/s
  Return code      : 0
Example: fuzzit
  Executions       : 1000
  Corpus           : 4
  Coverage         : 63
  Final speed      : 1075/s
  Memory           : 34.86 MB
  Runtime          : 1.22 s
  Overall speed    : 821.51/s
  Return code      : 0
Example: aifc
  Executions       : 1000
  Corpus           : 4
  Coverage         : 101
  Final speed      : 523/s
  Memory           : 35.07 MB
  Runtime          : 1.81 s
  Overall speed    : 553.42/s
  Return code      : 0

The number of runs by default is 1000, which is poor for testing performance, but command line option can change that as needed by the user.

gerph added 4 commits Jan 11, 2020
The examples/csv/fuzz.py is actually testing the zipfile module,
so it has been renamed to reflect this. There are slight differences
in the Python 2 usage, which I've harmonised in the module.
Some simple checks for the behaviour of the modules under Python
2 compared to Python 3.
The example fuzzers provide a useful way to see how you might use
the pythonfuzz package. However, they're no use if they don't work.
Adding a tool that is able to run them all, and report on their
status will ensure that these do not get broken in the future.

The tool is able to be configured with the number of runs that it
expects to execute, and whether it will keep any crash or timeout
files that are generated. At the end of the run of each example,
it will report the collected information from the run. This should
be useful for checking the performance in the future, as well.

If any of the examples fails to function - ie it reports an error
itself, then the runner will exit with a non-0 return code. This
will allow it to be used as a gate for the examples being functional.
…ess.

In order to have an easy way to control which python runs the examples,
make the python version used into the one used by the harness itself.
That way if you invoke it with Python 3, it'll test with Python 3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.