Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update makefile to use examples harness #30

Merged

Conversation

@gerph
Copy link
Contributor

gerph commented Jan 12, 2020

The Makefile's default target has been updated so that it now requires
that the 'system' level tests build. This is one of 3 new targets that
have been added:

  • test_level_unittests, which should cause the unit tests to run.
    We don't have any yet, but some will be added shortly.

  • test_level_integration, which causes the integration tests to run.
    These are the tests in the 'tests' directory, which exercise the
    external interface in a test-like environment to see that it
    functions as desired.
    This level is dependant on the unittests passing before it will
    be run.

  • test_level_system, which causes the examples to be run.
    These examples are the sorts of programs that a user might write,
    and which we expect to be invoked from the command and to do
    their job.
    This level is dependant on the integration tests passing before
    it will be run.

In CI, the individual test targets (rather than the levels) would
probably be invoked, with each one still dependant on the earlier ones
in stages (or maybe just run them all in parallel, as they're all very
fast).

@yevgenypats
Copy link
Contributor

yevgenypats commented Jan 13, 2020

PR #27 and #28 are merged now. can I review this one now?

…ests.

The Makefile's default target has been updated so that it now requires
that the 'system' level tests build. This is one of 3 new targets that
have been added:

  - test_level_unittests, which should cause the unit tests to run.
    We don't have any yet, but some will be added shortly.

  - test_level_integration, which causes the integration tests to run.
    These are the tests in the 'tests' directory, which exercise the
    external interface in a test-like environment to see that it
    functions as desired.
    This level is dependant on the unittests passing before it will
    be run.

  - test_level_system, which causes the examples to be run.
    These examples are the sorts of programs that a user might write,
    and which we expect to be invoked from the command and to do
    their job.
    This level is dependant on the integration tests passing before
    it will be run.

In CI, the individual test targets (rather than the levels) would
probably be invoked, with each one still dependant on the earlier ones
in stages (or maybe just run them all in parallel, as they're all very
fast).
@gerph
Copy link
Contributor Author

gerph commented Jan 13, 2020

Hmm. Looks like I created this branch wrongly. It's god the python2 bits in it, but shouldn't have.
I'll force push with the one change it was meant to have in it.

@gerph gerph force-pushed the gerph:update-makefile-to-use-examples-harness branch from 13dd4c5 to da4bb4f Jan 13, 2020
@gerph
Copy link
Contributor Author

gerph commented Jan 13, 2020

I think this is right now; makefile runs the mutator examples:

charles@laputa ~/external/pythonfuzz (update-makefile-to-use-examples-harness↑)> make

+++ Check that we can install the product
source venv/python3/bin/activate && python setup.py install
running install
running bdist_egg
running egg_info
writing pythonfuzz.egg-info/PKG-INFO
writing dependency_links to pythonfuzz.egg-info/dependency_links.txt
writing requirements to pythonfuzz.egg-info/requires.txt
writing top-level names to pythonfuzz.egg-info/top_level.txt
reading manifest file 'pythonfuzz.egg-info/SOURCES.txt'
writing manifest file 'pythonfuzz.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.14-x86_64/egg
running install_lib
running build_py
copying tests/test_nocrash.py -> build/lib/tests
copying tests/test_crash.py -> build/lib/tests
copying pythonfuzz/fuzzer.py -> build/lib/pythonfuzz
copying pythonfuzz/main.py -> build/lib/pythonfuzz
copying pythonfuzz/corpus.py -> build/lib/pythonfuzz
creating build/bdist.macosx-10.14-x86_64/egg
creating build/bdist.macosx-10.14-x86_64/egg/tests
copying build/lib/tests/test_nocrash.py -> build/bdist.macosx-10.14-x86_64/egg/tests
copying build/lib/tests/__init__.py -> build/bdist.macosx-10.14-x86_64/egg/tests
copying build/lib/tests/test_crash.py -> build/bdist.macosx-10.14-x86_64/egg/tests
copying build/lib/tests/unittest_mutators.py -> build/bdist.macosx-10.14-x86_64/egg/tests
creating build/bdist.macosx-10.14-x86_64/egg/pythonfuzz
copying build/lib/pythonfuzz/__init__.py -> build/bdist.macosx-10.14-x86_64/egg/pythonfuzz
copying build/lib/pythonfuzz/fuzzer.py -> build/bdist.macosx-10.14-x86_64/egg/pythonfuzz
copying build/lib/pythonfuzz/dictionary.py -> build/bdist.macosx-10.14-x86_64/egg/pythonfuzz
copying build/lib/pythonfuzz/main.py -> build/bdist.macosx-10.14-x86_64/egg/pythonfuzz
copying build/lib/pythonfuzz/corpus.py -> build/bdist.macosx-10.14-x86_64/egg/pythonfuzz
byte-compiling build/bdist.macosx-10.14-x86_64/egg/tests/test_nocrash.py to test_nocrash.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/tests/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/tests/test_crash.py to test_crash.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/tests/unittest_mutators.py to unittest_mutators.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/pythonfuzz/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/pythonfuzz/fuzzer.py to fuzzer.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/pythonfuzz/dictionary.py to dictionary.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/pythonfuzz/main.py to main.cpython-37.pyc
byte-compiling build/bdist.macosx-10.14-x86_64/egg/pythonfuzz/corpus.py to corpus.cpython-37.pyc
creating build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying pythonfuzz.egg-info/PKG-INFO -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying pythonfuzz.egg-info/SOURCES.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying pythonfuzz.egg-info/dependency_links.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying pythonfuzz.egg-info/requires.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
copying pythonfuzz.egg-info/top_level.txt -> build/bdist.macosx-10.14-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/pythonfuzz-1.0.3-py3.7.egg' and adding 'build/bdist.macosx-10.14-x86_64/egg' to it
removing 'build/bdist.macosx-10.14-x86_64/egg' (and everything under it)
Processing pythonfuzz-1.0.3-py3.7.egg
Removing /Users/charles/external/pythonfuzz/venv/python3/lib/python3.7/site-packages/pythonfuzz-1.0.3-py3.7.egg
Copying pythonfuzz-1.0.3-py3.7.egg to /Users/charles/external/pythonfuzz/venv/python3/lib/python3.7/site-packages
pythonfuzz 1.0.3 is already the active version in easy-install.pth

Installed /Users/charles/external/pythonfuzz/venv/python3/lib/python3.7/site-packages/pythonfuzz-1.0.3-py3.7.egg
Processing dependencies for pythonfuzz==1.0.3
Searching for numpy==1.17.3
Best match: numpy 1.17.3
Processing numpy-1.17.3-py3.7-macosx-10.14-x86_64.egg
numpy 1.17.3 is already the active version in easy-install.pth
Installing f2py script to /Users/charles/external/pythonfuzz/venv/python3/bin
Installing f2py3 script to /Users/charles/external/pythonfuzz/venv/python3/bin
Installing f2py3.7 script to /Users/charles/external/pythonfuzz/venv/python3/bin

Using /Users/charles/external/pythonfuzz/venv/python3/lib/python3.7/site-packages/numpy-1.17.3-py3.7-macosx-10.14-x86_64.egg
Searching for psutil==5.6.3
Best match: psutil 5.6.3
Processing psutil-5.6.3-py3.7-macosx-10.14-x86_64.egg
psutil 5.6.3 is already the active version in easy-install.pth

Using /Users/charles/external/pythonfuzz/venv/python3/lib/python3.7/site-packages/psutil-5.6.3-py3.7-macosx-10.14-x86_64.egg
Searching for coverage==4.5.4
Best match: coverage 4.5.4
Processing coverage-4.5.4-py3.7-macosx-10.14-x86_64.egg
coverage 4.5.4 is already the active version in easy-install.pth
Installing coverage script to /Users/charles/external/pythonfuzz/venv/python3/bin
Installing coverage-3.7 script to /Users/charles/external/pythonfuzz/venv/python3/bin
Installing coverage3 script to /Users/charles/external/pythonfuzz/venv/python3/bin

Using /Users/charles/external/pythonfuzz/venv/python3/lib/python3.7/site-packages/coverage-4.5.4-py3.7-macosx-10.14-x86_64.egg
Finished processing dependencies for pythonfuzz==1.0.3

+++ Install the test requirements
source venv/python3/bin/activate && pip install -r requirements-test.txt
Ignoring mock: markers 'python_version < "3.3"' don't match your environment

+++ Running unit tests

+++ Unit tests passed (we don't have any yet)

+++ Running integration tests
source venv/python3/bin/activate && cd tests && python -munittest -v test_crash test_nocrash
test_find_crash (test_crash.TestFindCrash) ... ok
test_find_crash (test_nocrash.TestFindCrash) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.215s

OK

+++ Integration tests passed

+++ Running system tests
source venv/python3/bin/activate && examples/run_all_examples.py --runs 1000
Looking in examples for examples
Example: codeop
  Executions       : 1000
  Corpus           : 5
  Coverage         : 89
  Final speed      : 853/s
  Memory           : 35.00 MB
  Runtime          : 1.65 s
  Overall speed    : 606.79/s
  Return code      : 0
Example: zipfile
  Executions       : 1000
  Corpus           : 2
  Coverage         : 106
  Final speed      : 469/s
  Memory           : 35.03 MB
  Runtime          : 1.90 s
  Overall speed    : 525.55/s
  Return code      : 0
Example: htmlparser
  Executions       : 1000
  Corpus           : 4
  Coverage         : 110
  Final speed      : 779/s
  Memory           : 35.50 MB
  Runtime          : 1.62 s
  Overall speed    : 617.58/s
  Return code      : 0
Example: zlib
  Executions       : 1000
  Corpus           : 2
  Coverage         : 35
  Final speed      : 442/s
  Memory           : 34.45 MB
  Runtime          : 1.17 s
  Overall speed    : 851.09/s
  Return code      : 0
Example: xml
  Executions       : 1000
  Corpus           : 4
  Coverage         : 69
  Final speed      : 830/s
  Memory           : 35.34 MB
  Runtime          : 1.41 s
  Overall speed    : 708.36/s
  Return code      : 0
Example: fuzzit
  Executions       : 1000
  Corpus           : 4
  Coverage         : 63
  Final speed      : 1160/s
  Memory           : 34.89 MB
  Runtime          : 1.16 s
  Overall speed    : 863.94/s
  Return code      : 0
Example: aifc
  Executions       : 1000
  Corpus           : 4
  Coverage         : 93
  Final speed      : 773/s
  Memory           : 35.25 MB
  Runtime          : 1.69 s
  Overall speed    : 592.03/s
  Return code      : 0

+++ System tests passed

+++ All tests passed
@gerph gerph marked this pull request as ready for review Jan 13, 2020
@gerph gerph changed the title WIP Update makefile to use examples harness Update makefile to use examples harness Jan 13, 2020
@yevgenypats yevgenypats merged commit 4dce5aa into fuzzitdev:master Jan 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.