Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
106 lines (83 sloc) 3.05 KB

deflake.py

Helps debug a non determinate test (or any flaky program) by running it until it exits with a non-zero exit code.

deflake can be run on the command-line or imported as a module. See below for more details. At the command-line, run python deflake.py --help for a list of options. Default maximum runs is 10. Default pool-size is 1.

Install

$ pip install deflake

Example Running as Script

deflake will exit with 1 if any processes return non-zero.

# up to 25 times until failure
$ deflake "my_flaky_program arg1 arg2"
PASS
PASS
PASS
FAIL (run 4)
$ echo $?
1
$

Example as Python Class

In this case we use the Deflake class, trying to deflake ls. ls is pretty stable, so we don't expect a non-zero exit status. We should see PASS ten times (the default maximum runs) until deflake gives up. When running as a class, the run method runs the processes, and returns a list with the output from calling run.

>>> from deflake import Deflake
>>> d =Deflake("ls", quiet=True)
<deflake.Deflake object at 0x7fd1b40d76d0>
>>> results = d.run()
['PASS', 'PASS','PASS','PASS','PASS','PASS','PASS','PASS','PASS','PASS']
>>>

Multiprocessing

Use the pool-size or -p option to run your program in concurrent pools of processes. The default is no multiprocessing or a pool size of 1. Setting a pool-size to > 1 can make your deflaking faster, but it won't always work, depending on the program you are trying to deflake. If your program writes to a file, for example, then multiple processes may try to write to that file simultanously and render your deflaking, well, flaky. See below for a possible solution.

If the program you're deflaking writes to predetermined files (for example a log file), and you want to multiprocess using the pool-size option, deflake might report a failure due to multiple processes trying to write to the same file. In this case you can use the special #count# replacement token to change the name of the log file each process writes to. If you need to change the default #count# token to something else, use the --counter-token or -c option. Whatever replacement token you use, it will be replaced with the iterator used when looping through the processes. For example:

$ deflake --max-runs 5 --pool-size 4 'my_flaky_program --log-file log#count#.txt'

The previous run would generate::

  • log1.txt
  • log2.txt
  • log3.txt
  • log4.txt
  • log5.txt

Developing

To work on this package:

  1. Clone the repo.
  2. pip install -e .
  3. Test: python tests/test.py.
  4. Fix/add stuff.
  5. Test. $ pip install nose. Then, cd tests, and $ nosetests -vs tests/test.py.
  6. Increment version in setup.py
  7. Submit pull request
  8. Tag $ git tag -a x.x.x -m "Description"
  9. $ git push --tags

To submit to pypi, make sure your .pypirc is set up:

[distutils]
index-servers =
    python-local-repo
    pypi

[pypi]
username:USERNAME
password:PASS

Also, increment the version in setup.py

Then:

python setup.py sdist upload -r pypi

Known Issues

  • Extra processes kicked off in Python3x