Skip to content

Latest commit

 

History

History
1089 lines (777 loc) · 30.1 KB

awesome-python-testing.rest

File metadata and controls

1089 lines (777 loc) · 30.1 KB
Raw

Awesome Python Testing

https://en.wikipedia.org/wiki/Software_testing

https://en.wikipedia.org/wiki/Python_(programming_language)

Background

Software Testing

https://en.wikipedia.org/wiki/Software_testing

Unit, Functional, Integration, Acceptance

Workflow Evolution

1. edit, edit, commit
2. edit, commit
3. todo, edit, commit
4. todo, edit, test, commit
5. todo, test, edit, test, commit
6. todo, test, edit, test, commit, tag

7. todo, branch, test, edit, test, commit, { tag, push, send patch }
8. todo, qnew, test, edit, test, commit, finish, { tag, push, send patch }

Python

Python 3

Python 2 <--> Python 3

2to3
six
nine
future

IPython

def test_true(): assert True == True
%logstart log_input_to_here.py
%logstart -o log_input_and_output_to_here.py
%ed log_input_and_output_to_here.py

!nosetests ./log_input_and_output_to_here.py
!nosetests --help
!nosetests --ipdb   # pip install ipdbplugin # nose-progressive

# Run a file in IPython's namespace, print timing, skip sys.exit
%run -i -t -e ./log_input_and_output_to_here.py

# Run a file in IPython's namespace, print timing, skip sys.exit, and pdb
%run -i -t -e -d ./log_input_and_output_to_here.py

# Add >>> and ... prompts (useful for recording doctests)
%doctest_mode

# List defined aliases
%alias

# Get help for an alias
%logstart?
%alias?

# List available aliases and commands
%<TAB>

# Run a statement through the Python line profiler
%prun test_true()
%prun (2**2)*2)

# Time a number of loops
%timeit test_true()
%timeit (lambda x: (x**2)*2)(10)

# Run tests (see ipython_nose)
%nose

IPython Notebook

.

IPython Blocks

Logging

Python Logging Primer

A primer on Python logging (and testing):

#### Logging examples

# test setup (fixtures)
x = 2
data = {'x': x}

# asserts get compiled out with python -O2
assert x == 2

# these always run
class TestTwo(unittest.TestCase):
    def test_001_x_equals_2(self):
        self.assertEqual(self, x, 2)
unittest.main()

if x != 2:
    ### Exceptions
    err = ('...', x)
    raise Exception(err)
    raise ValueError(err)

    ### Logging
    import logging
    log = logging                       # global logger
    log = __import__('logging')         # global logger
    log = logging.getLogger(__name__)   # local logger (<__name__>.py)
    log.debug(err)
    log.info(err)
    log.error(err)
    log.exception(err)
    log.critical(err)
    log.fatal(err)
    # see: supervisor.loggers.LevelsByName and .LevelsByDescription
    TRAC = LevelsByName.TRAC = 5
    BLAT = LevelsByName.BLAT = 3
    log.log(err, TRAC)
    logging.addLevelName(TRAC, 'TRAC')
    logging.addLevelName(BLAT, 'BLAT')
    log.setLevel(TRAC)
    log.log(err, BLAT)  # not printed because msg.level < log.level

    log.log((err, 'string'), BLAT)

# run unittest.main() when run like:
#   python ./filename.py
# but not when imported like:
#   import filename
if __name__ == "__main__":
    import sys
    sys.exit(unittest.main())

See also:

structlog

logbook

Logbook is a drop-in replacement for the Python logging module with a number of simplifications and lots of additional handlers.

Logging Strategies

syslog

rsyslog
syslog-ng

sentry

raven-python

graylog

  • Elasticsearch, MongoDB, Java 8+
GELF

GELF (Graylog Extended Log Format)

graypy

logstash

ELK
  • ElasticSearch: storage, indexing, search
  • Logstash: logging adapters
  • Kibana: ElasticSearch Search UI
Elasticsearch
Logstash
Kibana
python-logstash
logstash_formatter

loggly

Test-Driven Development (TDD)

https://en.wikipedia.org/wiki/Test-driven_development

Behavior-Driven Development (BDD)

https://en.wikipedia.org/wiki/Behavior-driven_development

Mocks

https://en.wikipedia.org/wiki/Mock_object

Test Runners

Nose

nosetests --help
nosetests --with-xunit

# python -m pdb --help
nosetests --pdb
nosetests --pdb-failures
nosetests --pdb-errors

# pip install nose_ipdb
nosetests --ipdb --ipdb-failure

# pip install nose-progressive
nosetests --with-progressive

nose-ipdb

nose-ipdb is a Nose plugin for running pdb debugger with IPython (so tab completion, obj? and obj?? all work.

nose-parameterized

nose-parameterized is a Nose plugin for separating test data from test code.

See also:

nose-progressive

nose-progressive is a Nose plugin which adds a progress bar, better/different configurable tracebacks, and editor shortcuts

ipython_nose

ipython_nose is a Nose plugin for running test_* functions within the current Jupyter Notebook.

!pip install -e git+https://github.com/taavi/ipython_nose#egg=ipython_nose
%load_ext ipython_nose
def test_func():
    assert sum([1, 1]) == 2
%nose  # discover and run "test_*" functions

nosebook

  • nosebook is a Nose plugin for running test functions in Jupyter Notebooks from the CLI.

nose and pdbpp

Nose and pdbpp:

pip install pdbpp
nosetests --pdb

py.test

py.test --pdb  # --full-trace

py.test and pdbpp

Nose and pdbpp:

pip install pytest
py.test --pdb

pytest-sugar

testtools

GUI Test Runners

Tox

Continuous Integration (CI) and Continuous Delivery (CD)

Buildbot

Jenkins

xUnit XML

Travis

Version Control

Patches, Tags, Branches, Merging

Bisect, Blame

Debugging

Debugging: Console, CLI, Terminal, REPL, IPython

pdb

import pdb; pdb.set_trace()    # pdb, pdbpp

pudb

ipdb

import ipdb; ipdb.set_trace()  # ipdb
nosetests --ipdb --ipdb-failure  # --with-progressive  # nose-progressive

pdbpp

pdb++ (pdbpp) is a "drop-in replacement for pdb" with additional debugging commands, tab completion, syntax highlighting, sticky mode,

  • When pdb++ is installed (in sys.path) pdb++ overloads pdb:
pip install pdbpp funcsigs  # installs pdb.py
python -m pdb ./code.py

nosetests --pdb ./code.py
py.test --pdb ./code.py

Vim

vimpdb

vimpdb integrates vim and pdb

vim-ipython
vim-unstack
jedi-vim
python-mode
westurner/dotvim

Debugging: GUI

IDEs

Debugging: Web

Advanced Debugging

Performance Instrumentation and Profiling

Profiling

Benchmarks

Performance

Databases

Fixtures

Schema Migrations

Django Migrations

SQLAlchemy Migrations

Web Frameworks

https://en.wikipedia.org/wiki/Web_application_framework

Plone

Django

Flask

Bottle

Bottle is a one file WSGI Python web framework.

pgs is written with bottle.py:

Pyramid

Web Applications

WebTest

WebTest is a tool for testing WSGI applications without running a webserver (e.g. Pyramid, Flask, Django, Bottle)

Splinter

  • multi webdrivers (chrome webdriver, firefox webdriver, phantomjs webdriver, zopetestbrowser, remote webdriver)
  • css and xpath selectors *

ghost.py

Webkit based scriptable web browser for python.

  • PyQT or PySide

Selenium