A demonstration of literal testing ideas and techniques.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Literal Testing Example

A demonstration of literal testing techniques.


By literal testing, I mean writing documents which also test your project, and tests which document your project. It's actually a simple idea, and that's why this example project exists -- to demonstrate that it is indeed simple.

Python, doctest, and doctest blocks

We're using Python 2.x.

Everyone in Python should know about doctest. It's part of Python's standard library and has been for a long time. Not everyone knows...

  • about ReStructuredText,
  • how you can embed doctest blocks in ReStructuredText documents,
  • how you can build beautiful documents from simple structured text,
  • and how test runners will discover doctest blocks in these text documents.

It's quite elegant indeed.

Getting Started


Just fire up a virtualenv, and proceed.

Running Tests

nose is one test runner which supports doctest:

python setup.py nosetests

This also installs required test libraries, thanks to tests_require in setup.py.

Testing with this Document

Through setup.cfg, nose is configured to look for doctests in .rst files, including this one. Conveniently, a doctest block in one area of the document:

>>> spam = 'eggs'

is available in later doctest blocks in the same document:

>>> spam is not None
>>> spam == 'eggs'

Do clean up after yourself, unless you have a good reason for an object to persist throughout the document to simplify testing.

>>> del spam

Testing with Mock Objects

When working with another service, sometimes you're better off mocking the interaction with that service than to try to test it directly. For this, there are mock libraries for Python. Let's use MiniMock:

>>> from minimock import Mock, mock

You see what we did there? In addition to showing you how to get started with MiniMock, you can now call mock and Mock in your doctest blocks, which you can write throughout this document. Here's a simple example of how it works (see the MiniMock documentation for more details and better examples). Imagine we are using a function called spam in our library, and our simple function looks like this:

>>> def eat_spam(count=0):
...     [spam('spam') for i in range(count)]

The function spam by itself might have a definition and usage like this:

>>> def spam(eggs):
...     print 'Do something with %s that takes a long time...' % eggs
>>> spam('this string')
Do something with this string that takes a long time...

Since spam takes a long time, we'd rather test using a mock version of it and trust that the spam developers have good test coverage. (Really, you should validate that trust... get the source!)

>>> spam = Mock('spam')

Now when we are testing our usage of spam in our application, we are using the mock version, not the live one.

>>> eat_spam()
>>> eat_spam(1)
Called spam('spam')
>>> eat_spam(3)
Called spam('spam')
Called spam('spam')
Called spam('spam')

Note that MiniMock is simply printing that spam was called, and it was called with a single argument 'spam'. The actual spam function is never called (there is no "Do something with spam that takes a long time..."). That's what MiniMock does, and doctest will verify for us that spam is called as we expect.

Consider it whitebox or glassbox testing. This approach goes under the covers a bit to test how eat_spam is calling spam, but if spam were truly something complicated to test, then this test approach goes a long way without requiring sophisticated test setup. In using this approach, it's up to you to keep this document up to date when changing details on how the mocked interfaces are being used. The mock interfaces in this document serve as tests, but also as documentation on intended use of dependencies. If all goes well, it should be obvious when the docs are out-of-sync with the implementation, since you will be running tests against this doc before using any code in the project.

Before we forget, let's clean up this section.

>>> del spam, eat_spam

Notice we kept mock and Mock around. That's so you can use them later in this document to write good tests.