2017-02-02-fixtures.md

Title: Basic pytest Fixtures
Tags: Python Unit Tests at Hypothesis
Alias: /post/fixtures/

This post covers the basics of **pytest fixtures** and how we use them at Hypothesis.

A "test fixture" is some code that needs to be run in order to set things up
for a test. When we created the objects that we needed to pass into the
method we were testing in the _arrange_ part of the
[arrange, act, assert recipe](/posts/arrange-act-assert), those were fixtures.
[Test factories](/posts/factories) are helpers for easily creating fixtures.
Fixtures don't have to be objects that we pass into the method we're testing
as arguments, they can be anything that needs to be set up for a test.
For example if we want to test what a method does when there are three
documents in the database, setting up the db with those three documents is a
fixture.

Setup and teardown methods
--------------------------

Most unit testing frameworks, for example
[the standard unittest library that ships with Python](https://docs.python.org/2/library/unittest.html),
use the familiar `setup()` and `teardown()` methods to set up all the necessary
fixtures before running a test method and, if necessary, tear things down after
running the test (before running the next test). The test code might look
something like this:

```python
class MyTestClass(object):
    def setup(self):
        self.fixture_object = SomeClass()

    def test_something(self):
        # Some test code that uses self.fixture_object
```

The test framework runs the `setup()` method once before it runs each test
method (and the `teardown()` method, if there is one, gets run once after each
test).

The drawbacks of `setup()` methods begin to show when a test class contains
different test methods that use different fixture objects or setup:

```python
class MyTestClass(object):
    def setup(self):
        self.first_fixture_object  = SomeClass()
        self.second_fixture_object = SomeOtherClass()

    def test_something(self):
        # Some test code that uses self.first_fixture_object

    def test_something_else(self):
        # Some test code that uses self.second_fixture_object

    ...
```

Some of the test methods only need `first_fixture_object`, and some
only need `second_fixture_object`, but _both_ fixture
objects get created for each test. In a small example this isn't important but
over a large test base unnecessarily running a lot of unused fixtures for a lot
of tests can start to add significantly to how long it takes to run the tests
every time any developer needs to do so.

As they get bigger and more complicated tests like this can also become hard
to read. A whole lot of setup is done in the `setup()` method, a lot of which
may be depended upon by the test method that you're currently trying to
understand, and a lot of it not. You have to scan the whole test method body
looking for all the `self.something`s to see what fixtures this test depends
on, and then look in `setup()` to see what each of these fixtures is.

Worse, if a fixture sets up something in the environment, for example adding
some objects to the test database, then a test method may be depending on that
fixture (assuming that certain objects exist in the db) without explicitly
referencing that fixture in the test body at all.

It can quickly get to the point where `setup()` code can be difficult to change
without potentially breaking multiple test methods for unclear reasons, and
where it can become hard to tell exactly what a given test method is really
testing.

The problem gets worse when different test fixtures conflict with each other.
This can happen when there's a singleton resource such as a database.
If one test needs to test what happens when there are no users in the database,
and another needs to test what happens when there are three users in the
database, well a `setup()` method can't setup the database with zero users in
it and also setup the same database with three users in it as the same time.
Hacking around this kind of issue can lead to hard to read code and some nasty
coupling between test methods.

One solution would be to break the methods up into multiple test classes.
You think of each test class as its own fixture. If some tests need no users
and one group in the db then that's one fixture and one test class, if some
other tests need to test what happens when there's three users and no groups in
the db then that's another class:

```python
class TestWithNoUsersAndOneGroup(object):
    def setup(self):
        # Add one group to the test db.
        ...

    def test_something(self):
        ...

    def test_something_else(self):
        ...

class TestWithThreeUsersAndNoGroup(object):
    def setup(self):
        # Add three users to the test db.
        ...

    def test_another_thing(self):
        ...
```

You may end up with some code duplication between the different `setup()`
methods, but that can always be moved into helper methods that each of the
`setup()` methods calls.

Clearly coming up with good names for the test classes can get pretty awkward
if the different fixtures that they represent become complex and varied.

In practice, test classes hardly ever get written like this.
Most commonly you see test classes named things like `TestFoo` containing all of
the tests for the `Foo` class (or perhaps the `foo()` method),
and this is how test classes are mostly used in h as well - to group together
the tests for a given unit of code that's being tested.

Another alternative is to use test helper methods...

Test helper methods
-------------------

The problems with `setup()` methods can be worked around by simply _not using_
`setup()` methods or `self`, and instead using simple helper methods -
non-test methods in the test classes or modules that are called by each test
method as needed:

```python
class MyTestClass(object):
    def add_one_group_to_the_db(self):
        # Code that adds one group to the db and returns the group.

    def add_three_users_to_the_db(self):
        # Code that adds three users to the db and returns the users/

    def test_something(self):
        group = add_one_group_to_the_db()
        ...

    def test_something_else(self):
        group = add_one_group_to_the_db()
        ...

    def test_another_thing(self):
        users = add_three_users_to_the_db()
        ...
```

This works just fine:

* Each test calls only the helper methods that test needs.
  No helper methods are run unnecessarily for any tests that don't need them.

* It's easy to look at a test and see exactly what setup is done for that test
  and what isn't, no more implicit dependencies on things that are done or
  not done in a `setup()` method.

* The tests are independent from each other, modifying a test or adding
  a new test isn't going to break existing tests. If a new test wants to
  change what a helper method does, but doing so would break existing tests,
  you can always just add a new helper method and call that instead.

But simple helper methods like this do have a couple of drawbacks:

* Each test needs to call each helper method that it needs, which clutters up
  the test bodies with a lot of repetitive, boilerplate code (this gets worse
  when helper methods require arguments to be passed to them).

* There isn't a good way to do any tearing down of something that was set up
  by a call to a helper method (you wouldn't want each test method to have
  to call corresponding teardown methods).

Pytest comes with a feature that it calls pytest fixtures that solve this
problem very nicely. Pytest fixtures are like test helper methods on
steroids...

pytest fixtures
---------------

[pytest's fixtures](http://docs.pytest.org/en/latest/fixture.html) are its
answer to `setup()` and `teardown()` methods (pytest actually
[supports `setup()` and `teardown()` as well](http://docs.pytest.org/en/latest/xunit_setup.html)
but fixtures are better). Pytest fixtures are a form of
[dependency injection](https://en.wikipedia.org/wiki/Dependency_injection) that
allow you to do something very similar to what you can do with test helper
methods but without the drawbacks.

An example of a simple fixture is [h's `pyramid_request` fixture](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/h/conftest.py#L224):

```python
@pytest.fixture
def pyramid_request():
    """Dummy Pyramid request object."""
    request = testing.DummyRequest()
    request.auth_domain = text_type(request.domain)
    request.create_form = mock.Mock()
    request.matched_route = mock.Mock()
    request.is_xhr = False
    return request
```

[Pyramid](https://trypyramid.com/) is the web framework that h uses to receive
HTTP requests from users' web browsers and send back HTTP responses to those
requests. [The request object](http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/webob.html?highlight=request)
is the object that Pyramid makes available to our code that represents the HTTP
request that we're currently responding to. Many methods in h require the
request object as an argument, so tests often need a request object to pass
to the method they're testing.

The `pyramid_request()` function above creates a [DummyRequest](http://docs.pylonsproject.org/projects/pyramid/en/latest/api/testing.html?highlight=dummyrequest#pyramid.testing.DummyRequest)
object, sets various fields on the request that are needed for h, and returns
it.

The `@pytest.fixture` decorator on the top of the function registers it with
pytest as a fixture function, which means that a test can make use of it by
simply taking it as an argument. For example, let's look
at [a test for the `paginate()` function](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/h/paginator_test.py#L13):

```python
def test_current_page_defaults_to_1(pyramid_request):
    """If there's no 'page' request param it defaults to 1."""
    pyramid_request.params = {}

    page = paginate(pyramid_request, 600, 10)

    assert page['cur'] == 1
```

[`paginate()`](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/h/paginator.py#L11)
is a function that takes the request and returns some data, including the
current page, that's used to render this pagination control at the bottom of
search results pages:

<img src="{static}/images/paginator.png">

The test needs a Pyramid request argument in order to call the paginator
function, so it adds the `pyramid_request` fixture to its arguments:

<pre><code>def test_current_page_defaults_to_1(self, <strong>pyramid_request</strong>):
    ...</code></pre>

When pytest comes to run this test it'll see the `pyramid_request` argument,
it knows that this matches up to an `@pytest.fixture` function, so it first
calls the `pyramid_request()` fixture function and then passes
**the value that the fixture function returns** to the test as the
`pyramid_request` argument. What pytest does is something like this:

```python
pyramid_request_fixture = pyramid_request()
test_current_page_defaults_to_1(pyramid_request=pyramid_request_fixture)
```

The `pyramid_request()` fixture returns a Pyramid `DummyRequest` object all set
up for use in h, and pytest passes that `DummyRequest` to the test as the
`pyramid_request` argument. It's as simple as that - all the test has to do
is have an argument with the same name as the fixture function and it gets the
request object passed right into it.

At its core, that's all a pytest fixture is - a perfectly normal Python
function that pytest runs before running a test, only if that test has the
fixture as an argument. An `@pytest.fixture` decorator is needed on the top of
the function to register it with pytest as a fixture.

Fixtures are no different to test helper methods, except the boilerplate of
each test needing to call every helper method that it needs is avoided.
It's not often needed, but a fixture can also contain teardown code to be
called after any test that used that fixture simply by
[using yield instead of return](http://docs.pytest.org/en/latest/fixture.html?highlight=finalizer#fixture-finalization-executing-teardown-code).


Lots of different tests can reuse the same fixture
--------------------------------------------------

Just like different tests can call the same helper method, once you've
written a fixture once you can use it in as many tests as you want just by
adding it as an argument to each test:

```python
@pytest.fixture
def pyramid_request():
    ...

def test_current_page_defaults_to_1(pyramid_request):
    ...

def test_numbers_large_result_set(pyramid_request):
    ...

def test_numbers_small_result_set(pyramid_request):
    ...
```

One test can use lots of different fixtures
-------------------------------------------

And of course just like a test can call multiple helper methods, a test can
also use multiple fixtures by just having all of the fixture names as
arguments.

In [the factories post](/posts/factories) we looked at the `factories` object
which can be used to easily create realistic objects for use in tests.
[`factories` is a fixture](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/h/conftest.py#L144)
that's used by any test that wants to use it to create test objects.

[`pyramid_settings`](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/h/conftest.py#L243)
is another fixture that returns the settings used to configure the h app in the
test environment.

A test method can use all three fixtures, `pyramid_request`, `factories` and
`pyramid_settings` just by taking all three as arguments by name:

```python
def test_that_uses_multiple_fixtures(pyramid_request, factories, pyramid_settings):
    ...
```

Fixtures don't have to return anything
--------------------------------------

Sometimes a test just needs a fixture function to be run before the test,
because the test depends on some setup that the fixture function does,
but the test doesn't actually need to use the value that the fixture function
returns.
A fixture function doesn't even have to return anything, it might just do some
setup that's needed by some tests, for example creating some entries in the
database, and return nothing. If a test wants that fixture function to be run
before it then it adds the fixture as an argument, if it doesn't then it
doesn't. A good example of this is the `routes` fixture, which is found in
many places in the h tests.

**Routes** are
[Pyramid's way of deciding which method it should call to handle a request for a given URL](http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/urldispatch.html). For example if a user loads [the /users/seanh page](https://hypothes.is/users/seanh)
then Pyramid calls the [`h.views.activity.UserSearchController.search()` method](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/h/views/activity.py#L278)
to generate the response. [The routes.py file](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/h/routes.py)
tells Pyramid how to do this by configuring which URLs map to which methods.
Routes can also be used in reverse, to _generate_ URLs. For example if some
code wants the URL for the user search page for the "seanh" user then it can
get it from [the route_url() method](http://docs.pylonsproject.org/projects/pyramid/en/latest/api/request.html?highlight=route_url#pyramid.request.Request.route_url)
by doing `request.route_url("activity.user_search", username="seanh")`, which
will return `"https://hypothes.is/users/seanh"`.

Many methods in h call methods like `route_url()` and others that require all
the routes to be setup in order to work. If the routes aren't configured then
calls to `route_url()` will crash with some sort of "unknown route" error.
So the tests need some code to setup the routes before the test.
This is what a `routes` fixture does, for example:

```python
@pytest.fixture
def routes(self, pyramid_config):
    pyramid_config.add_route('activate', '/activate/{id}/{code}')
```

The fixture doesn't return anything, but it adds the `'activate'` route to the
Pyramid config so that any calls to `request.route_url("activate", ...)` will
now work. A test can use this fixture and then test a function that uses this
route without having it crash:

```python
def test_something(routes):
    result = some_function_that_uses_the_activate_route()

    assert result == <something>
```

@pytest.mark.usefixtures
------------------------

The test above doesn't actually use the `routes` argument in the body of the
test, and the value of `routes` is just `None` anyway, it's just there to tell
pytest to call the `routes()` function before it calls `test_something()`.

[`@pytest.mark.usefixtures()`](http://docs.pytest.org/en/latest/fixture.html?highlight=usefixtures#usefixtures)
is an alternative way for a test to use a fixture when it just needs the fixture
function to be run, but doesn't actually need the fixture value in the test
body:

```python
@pytest.mark.usefixtures('routes')
def test_something():
    result = some_function_that_uses_the_activate_route()

    assert result == <something>
```

`@pytest.mark.usefixtures("routes")` tells pytest to run the `routes` fixture
before this test, just the same as `routes` as an argument would do.

To use multiple fixtures this way, just pass multiple strings to `usefixtures()`:

```python
@pytest.mark.usefixtures("routes", "pyramid_request", "factories", "pyramid_settings")
```

Where to find fixtures
----------------------

When you see test methods that have arguments in h,
unless the test is using [parametrize](/posts/parametrize),
those arguments are probably fixtures.
But where do you find the fixture functions for the fixtures that a test uses?
And where should you put any new fixtures that you want to write for your tests?
Fixtures can be defined in a number of places:

1. pytest comes with some [builtin fixtures](http://pytest.org/dev/builtin.html#builtin-fixtures-function-arguments).

2. In a `conftest.py` file such as [`tests/h/conftest.py`](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/h/conftest.py)
   (for the tests in `tests/h`) or [`tests/memex/conftest.py`](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/memex/conftest.py)
   for the tests in `tests/memex`.

3. Fixture functions can go in the the test modules themselves.
   You'll see this all over the place in the h tests, the fixture functions
   usually all go at the bottom of the test file and in alphabetical order.
   For example, here's [a bunch of fixtures defined at the bottom of `views_test.py`](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/memex/views_test.py#L547).

    Fixtures defined in a test file are only available to the tests in that file.

    A fixture in a test file will override any fixture with the same name in a
    `conftest.py` file.

4. Fixture functions can go in test _classes_. A fixture defined in a class is
   only available to the tests in that class, and overrides any fixture with
   the same name defined outside of the class or in a `conftest.py` file.

    We use this a lot in h as well, if a fixture is only needed by the tests in
    one test class then we define the fixture as a method of that test class.
    It keeps related code together, reducing noise and travel. It also allows
    other test classes to have their own fixture with the same name without
    conflicting.
   
    Like any class method these fixtures need to take `self` as their first
    argument. We usually put all the fixtures at the bottom of the class and in
    alphabetical order. For example, here's
    [some fixtures defined at the bottom of the TestAddApiView class](https://github.com/hypothesis/h/blob/4ddc0029046a6288bc02fabff03103e300a964ce/tests/memex/views_test.py#L108).

**Tip**: You can find the definition of a fixture by using the `--fixtures`
argument to `pytest`, for example:

```console
$ tox -e py27-memex tests/memex/ -- --fixtures
```

This prints out the names of all of the available fixtures and the line and file at
which each one is defined. You can grep the output of this command to find a
particular fixture by name:

```console
$ tox -e py27-memex tests/memex/ -- --fixtures | grep '^annotation'
```

If you're only interested in the fixtures available to a particular test
module, class or method you can drill down:

```console
$ tox -e py27-h \
    tests/h/views/activity_test.py::TestSearchController::test_search_checks_for_redirects \
    -- --fixtures
```

Conclusion
----------

So far we've seen that fixtures are a more convenient alternative to test
helper methods, with less boilerplate required and with a good way to do
teardown code. You define each fixture in a separate function, but rather than
each test needing to call each helper method that it wants the tests can just
receive the fixture objects as arguments.

Pytest fixtures also have a couple of advanced features that take them well
beyond what simple helper methods can do. We'll cover some of those in
[Advanced pytest Fixtures](2017-02-12-advanced-fixtures.md).