Passthrus persist accross tests

[wilbertom]

Hello, I just ran into what I believe is a bug. Calls to responses.add_passthru persist across tests. This wouldn't be a big deal except running tests in a random order causes failures.

This works:

import re
import responses
import requests
import pytest


@responses.activate
def test_without_a_passthru():
    print(f"responses.passthru_prefixes: {responses.passthru_prefixes}")
    print(f"responses._default_mock.passthru_prefixes: {responses._default_mock.passthru_prefixes}")

    with pytest.raises(requests.exceptions.ConnectionError):
        response = requests.get('https://example.com')


@responses.activate
def test_with_a_passthru():
    print(f"responses.passthru_prefixes: {responses.passthru_prefixes}")
    print(f"responses._default_mock.passthru_prefixes: {responses._default_mock.passthru_prefixes}")

    responses.add_passthru(re.compile('.*'))
    response = requests.get('https://example.com')

    assert response.status_code == 200

This fails:

import re
import responses
import requests
import pytest

@responses.activate
def test_with_a_passthru():
    print(f"responses.passthru_prefixes: {responses.passthru_prefixes}")
    print(f"responses._default_mock.passthru_prefixes: {responses._default_mock.passthru_prefixes}")

    responses.add_passthru(re.compile('.*'))
    response = requests.get('https://example.com')

    assert response.status_code == 200


@responses.activate
def test_without_a_passthru():
    print(f"responses.passthru_prefixes: {responses.passthru_prefixes}")
    print(f"responses._default_mock.passthru_prefixes: {responses._default_mock.passthru_prefixes}")

    with pytest.raises(requests.exceptions.ConnectionError):
        response = requests.get('https://example.com')

In the failing example you can see that since we called responses.add_passthru in the first test case, the second test case fails because the passthru is still active.

Even though the passthru is still active, calling responses.passthru_prefixes returns an empty tuple but calling responses._default_mock.passthru_prefixes returns something different.

I think this is due to how it is assigned in the code. You are assigning a value, not a reference to _default_mock.passthru_prefixes but that's a quick guess without looking into the code too deep.

[knaperek]

This wouldn't be a big deal

I believe this is a big deal. One of the key principle in (unit) testing is test isolation.

What's worse, according to my testing all mocked responses persist across tests, so it totally breaks test isolation and makes the overall result depend on test execution order 👎

[iot-resister]

@knaperek stop parroting unit testing insanity. This lib is obviously for integration testing. Which mean testing some state and state means no isolation. If you need this for your 'unit' tests, then you are doing somethign wrong. your unit tests should be isolated from any http stuff. its the D in SOLID.

[knaperek]

The "unit" was intentionally enclosed with parenthesis, you can totally leave it out if it helps your understanding.
Often it comes down to the definition of unit and your mileage may vary, nevertheless, the unittest standard Python library is not exclusively targeted for "unit" tests only and in fact it is very often used for integration tests as well (example: Django's TestCase).

The distinction between unit and integration (or other) types of tests is irrelevant in this context though. Test isolation is a very important aspect of any type of automated tests. The need to preserve state in an integration test only lasts until that test completes (successfully or not). You certainly don't want that state to leak into other tests that are being executed in the same test run as a part of the same (or different) test suite.

The problem that I observed with this library is that fake responses defined for one test leak into other tests as well. That's a bugger, because it breaks test isolation, which in turn breaks determinism and independence on the order of execution. I believe you want to have your integration test results independent on their order of execution, right?

[iot-resister]

nah. I'm trying to do lifecycle stuff.

[nsotelo]

What's worse, according to my testing all mocked responses persist across tests, so it totally breaks test isolation and makes the overall result depend on test execution order

My case is even worse, the mocked responses are leaking into tests that aren't decorated with responses.activate, although I haven't been able to distil this into a minimal example yet.

UPDATE: I've figured out what was causing this, I had functions wrapped in @responses.activate calling other such decorated functions. Removing the innermost decorators solved the problem.