Using requests-cache data in unit tests

[JWCook]

Related issues: #87, #158, #169, and #244.

requests-cache is quite feature-rich when it comes to recording response data, but doesn't have any features for applying that data to common testing patterns. It's mainly intended for use in applications, scripts, etc., but it may be an appealing option to use for testing if:

• The application you're testing already uses requests-cache
• You already have a cache populated with responses from your application
• Other methods of recording test responses don't suit your needs

I'd like to hear if anyone else has thoughts on this!

Combining requests-cache with other test-focused libraries

There are some excellent libraries out there specifically made for mocking HTTP requests in unit tests:

• betamax
• requests-mock
• responses
• vcrpy (optionally with pytest-recording)

Instead of adding testing-specific features to requests-cache, I'm interested in the possibility of reusing its cached response data with one or more of those libraries.

betamax and vcrpy

VCR (the basis for both betamax and vcrpy) has the concept of a cassette library, which is a directory containing response data in either JSON or YAML format. It would be really convenient to be able to convert a cache into a cassette library. The filesystem backend and alternative serializers (including JSON and YAML) added in 0.7 get us about 80% of the way there.

requests-mock and responses

There is an example in the docs of combining requests-cache with requests-mock. This causes requests to use the cache if possible, and then use the requests-mock adapter where it would normally make a real request. This will raise an exception if you try to request a URL that isn't explicitly mocked (or cached).

The downside is that this uses a mix of different request matching strategies from the two libraries, which may or may not be a problem depending on the types of requests you're making. It might be preferable to define mock requests + responses based on cache contents. Here's a quick example:

def build_mock_adapter(cache: BaseCache) -> Adapter:
    """Build a requests-mock Adapter with mocked URLs and responses based on cache data"""
    adapter = Adapter()
    for response in cache.values():
        adapter.register_uri(
            response.request.method,
            response.request.url,
            content=response.content,
            headers=response.headers,
            status_code=response.status_code,
        )
    return adapter

A similar pattern should work for the responses library, with responses.add() instead of adapter.register_uri(), and some slightly different arguments.

[JWCook]

@sigmavirus24 @jamielennox @markstory @kevin1024
Hi there! As maintainers of the libraries I just mentioned, do any of you have opinions on this topic? Have any of your users brought up similar use cases, e.g. wanting to reuse response data from a cache or some other source?

[markstory]

Have any of your users brought up similar use cases, e.g. wanting to reuse response data from a cache or some other source?

I can't say it has come up since I've been maintaining responses (last 2 years). With responses we're focused only on the testing angle, we've made it straightforward to define mocks as pytest fixtures, which makes them easier to reuse allows users to load mocks from a file if they so choose (similar to your code example).

[sigmavirus24]

So for betamax, we've had the opposite direction desired - People wanting to use it as a caching library. There are specific heuristics needed to be implemented to do so though and so it never became a thing.

Without knowing anything about requests-cache, I presume however it stores content is predictable and one could ostensibly build a serializer for betamax to read/write it. That said, the metadata for either library is likely to be different. Our serializer pluggability is (I hope) well-enough documented for someone to write that glue code.

[JWCook]

I can't say it has come up since I've been maintaining responses (last 2 years). With responses we're focused only on the testing angle, we've made it straightforward to define mocks as pytest fixtures, which makes them easier to reuse allows users to load mocks from a file if they so choose (similar to your code example).

That makes sense. In that case it might suffice to just add an example to our docs of using requests-cache with responses. Thanks!

I presume however it stores content is predictable and one could ostensibly build a serializer for betamax to read/write it. That said, the metadata for either library is likely to be different. Our serializer pluggability is (I hope) well-enough documented for someone to write that glue code.

Yeah, the docs look straightforward, and that would definitely be doable. The underlying storage format for requests-cache is variable (files, SQLite, Redis, MongoDB, etc.), but they are all wrapped in a common dict-like interface, which should work fine with a custom betamax serializer.

Another option would be to make a custom requests-cache serializer that outputs to a VCR/betamax-compatible cassette library, or a tool to convert an existing cache (in any other format) to cassettes. That could have the benefit of being compatible with multiple libraries, and not requiring requests-cache as a test dependency (if it's otherwise not needed for tests).

I see that betamax can read VCR-recorded cassettes, with the caveat about the missing URL attribute. Are there any other major differences between VCR and betamax format?