Using vcrpy as a dynamic callback on HTTPretty

[twolfson]

In Node.js, I have written and used 2 libraries for HTTP mocking/playback: fixed-server and eight-track (currently actually spy-server/nine-track). The gist is fixed-server handles routing to determine whether to use a static fixture (e.g. something from disk) or to use a playback utility (e.g. eight-track). eight-track is similar to vcrpy but it doesn't patch underlying libraries -- it only accepts request objects.

More info can be found here: http://twolfson.com/2015-01-11-testing-with-other-services

We want to bring this functionality over to Python. Currently, we are using HTTPretty for static and pseudo-dynamic responses but nothing with automatic playback like vcrpy. Since HTTPretty has routing built in and supports dynamic responses, we wanted to use it as the "router" (i.e. fixed-server). This means vcrpy would become an optional dynamic callback.

https://github.com/gabrielfalcao/HTTPretty/tree/0.8.3#dynamic-responses-through-callbacks

We tried it out and after a bit of fidgeting, we got a wrapper function that can take parameters from HTTPretty, make the request via vcrpy, and return the response information to HTTPretty.

# Load in our dependencies
import urlparse
import httpretty
from vcr.cassette import Cassette
from vcr.stubs import VCRHTTPConnection, VCRHTTPSConnection

# Create our cassette
cassette = Cassette.load(path='cassette.yaml')


# Define our vcrpy-backed HTTPretty callback
def vcr_request_callback(request, url, httpretty_res_headers):
    # Interpret our URL and select the connection class
    # https://github.com/kevin1024/vcrpy/blob/v.1.7.4/vcr/stubs/__init__.py#L340-L350
    url_parts = urlparse.urlparse(url)
    if url_parts.scheme == 'http':
        VCRProtocolConnection = VCRHTTPConnection
    elif url_parts.scheme == 'https':
        VCRProtocolConnection = VCRHTTPSConnection
    else:
        raise RuntimeError('Unrecognized URL scheme "{scheme}" from "{url}"'
                           .format(scheme=url_parts.scheme, url=url))

    # Generate an HTTP/HTTPS connection using the `netloc` (accepted in Python 2)
    # https://github.com/kevin1024/vcrpy/blob/v.1.7.4/vcr/stubs/__init__.py#L307-L316
    # https://docs.python.org/2/library/httplib.html#httplib.HTTPConnection
    conn = VCRProtocolConnection(url_parts.netloc)
    conn.cassette = cassette

    # Populate our request
    # https://github.com/kevin1024/vcrpy/blob/v.1.7.4/vcr/stubs/__init__.py#L161-L174
    conn.request(method=request.method, url=url, body=request.body, headers=request.headers)

    # Collect our response
    # https://github.com/kevin1024/vcrpy/blob/v.1.7.4/vcr/stubs/__init__.py#L215-L271
    # DEV: We need to disable HTTPretty otherwise we encounter recursive requests
    # TODO: Research if this has implications for requests in a threaded application
    httpretty.disable()
    res = conn.getresponse()
    httpretty.enable()

    # Save updated cassettte
    # https://github.com/kevin1024/vcrpy/blob/v.1.7.4/vcr/cassette.py#L270-L277
    cassette._save()

    # Return our response information to HTTPretty
    return (res.status, dict(res.headers), res.read())

# Register an HTTPretty endpoint to use vcr_request_callback
httpretty.register_uri(httpretty.GET, 'http://google.com/hello', body=vcr_request_callback)

# Register another endpoint to use a static response
httpretty.register_uri(httpretty.GET, 'http://google.com/world', body='world')

Now that the proof of concept is complete, we would love to add it into vcrpy or maybe build a wrapper library that combine the two. Before doing that though, we wanted to get your feedback and thoughts.

Side notes: We are actually using httpretty-fixtures in Python to make our fixtures more reusable across tests.

[twolfson]

I should also mention that we saw #8 but didn't feel as though it lined up with what we were trying to accomplish.

[colonelpanic8]

This is pretty interesting. Would there be some way to easily route ALL requests through vcrpy?

[twolfson]

Yea, httpretty uses regular expressions for its routes. As a result, we could use r'*' (at least in theory) to route all traffic via vcrpy. We also would need to iterate over all the HTTP verbs but those are listed here:

https://github.com/gabrielfalcao/HTTPretty/blob/a615e3e5a6cf57cdcc6a21c8b78120ca3ede241e/httpretty/http.py#L121

I do something similar as a catch-all endpoint in fixed-server(always make sure it's at the end of a list of fixtures):

fakeServer.addFixture('proxy', {
  method: 'all',
  route: '*',
  response: serverNineTrack
});

[agriffis]

This would be awesome. What turned me away from httpretty is needing to build responses in advance rather than being able to record and replay. (I'd also love to see #8 implemented, but that seems a longer way off.)

[Azulinho]

+1

[neozenith]

A lot of changes have happened to VCRpy since this ticket was opened. As this ticket has become stale and we have a lot of old tickets that need to be groomed, I'm closing this for now to make it easier to see what is still relevant.

However if it is still needed, please feel free to re-open or create a new ticket.

Thanks! 🙏