General way to turn mocking off?

[maelle]

Reg https://enpiar.com/r/httptest/index.html#how-do-i-switch-between-mocking-and-real-requests in vcr one can turn off the use of cassettes just by setting an environment variable. This can be useful to have, say, a weekly scheduled CI run of the tests that use actual API responses. I see you suggest some lines to add to helper.R, but why not add this behaviour to httptest itself so the user would only need to set the environment variable?

[maelle]

with an environment variable (another one?) one could also switch between capturing requests and using the mocks? which might make it easier to record mocks the first time around.

[maelle]

In ropensci-books/exemplighratia#1

• I first set the RECORDING environment variable, then ran the tests. This recorded the mocks.
• Then I set it to FALSE.

[nealrichardson]

This has come up a few times, and I'm open to suggestion for how to make it work. Another approach that I think I like better than env vars is more like what start_vignette() does: if the given mock directory exists, use it; if not, create the directory and record real requests. Something like:

with_mock_dir(path, {
  ...
})

This would let you, say, re-record some mocks (by deleting that directory) without having to re-record everything, and have some mock tests that are based on live requests while also have some that are not.

The reason why this isn't built into httptest already, and why I am less enthusiastic about global solutions, gets to some of the original design principles of httptest. It doesn't assume that every API mock came from a real request to a real server, and it is designed so that you are able to see and modify test fixtures. Among the considerations:

1. In many cases, API responses contain way more content than is necessary to test your R code around them: 100 records when 2 will suffice, request metadata that you don't care about and can't meaningfully assert things about, and so on. In the interest of minimally reproducible examples, and of making tests readable, it often makes sense to take an actual API response and delete a lot of its content, or even to fabricate one entirely.
2. And then it's good to keep that API mock fixed so you know exactly what is in it. If I re-recorded a Twitter API response of, say, the most recent 10 tweets with #rstats, the specific content will change every time I record it, so my tests can't say much about what is in the response without having to rewrite them every time too.
3. Some conditions (rate limiting, server errors, e.g.) are difficult to test with real responses, but if you can hand-create a API mock with, say, a 503 response status code and test how your code handles it, you can have confidence of how your package will respond when that rare event happens with the real API.
4. Re-recording all responses can make for a huge code diff, which can blow up your repository size and make code review harder.

I say this not to argue that there shouldn't be a nice way to have tests that work either with mocks or with live requests: I recognize the convenience and value of that--and that not everyone wants to edit JSON to make trivial API fixtures. I point it out since you're comparing with vcr and think it's relevant context to explain the different feature sets and design choices.

[maelle]

this is such useful context!! I'll probably work this into the book either before or after the demos.

I was also wondering whether you usually work with good APIs, i.e. APIs that use versioning?

[maelle]

A bit off-topic...

Some conditions (rate limiting, server errors, e.g.) are difficult to test with real responses

I am looking into how to test for the behavior of the package in case of errors. Now, because mock filepaths are based on the request, how do you create two mock files for the same request, one with errors, one with no error? Using https://enpiar.com/r/httptest/reference/mockPaths.html, right?

[maelle]

Actually for errors I see the pattern mocking (general mocking) + fake_response e.g. https://github.com/cran/datapackage.r/blob/0bc1caa1903cf36b2d5f318265426eb072c361d7/tests/testthat/test-profile.R#L68

[nealrichardson]

Yes, one way to have the same request respond differently is to change/append to the mockPaths--this is how the vignette mock setup works. It's trickier if you are testing something that retries the same request internally because you need some kind of hook to know when to switch mock paths. https://github.com/Crunch-io/rcrunch/blob/d1b5cade5e7b0608ddc1ce5de1a576e2d3614d84/tests/testthat/test-progress.R#L28-L37 is one attempt I did a while ago to poll an endpoint and confirm that it kept requesting until it "completed" (corresponding mocks here), but it's not trivial.

For errors, the example you found is a bit odd in that it's using with_mock to alter httptest. A better approach IMO is more like I described here: https://enpiar.com/2017/06/21/7-hard-testing-problems-made-easy-by-httptest/#5-rare-or-difficult-to-trigger-server-behavior. You can do a real request and modify the resulting response object to have whatever properties you want (response status etc.), or call fake_response() and construct whatever you want, and then call save_response() to write the mock out. (This surely deserves better documentation/how-to guidance.)

Back on my other points from yesterday, one concrete example of when you might want to delete content from response fixtures is with pagination. If you don't want to expose pagination to the R users (sometimes you do, sometimes it's an implementation detail they shouldn't have to care about), you might want an API wrapper that, if it gets a paginated response, makes subsequent requests to retrieve all records. In this package, I wrote a wrapper that did that, but IIRC the server controlled the page size and it was 100 records. I didn't want or need to have that many records in my test fixtures, so I recorded the responses and deleted all but 3 from the first page and 2 from the second. So I could test that the R function made the right requests to consume the multiple pages of results, and by asserting that there were 5 results, I knew that it was paging correctly.

I was also wondering whether you usually work with good APIs, i.e. APIs that use versioning?

I've worked with both good and really bad. Good APIs definitely make the code easier but I'm not sure how much they affect the testing strategy--what do you think? I guess with a well documented API you can get away with being less defensive/paranoid about just accepting whatever the API throws at you and can write more focused tests, perhaps even with mock fixtures that come from the API documentation (as I do in the httptest vignette with the Twitter API).

[maelle]

Thanks, this is very useful and I'll digest the information!

Regarding APIs what I was thinking is that if the response structure changes often and unexpectedly one will probably want to not have to tinker too much manually with the mock files, to be able to re-record them easily?

[maelle]

• I think this could be considered closed by add with_mock_dir #48

• Reg real requests in the HTTP testing book, in the httptest chapter I discuss using different tests for real requests, and I mentioned that in passing in add with_mock_dir #48