Refactor some serialization/deserialization logic into a separate CachedResponse class #187
Assignees
Labels
Milestone
Comments
This was referenced Mar 16, 2021
JWCook
added a commit
that referenced
this issue
Mar 22, 2021
Refactor cached response info & initialization into a separate class Closes #99, #148, #186 , #187, #188, and does some of the work for #169 and #184. ## Summary The goal of this PR is to reduce code complexity, and hopefully make things a little easier for others who want to contribute. Some of the logic around getting and saving responses had gotten a bit complicated as more features have been added, mainly in: * `CachedSession.request()` * `CachedSession.send()` * `BaseCache.set_response()` (and associated helper methods) * `BaseCache.get_response()` (and associated helper methods) I've consolidated most of the logic around expiration, serializiation compatibility, and other response object handling into a separate class, `CachedResponse`. This simplifies the above methods, removes most of the code duplication, and fixes a few other issues. According to [radon](https://radon.readthedocs.io/), all classes/modules/methods except two have an 'A' maintainability rating (CC <= 5). ## Changes **Short version:** Most of the important bits are here: [requests_cache/response.py](https://github.com/reclosedev/requests-cache/blob/b0f2c132fc320c9b1e32e839add63240b2805bbb/requests_cache/response.py) ### Serialization/Deserialization * Response creation time and expiration time are stored in `CachedResponse`, so the timestamp from the `(response, timestamp)` tuple is no longer needed * `CachedResponse.is_expired` can be used to indicate if an old response was returned as the result of an error with `old_data_on_error=True` (#99) * Replace `_RawStore` with `CachedHTTPResponse` class to wrap raw responses, and: * Maintain support for streaming requests (#68) * Improve handling for generator usage * Add support for use with `pandas.read_csv()` and similar readers (#148) * Add support for use as a context manager (#148) * Add support for `decode_content` param * Fix streaming requests when used with memory backend (#188) * Verified that `PreparedRequest.body` is always encoded in utf-8, so no need to detect encoding * Re: [todo note here](https://github.com/reclosedev/requests-cache/blob/08886efe1841aae9c6a5e133008b69296d23b8b9/requests_cache/backends/base.py#L238); see [requests.PreparedRequest.prepare_body()](https://github.com/psf/requests/blob/54336568789e0ec41f70c800a96384e4fac58dd9/requests/models.py#L455) ### Expiration * Add optional `expire_after` param to `CachedSession.remove_old_responses()` * Wrap temporary `_request_expire_after` in a contextmanager * Remove `expires_before` param from remove_old_entries, and always use the current time * Remove `relative_to` param from `CachedSession._determine_expiration_datetime` for unit testing and patch `datetime.now` in unit tests instead * Rename `response.expire_after` and `response.cache_date` to `expires` and `created_at`, respectively, based on browser caching ### Docs * Add type annotations to public methods in `CachedSession`, `CachedResponse`, and `BaseCache` * Add some more docstrings and code comments * Add intersphinx links for `urllib` classes & methods * Update user guide for using requests-cache with `requests_mock.Adapter` ### Tests * Add an update tests for all new and changed code, and add coverage for additional edge cases * Add fixture for combining requests-cache with requests-mock, using a temporary SQLite db in `/tmp` * Refactor all tests in `test_cache` as pytest-style functions instead of `unittest.TestCase` methods * Update most tests to use requests-mock instead of using httpbin.org (see #169) ### Misc * Split some of the largest functions into multiple smaller functions * Make use of dict ordering from python3.6+ in _normalize_parameters() * Fix linting issues raised by flake8 ## Backwards-compatibility These changes don't break compatibility with code using requests-cache <= 0.5.2, but they aren't compatible with previously cached data due to the different serialization format. Retrieving a previously cached response will fail quietly and simply fetch and cache a new response. Since the docs already warn about this potentially happening with new releases, I don't think this is a problem.
JWCook
added
the
serialization
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
BaseCachemethodsreduce_response(),restore_response(), and_picklable_field()can be simplified using a serializable class that wrapsrequests.Response. Similarly, the innerrawresponse can be stored using a serializable class that wrapsurllib3.response.HTTPResponseto replace_RawStore(see #148). It would also make sense to put expiration logic here.This can then reduce complexity in these methods, which currently have a bit of code duplication:
CachedSession.request()CachedSession.send()BaseCache.set_response()BaseCache.get_response()Example of a similar class for
aiohttp.ClientResponse: https://github.com/JWCook/aiohttp-client-cache/blob/main/aiohttp_client_cache/response.pyThe text was updated successfully, but these errors were encountered: