Merge pull request #189 from JWCook/refactor

Refactor cached response info & initialization into a separate class

Closes #99, #148, #186 , #187, #188, and does some of the work for #169 and #184.

## Summary
The goal of this PR is to reduce code complexity, and hopefully make things a little easier for others who want to contribute. Some of the logic around getting and saving responses had gotten a bit complicated as more features have been added, mainly in:
* `CachedSession.request()`
* `CachedSession.send()`
* `BaseCache.set_response()` (and associated helper methods)
* `BaseCache.get_response()` (and associated helper methods)

I've consolidated most of the logic around expiration, serializiation compatibility, and other response object handling into a separate class, `CachedResponse`. This simplifies the above methods, removes most of the code duplication, and fixes a few other issues. According to [radon](https://radon.readthedocs.io/), all classes/modules/methods except two have an 'A' maintainability rating (CC <= 5).

## Changes
**Short version:** Most of the important bits are here: [requests_cache/response.py](https://github.com/reclosedev/requests-cache/blob/b0f2c132fc320c9b1e32e839add63240b2805bbb/requests_cache/response.py)

### Serialization/Deserialization
* Response creation time and expiration time are stored in `CachedResponse`, so the timestamp from the `(response, timestamp)` tuple is no longer needed
* `CachedResponse.is_expired` can be used to indicate if an old response was returned as the result of an error with `old_data_on_error=True` (#99)
* Replace `_RawStore`  with `CachedHTTPResponse` class to wrap raw responses, and:
    * Maintain support for streaming requests (#68)
    * Improve handling for generator usage
    * Add support for use with `pandas.read_csv()` and similar readers (#148)
    * Add support for use as a context manager (#148)
    * Add support for `decode_content` param
* Fix streaming requests when used with memory backend (#188)
* Verified that `PreparedRequest.body` is always encoded in utf-8, so no need to detect encoding
    *  Re: [todo note here](https://github.com/reclosedev/requests-cache/blob/08886efe1841aae9c6a5e133008b69296d23b8b9/requests_cache/backends/base.py#L238); see [requests.PreparedRequest.prepare_body()](https://github.com/psf/requests/blob/54336568789e0ec41f70c800a96384e4fac58dd9/requests/models.py#L455)

### Expiration
* Add optional `expire_after` param to `CachedSession.remove_old_responses()`
* Wrap temporary `_request_expire_after` in a contextmanager
* Remove `expires_before` param from remove_old_entries, and always use the current time
* Remove `relative_to` param from `CachedSession._determine_expiration_datetime` for unit testing and patch `datetime.now` in unit tests instead
* Rename `response.expire_after` and `response.cache_date` to `expires` and `created_at`, respectively, based on browser caching

### Docs
* Add type annotations to public methods in `CachedSession`, `CachedResponse`, and `BaseCache`
* Add some more docstrings and code comments
* Add intersphinx links for `urllib` classes & methods
* Update user guide for using requests-cache with `requests_mock.Adapter`

### Tests
* Add an update tests for all new and changed code, and add coverage for additional edge cases
* Add fixture for combining requests-cache with requests-mock, using a temporary SQLite db in `/tmp`
* Refactor all tests in `test_cache` as pytest-style functions instead of `unittest.TestCase` methods
* Update most tests to use requests-mock instead of using httpbin.org (see #169)

### Misc
* Split some of the largest functions into multiple smaller functions
* Make use of dict ordering from python3.6+ in _normalize_parameters()
* Fix linting issues raised by flake8

## Backwards-compatibility
These changes don't break compatibility with code using requests-cache <= 0.5.2, but they aren't compatible with previously cached data due to the different serialization format. Retrieving a previously cached response will fail quietly and simply fetch and cache a new response. Since the docs already warn about this potentially happening with new releases, I don't think this is a problem.

example_request.py

@@ -5,7 +5,7 @@
 
 import requests_cache
 
-requests_cache.install_cache('example_cache', backend='memory')
+requests_cache.install_cache('example_cache', backend='sqlite')
 
 
 def main():
@@ -15,11 +15,9 @@ def main():
     response = requests.get('https://httpbin.org/get')
     assert response.from_cache
 
-    # Changing the expires_after time causes a cache invalidation,
-    # thus /get is queried again ...
+    # Caching with expiration
+    requests_cache.clear()
     response = requests.get('https://httpbin.org/get', expire_after=1)
-    assert not response.from_cache
-    # ... but cached for 1 second
     response = requests.get('https://httpbin.org/get')
     assert response.from_cache
     # After > 1 second, it's cached value is expired

pyproject.toml

@@ -16,6 +16,7 @@ source = ['requests_cache']
 profile = "black"
 line_length = 100
 skip_gitignore = true
+skip = ['requests_cache/__init__.py']
 known_first_party = ['test']
 # Things that are common enough they may as well be grouped with stdlib imports
 extra_standard_library = ['pytest', 'setuptools']

requests_cache/__init__.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+# flake8: noqa: E402,F401
 """
     requests_cache
     ~~~~~~~~~~~~~~
@@ -18,15 +19,17 @@
         # will take approximately 5 seconds instead 50
 
 
-    :copyright: (c) 2012 by Roman Haritonov.
+    :copyright: (c) 2021 by Roman Haritonov.
     :license: BSD, see LICENSE for more details.
 """
 __docformat__ = 'restructuredtext'
 __version__ = '0.6.0'
 
-# Quietly ignore importerror, if setup.py is invoked outside a virtualenv
+# Quietly ignore ImportError, if setup.py is invoked outside a virtualenv
 try:
+    from .response import AnyResponse, CachedHTTPResponse, CachedResponse, ExpirationTime
     from .core import (
+        ALL_METHODS,
         CachedSession,
         CacheMixin,
         clear,

requests_cache/backends/__init__.py

@@ -1,13 +1,28 @@
-# noqa: F401
+# flake8: noqa: F401
 """
     requests_cache.backends
     ~~~~~~~~~~~~~~~~~~~~~~~
 
     Classes and functions for cache persistence
 """
-
-
-from .base import BACKEND_KWARGS, BaseCache
+from .base import BaseCache
+
+# All backend-specific keyword arguments combined
+BACKEND_KWARGS = [
+    'connection',
+    'db_name',
+    'endpont_url',
+    'extension',
+    'fast_save',
+    'ignored_parameters',
+    'include_get_headers',
+    'location',
+    'name',
+    'namespace',
+    'read_capacity_units',
+    'region_name',
+    'write_capacity_units',
+]
 
 registry = {
     'memory': BaseCache,