Is there a way to return a cached response when using a Retry adapter and force_refresh=True?

[dmitchell7]

I am wondering if there is a way to return a cached response when one is available when the request raises a requests.exceptions.RetryError and force_refresh=True. For my use case I need/always want to make the request and only use a cached response if the request fails.

Example:

from requests_cache import CachedSession, SQLiteCache
from requests.adapters import HTTPAdapter, Retry
from datetime import timedelta

cached_backend = SQLiteCache(use_cache_dir=True)
cached_session = CachedSession(
    backend=cached_backend,
    cache_control=True,
    expire_after=timedelta(hours=1),
    stale_while_revalidate=timedelta(minutes=30),  
)
retries = Retry(
    total=5,
    backoff_factor=0.1,  # will sleep for [0.0s, 0.2s, 0.4s, 0.8s, …] between retries.
    status_forcelist=[500, 502, 503, 504],
)
adapter=HTTPAdapter(max_retries=retries)
cached_session.mount("https://", adapter=adapter)
cached_session.mount("http://", adapter=adapter)


response = cached_session.get('https://httpbin.org/status/500', force_refresh=True) << raises a requests.exception.RetryError but does not return cached response if available

[JWCook]

Yes! I think the stale_if_error setting is what you're looking for. Let me know if that does what you want.

[JWCook]

Oh, I missed the part about needing to use force_refresh=True. That will always skip reading from the cache, so it will take precedence over stale_if_error.

Let me think about the best way to handle that.

[JWCook]

Instead of force_refresh, you could set requests to expire immediately:

from requests_cache import CachedSession, EXPIRE_IMMEDIATELY

backend = SQLiteCache(use_cache_dir=True)
session = CachedSession(
    backend=backend,
    expire_after=EXPIRE_IMMEDIATELY,
    stale_if_error=True,
)

[dmitchell7]

Thanks for the response! I don't think this works for me with the Retry HTTPAdapter. It appears that a successful request/response is not cached with this method. (snipping some private data)

>>> response1 = session.get("https://myapi/v2.0/users", force_refresh=True)
>>> response1.from_cache
False
>>> response1.json()
[{'some_data': 1}]
>>> session.cache.urls()
[]

And then if the API route is modified to only return a 500 error it does not return any stale cached data.

    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool('myapi', port=443): Max retries exceeded with url: /api/v2.0/users (Caused by ResponseError('too many 500 error responses'))

I think I was able to do what I'm trying to do by creating my own CacheSession object and using a tenacity retry on get calls. The decorator will retry the function if an exception is raised and modify the kwargs to be force_refresh=False so that it will check for a cached response.

class CachedRequestsSession(CachedSession):
    def __init__(self):
        ...initialize the CachedSession

    @retry_cache_request()
    def get(self, url, params=None, **kwargs):
        return self.request("GET", url, params=params, **kwargs)

I think maybe my use case is not the intended usage in that I want to prefer to get data from the API every time and only read from cache when there is an error.