-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disabling requests-cache is not thread safe. Discussion of the future of _restclient's usage of requests-cache #14
Comments
With reference to CacheControl, the default cache is a |
Thanks for doing a little digging... Having not done that so far myself, the issues that you mentioned are problematic and do not sound like they would support our needs. Specifically a
#14 may deem it necessary to develop our own cache sub-package. For reference, this is something @jarq6c and @aaraney have previously discussed, but never formally tabled. At face value it seems that we should be able to learn from the implementations of both With all of the being said, I don't want this issue to dissolve into another subject, but just speak generally and voice the potential feature addition to address this bug. |
The disadvantages of a hypothetical The potential advantages of such a package go beyond its primary use to cache NWIS and NWM data. We could unify caching and storing of data through a singular interface. This might allow data/computational scientists using I'm imagining a subpackage that takes |
@jarq6c its ironic you mention an event cache, I may or may not have had the need to implement: class EventCache(object):
"""
Event cache manager
FIXME current multiprocessing works because of a guarantee of process order,
but this may need to be better protected from reader/writer issues
""" I would encourage thinking about the needs of caching and what exactly is being cached. A dataframe cache with clear semantics isn't too far fetched. |
FYI, a better option for using requests-cache is using Also, the library is being maintained again (with a fairly large release last week), and issues and PRs are welcome. |
Because of #78, multiprocessing usage should become more obsolete in the context of retrieving data. With that in mind, I am going to close this and we can reopen in the future to continue discussion. |
Due to the way
requests-cache
(used in_restclient
) is implemented, if the cache has been "installed" and a downstream package uses requests, the downstream package's requests will be cached. Meaning thatrequests_cache
implicitly changes the behavior of therequests
package for all downstream callers.Per
requests-cache
's documentation, they do provide a context manager to "disable" the cache. However, as @christophertubbs pointed out their implementation is not thread safe:In discussing this issue with @hellkite500, @hellkite500 mentioned the (actively developed) project CacheControl. Before active development to fix this behavior, its worth exploring a bit and determining what long term solution to implement.
The text was updated successfully, but these errors were encountered: