Merge pull request #369 from JWCook/appdirs

Add option to use user cache dir (~/.cache, etc.) for SQLite and Filesystem backends

HISTORY.md

@@ -3,37 +3,46 @@
 ## 0.8.0 (TBD)
 [See all issues and PRs for 0.8](https://github.com/reclosedev/requests-cache/milestone/3?closed=1)
 
-**Cache headers:**
-* Add support for `ETag` + `If-None-Match` headers
-* Add support for `Last-Modified` + `If-Modified-Since` headers
-* Add handling for `304 Not Modified` responses if returned for any other reason
-  (e.g., request headers manually set by the client)
+**Conditional requests:**
+* Add support for the following request + response headers to enable conditional requests:
+  * `ETag` + `If-None-Match` headers
+  * `Last-Modified` + `If-Modified-Since` headers
+  * `304 Not Modified` responses
 
-**Backends:**:
+**Backends:**
+* Filesystem: Add `use_cache_dir` option to use platform-specific user cache directory
 * Filesystem: Add `FileCache.paths()` method
 * Filesystem: Fix issue in which `redirects.sqlite` would get included in response paths
+* SQLite: Add `use_cache_dir` option to use platform-specific user cache directory
 * SQLite: Add `use_memory` option and support for in-memory databases
 * SQLite: Add `SQLiteCache.db_path` property
-* For consistency with other backends, rename:
-  * `DbCache` -> `SQLiteCache`
-  * `DbDict` -> `SQLiteDict`
-  * `DbPickleDict` -> `SQLitePickleDict`
-  * `DynamoDbCache` -> `DynamoCache`
-  * `DynamoDbDict` -> `DynamoDict`
-  * Add aliases for previous names for backwards-compatibility
 
 **Serialization:**
 * Use `cattrs` for serialization by default, which enables a more forwards-compatible serialization format
   (e.g., less prone to invalidation due to future updates)
 
-**Other Features:**
-* Add support for custom cache key callbacks
+**Other features:**
+* Add support for custom cache key callbacks with `key_fn` parameter
 
-**Breaking changes:**
-* Drop support for python 3.6
-    * Note: Any bugfixes for 0.8.x that also apply to 0.7.x will be backported
+**Depedencies:**
+* Require python 3.7+
+    * Note: python 3.6 support in 0.7.x will continue to be maintained until it reaches EOL (2021-12-23)
+    * Any bugfixes for 0.8 that also apply to 0.7 will be backported
+* Add new `appdirs` dependency (for user cache directories)
+* Update `cattrs` from optional to a required dependency
+* Update `itsdangerous` required to an optional dependency
+* Require requests 2.22+ and urllib3 1.25.5+
+
+**Deprecations & removals:**
 * Remove deprecated `core` module
 * Remove deprecated `BaseCache.remove_old_entries()` method
+* For consistency with other backends, rename:
+  * `DbCache` -> `SQLiteCache`
+  * `DbDict` -> `SQLiteDict`
+  * `DbPickleDict` -> `SQLitePickleDict`
+  * `DynamoDbCache` -> `DynamoCache`
+  * `DynamoDbDict` -> `DynamoDict`
+  * Add aliases for previous names for backwards-compatibility
 
 -----
 ### 0.7.4 (2021-08-16)

docs/security.md

@@ -13,6 +13,11 @@ Since this isn't always possible, requests-cache can optionally use
 It works by signing serialized data with a secret key that you control. Then, if the data is tampered
 with, the signature check fails and raises an error.
 
+Optional dependencies can be installed with:
+```bash
+pip install requests-cache[security]
+```
+
 ## Creating and Storing a Secret Key
 To enable this behavior, first create a secret key, which can be any `str` or `bytes` object.
 

pyproject.toml

@@ -1,10 +1,12 @@
 [tool.poetry]
 name = "requests-cache"
 version = "0.8.0"
-description = "A transparent, persistent cache for the requests library"
-authors = ["Roman Haritonov", "Jordan Cook"]
+description = "A transparent persistent cache for the requests library"
+authors = ["Roman Haritonov"]
+maintainers = ["Jordan Cook"]
 license = "BSD License"
 readme = "README.md"
+documentation = "https://requests-cache.readthedocs.io"
 homepage = "https://github.com/reclosedev/requests-cache"
 repository = "https://github.com/reclosedev/requests-cache"
 keywords = ["requests", "cache", "http", "persistence", "sqlite", "redis", "mongodb", "gridfs", "dynamodb"]
@@ -14,75 +16,89 @@ classifiers = [
     "Typing :: Typed",
 ]
 include = [
-    { path = "*.md", format = "sdist" },
-    { path = "*.yml", format = "sdist" },
-    { path = "docs", format = "sdist" },
-    { path = "examples", format = "sdist" },
-    { path = "tests", format = "sdist" },
+    {format="sdist", path="*.md"},
+    {format="sdist", path="*.yml"},
+    {format="sdist", path="docs"},
+    {format="sdist", path="examples"},
+    {format="sdist", path="tests"},
 ]
 
-[tool.poetry.urls]
-"Documentation" = "https://requests-cache.readthedocs.io"
-
 [tool.poetry.dependencies]
-python = "^3.7"
-attrs = "^21.2"
-cattrs = "^1.8"
-itsdangerous = "^2.0"
-requests = "^2.22"
-urllib3 = "^1.25.4"
-url-normalize = "^1.4"
+python        = "^3.7"
 
-# Optional serialization dependencies
-bson = {version = ">=0.5", optional = true}
-pyyaml = {version = ">=5.4", optional = true}
-ujson = {version = ">=4.0", optional = true}
+# Required dependencies
+requests      = "^2.22"    # Needs no introduction
+urllib3       = "^1.25.5"  # Use a slightly newer version than required by requests (for bugfixes)
+appdirs       = "^1.4.4"   # For options that use platform-specific user cache dirs
+attrs         = "^21.2"    # For response data models
+cattrs        = "^1.8"     # For response serialization
+url-normalize = "^1.4"     # For reducing duplicate cache items
 
 # Optional backend dependencies
-boto3 = { version = "^1.15", optional = true }
-botocore = { version = "^1.18", optional = true }
-pymongo = { version = "^3.0", optional = true }
-redis = { version = "^3.0", optional = true }
+boto3                      = {optional=true, version="^1.15"}
+botocore                   = {optional=true, version="^1.18"}
+pymongo                    = {optional=true, version="^3.0"}
+redis                      = {optional=true, version="^3.0"}
 
-# Documentation dependencies needed for Readthedocs builds (rtd doesn't support poetry.dev-dependencies)
+# Optional serialization dependencies
+bson                       = {optional=true, version=">=0.5"}
+itsdangerous               = {optional=true, version="^2.0"}
+pyyaml                     = {optional=true, version=">=5.4"}
+ujson                      = {optional=true, version=">=4.0"}
+
+# All the bells and whistles for building documentation;
+# defined here because readthedocs doesn't (yet?) support poetry.dev-dependencies
 # [tool.poetry.dev-dependencies]
-furo = {version = ">=2021.8.11-beta.42", optional = true}
-myst-parser = {version = "^0.15.1", optional = true}
-Sphinx = { version = "4.1.2", optional = true }
-sphinx-autodoc-typehints = { version = "^1.11", optional = true }
-sphinx-automodapi = {version = "^0.13", optional = true}
-sphinx-copybutton = { version = ">=0.3,<0.5", optional = true }
-sphinx-inline-tabs = {version = "^2021.4.11-beta.9", optional = true, python = ">=3.8"}
-sphinxcontrib-apidoc = { version = "^0.3", optional = true }
-linkify-it-py = {version = "^1.0.1", optional = true}
+furo                       = {optional=true, version=">=2021.8.11-beta.42"}
+linkify-it-py              = {optional=true, version="^1.0.1"}
+myst-parser                = {optional=true, version="^0.15.1"}
+sphinx                     = {optional=true, version="4.1.2"}
+sphinx-autodoc-typehints   = {optional=true, version="^1.11"}
+sphinx-automodapi          = {optional=true, branch="main", git="https://github.com/astropy/sphinx-automodapi.git"}  # Unreleased bugfixes
+sphinx-copybutton          = {optional=true, version=">=0.3,<0.5"}
+sphinx-inline-tabs         = {optional=true, version="^2021.4.11-beta.9", python=">=3.8"}
+sphinxcontrib-apidoc       = {optional=true, version="^0.3"}
 
 [tool.poetry.extras]
-all = ["boto3", "botocore", "pymongo", "redis", "ujson"]
-bson = ["bson"]  # BSON comes with pymongo, and can also be used as a standalone codec
-json = ["ujson"]
-yaml = ["yaml"]
+# Package extras for optional backend dependencies
 dynamodb = ["boto3", "botocore"]
-mongodb = ["pymongo"]
-redis = ["redis"]
-docs = ["furo", "linkify-it-py", "myst-parser", "sphinx", "sphinx-autodoc-typehints",
-        "sphinx-automodapi", "sphinx-copybutton", "sphinx-inline-tabs", "sphinxcontrib-apidoc"]
+mongodb  = ["pymongo"]
+redis    = ["redis"]
+
+# Package extras for optional seriazliation dependencies
+bson     = ["bson"]   # BSON comes with pymongo, but can also be used as a standalone codec
+json     = ["ujson"]  # Will optionally be used by JSON serializer for improved performance
+security = ["itsdangerous"]
+yaml     = ["yaml"]
+
+# All optional packages combined, for demo/evaluation purposes
+all      = ["boto3", "botocore", "itsdangerous", "pymongo", "redis", "ujson"]
+
+# Documentation
+docs     = ["furo", "linkify-it-py", "myst-parser", "sphinx", "sphinx-autodoc-typehints",
+            "sphinx-automodapi", "sphinx-copybutton", "sphinx-inline-tabs", "sphinxcontrib-apidoc"]
 
 [tool.poetry.dev-dependencies]
-mypy = "^0.910"
-nox = "^2021.6.12"
-nox-poetry = "^0.8.6"
-pre-commit = "^2.12"
-psutil = "^5.0"
-pytest = "^6.2"
-pytest-clarity = "^1.0.1"
-pytest-cov = ">=2.11"
-pytest-rerunfailures = "^10.1"
-pytest-xdist = ">=2.2"
-requests-mock = "^1.8"
-responses = "0.10.15"
-rich = ">=10.0"
-sphinx-autobuild = "^2021.3.14"
-timeout-decorator = "^0.5"
+# For unit + integration tests
+psutil                = "^5.0"
+pytest                = "^6.2"
+pytest-clarity        = "^1.0.1"
+pytest-cov            = ">=2.11"
+pytest-rerunfailures  = "^10.1"
+pytest-xdist          = ">=2.2"
+requests-mock         = "^1.8"
+responses             = "0.10.15"
+timeout-decorator     = "^0.5"
+
+# For linting, etc.; additional tools are managed by pre-commit
+mypy                  = "^0.910"
+pre-commit            = "^2.12"
+
+# For convenience in local development
+nox                   = "^2021.6.12"
+nox-poetry            = "^0.8.6"
+rich                  = ">=10.0"
+sphinx-autobuild      = "^2021.3.14"
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]

requests_cache/backends/filesystem.py

@@ -38,23 +38,23 @@
 from contextlib import contextmanager
 from glob import glob
 from os import listdir, makedirs, unlink
-from os.path import abspath, basename, dirname, expanduser, isabs, join, splitext
+from os.path import basename, dirname, join, splitext
 from pathlib import Path
 from pickle import PickleError
 from shutil import rmtree
-from tempfile import gettempdir
 from typing import List, Union
 
 from ..serializers import SERIALIZERS
 from . import BaseCache, BaseStorage
-from .sqlite import SQLiteDict
+from .sqlite import SQLiteDict, get_cache_path
 
 
 class FileCache(BaseCache):
     """Filesystem backend.
 
     Args:
         cache_name: Base directory for cache files
+        use_cache_dir: Store datebase in a user cache directory (e.g., `~/.cache/`)
         use_temp: Store cache files in a temp directory (e.g., ``/tmp/http_cache/``).
             Note: if ``cache_name`` is an absolute path, this option will be ignored.
         extension: Extension for cache files. If not specified, the serializer default extension
@@ -75,9 +75,16 @@ def paths(self) -> List[str]:
 class FileDict(BaseStorage):
     """A dictionary-like interface to files on the local filesystem"""
 
-    def __init__(self, cache_name, use_temp: bool = False, extension: str = None, **kwargs):
+    def __init__(
+        self,
+        cache_name,
+        use_temp: bool = False,
+        use_cache_dir: bool = False,
+        extension: str = None,
+        **kwargs,
+    ):
         super().__init__(**kwargs)
-        self.cache_dir = _get_cache_dir(cache_name, use_temp)
+        self.cache_dir = get_cache_path(cache_name, use_cache_dir=use_cache_dir, use_temp=use_temp)
         self.extension = extension if extension is not None else _get_default_ext(self.serializer)
         self.is_binary = False
         makedirs(self.cache_dir, exist_ok=True)
@@ -135,17 +142,6 @@ def paths(self) -> List[str]:
         return glob(self._path('*'))
 
 
-def _get_cache_dir(cache_dir: Union[Path, str], use_temp: bool) -> str:
-    # Save to a temp directory, if specified
-    if use_temp and not isabs(cache_dir):
-        cache_dir = join(gettempdir(), cache_dir, 'responses')
-
-    # Expand relative and user paths (~/*), and make sure parent dirs exist
-    cache_dir = abspath(expanduser(str(cache_dir)))
-    makedirs(cache_dir, exist_ok=True)
-    return cache_dir
-
-
 def _get_default_ext(serializer) -> str:
     for k, v in SERIALIZERS.items():
         if serializer is v: