Implement client cache controller for Edgar

[paultiq]

The plumbing has been in place for a bit, but without any Edgar specific rules.

Could you take a look at the rules here? Specifically over what paths should/shouldn't be cached, and what cache periods make sense for defaults:

**This cache controller caches, by default:

• /submissions URLs for up to 10 minutes by default, set in MAX_SUBMISSIONS_AGE_SECONDS
• .*index/.* URLs for up to (edit: changed to 30) 30 minutes by default, set in MAX_INDEX_AGE_SECONDS
• /Archives/edgar/data URLs indefinitely (forever)**
• Everything else follows default caching rules, which generally means it's always revalidated.

The implementation is in httpclient_cache.EdgarController.

Note: also added support for swapping in different storage backends. See https://hishel.com/ for usage. To use S3, for instance, create a hishel.S3Storage and hishel.AsyncS3Storage object, and pass to install_cached_client.

Example:

from edgar import httpclient_cache, set_identity, Company
from pathlib import Path
import logging

logging.basicConfig(
    format='%(asctime)s %(name)s %(levelname)-8s %(message)s',
    level=logging.INFO,
    datefmt='%Y-%m-%d %H:%M:%S')

logging.getLogger("hishel.controller").setLevel(logging.DEBUG)

httpclient_cache.install_cached_client(cache_directory = Path(r"."), controller_args = {"allow_heuristics": True, "allow_stale": True, "always_revalidate": False})

# set_identity("your@email.com")

filings = Company('MS').get_filings(form="10-Q")

[dgunning]

The index files are updated once per day and are heavily used. These can be cached for longer than 10 minutes.
This would match
https://www.sec.gov/Archives/edgar/full-index/

I wonder if on balance it should be longer like 30 minutes. We could go longer than that but caching is really for heavy users so who might also want to get new indexes in a timely fashion.

[paultiq]

• Changed MAX_INDEX_AGE_SECONDS to 30 minutes, but also changed the logic slightly:

I also changed the cache rule so it revalidates rather than expiring the cache after 30 minutes. So, within 30 minutes, it doesn't check for new data. After 30 minutes, it'll check again and then suppress checks for 30 minutes.

This sounds like a lot, but it's just using the built-in features of Hishel's controller:

We're just picking which of the three options is valid for https://github.com/karpetrosyan/hishel/blob/37c0ac2c0159fe281a256d0818c4c9818c98bdf3/hishel/_controller.py#L371-L374

   Returns:
      Response: This response is applicable to the request.
      Request: This response can be used for this request, but it must first be revalidated.
      None: It is not possible to use this response for this request.

So, within 30 minutes, the controller returns the Response (the cached value). After 30 minutes, it returns the Request (requires revalidation, but will use the cache afterwards).

[dgunning]

Approved