Improved caching for offline operation

[N-Coder]

All of the following is required to allow offline operation

Cache Management

• allow custom cache validation
  • cached results containing/wrapping a failed future shouldn't be kept in cache
• failed Downloads should be dropped from cache
• when restarting a failed Download, its previously successfully downloaded parts should be kept/reused and only the failed parts/chunks retried
• add ttl-based cache clearing
• make cache persistable / serializeable
• use factory for StudIP API data model classes based on object UIDs, preventing duplicate objects
• directly cache the results of the StudIP API Session getter functions
• only persist the now unique instances of the StudIP API data model classes
• reuse expired cache-value as fallback when new attempt fails too often or takes to long
• implement circuit-breaker algorithm for checking for network connectivity and deciding whether (or how often) a function should be retried or the expired value should be reused

Task Management

• not being able to login in when starting loop will always prevent startup when offline
• session termination during runtime is not handled
• store cached values on disk periodically and load then on program start
• allow eager loading of certain directories (with lower priority than user/FUSE operations)
• implement framework for lifecycle tasks
  1. startup tasks (e.g. restore model),
  2. low priority background tasks (e.g. reset semester selection, save model),
  3. required/dependency (cached) tasks (e.g. login) and
  4. shutdown tasks (e.g. reset semester selection, save model)

Future Work

• cached functions and download parts should be retried immediately on temporary network failures
• use ephemeral caching on all other functions (if really needed)
• test fuse functionality, cache logic and API functions
• allow user to switch between online and offline operation
  • provide two hidden directories that also contain the full directory tree, but using only cached values (/.cached/*) / only online values (/.online/*)
  • or make appindicator allowing to show and switch modes
• make all caching and circuit-breaker related values (timeouts, thresholds,...) configurable
• properly implement and tune HTTP timeout and status code handling

[N-Coder]

Restructured issue description, initial description was

• allow custom cache validation
  • cached results containing/wrapping a failed future shouldn't be kept in cache
• failed Downloads should be dropped from cache
• when restarting a failed Download, its previously successfully downloaded parts should be kept/reused and only the failed parts/chunks retried
• add ttl-based cache clearing
• cached functions should be retried on temporary network failures
• reuse expired cache-value as fallback when new attempt fails too often or takes to long
  • this requires an algorithm for checking for network connectivity and deciding whether (or how often) a function should be retried or the expired value should be reused
  • should provide two hidden directories that also contain the full directory tree, but using only cached values (/.cached/*) / only online values (/.online/*)
• allow eager loading of certain directories (with lower priority than user/FUSE operations)
• also store cached values on disk and load then on program start
• all the above should allow offline operation

Session management is also not ready for offline operation:

• not being able to login in when starting loop will always prevent startup
• session termination during runtime is not handled

Blindly caching everything mit not be efficient enough, so for being able to persist cached values we need to

• use factory for StudIP API data model classes based on object UIDs, preventing duplicate objects
• directly cache the results of the StudIP API Session getter functions
• only persist the now unqiue instances of the StudIP API data model classes
• use ephemeral caching on all other functions (if really needed)
• this should also allow simple Mocking of the API leading to better testability

[N-Coder]

Awesome List

with generic tools to solve various of our problems here

Model Data / Schema Instance Repository

• Blog Article
• attrs / cattrs
• marshmallow
• Scrapy HTML Data Scraping Tool

Model Data / Ephemeral Data Caching

• async_lru
• aiocache
• shared / cached tasks in trio (TODO comment)

HTTP Caching

• cachecontrol
• httpcache
• requests-cache

(HTTP) Retrying

• Blog Article
  • riprova (aiohttp example)
  • backoff
  • tenacity
  • retrying
• aiohttp sample code
• urllib3 util.retry.Retry (also used by requests)

Downloads

• requests-toolbelt stream download utils

Other Event Loops

On Threads...

• Candidates
  • Eventlet
  • gevent + greenlet + libev or libuv
  • Twisted
  • Tornado
  • Asyncio
  • Curio
  • Trio (Examples / Cookbook / Patterns)
• Comparisions
  • Gevent vs Eventlet
  • Twisted / Tornado / Other Async Web Frameworks
  • On Event Loops
  • Python asyncio library diversity
  • Curio compared to Asyncio
  • Trio compared to Tornado/Twisted/asyncio/Curio
  • Curio and Trio Thoughts
  • Trio-Asyncio interoperability
  • Multio - Curio and Trio abstraction layer
• HTTP Functionality
  • Twisted + Web Client or Twisted + Requests 1 and 2
  • Tornado Web Client
  • asks - HTTP for Curio and Trio
  • unified HTTP lib
  • urllib3 generic async and h11 support (PR)
  • requests 3 (old PR)
• old info on asyncio
  • aiohttp vs. sync requests
  • Asyncio Coroutine Patterns: Beyond await

Utils

https://pythonhosted.org/aiotk/api.html
https://github.com/vodik/asyncitertools
https://github.com/asyncdef/aitertools/
http://paco.readthedocs.io/en/latest/index.html
https://github.com/python-trio/async_generator