how to incorporate hierarchical caching into dask?

[mrocklin]

Migrated from mrocklin fork. Originally posted by @phrrngtn

[another feature request coming out of sidebar conversation with mrocklin]

apologies for the very half-baked suggestion. I have not worked with dask enough to identify clearly what it is that I think I want!

Use case: I want complicated dasks on immutable/append-only data to re-use previously computed results if they are available.

hierarchical means that dask sub-trees can be cached (not just the whole thing). OK with cache busting if the scheduler/optimizer picks a different tree. Assume that cache configuration has to be tied to the scheduler (i.e. the thing that runs the dask rather than the dask itself?)

[mrocklin]

This is mostly solved by cachey and #502 . This could still use actual use on real problems but, for the most part, this feels done.

[phrrngtn]

great! I only just recently got my own 'expression tree rewriting' stuff
done: execute our Python-based internal DSL with what is essentially a
mocked out implementation and run a visitor on the tree to rewrite it to
the 'real' implementation. I will attempt a visitor that targets dasks and
cachey.

pjjH