Local disk caching of remote file ranges

[jbit84]

For our analytical use cases the missing 'killer feature' for DuckLake would be an automatic disk—based caching feature working in the interplay between DuckDB and DuckLake. This would be a natural continuation of the External File Cache introduced in 1.3.0.

I would not be surprised if it's already in the roadmap, but as it has not been explicitly communicated I wanted to make a feature request.

Our users do exploratory/research work, and often end up rerunning similar or slightly tweaked queries against the same data sets. Being able to cache the 'working set' of row groups would significantly:

• Reduce cost of data transfer from cloud storage
• Speed up queries as data would be loaded from fast local SSD

I realize the bulk of the implementation would likely be in the DuckDB core, but value wise this feature really kicks in as an integral part of the DuckLake offering.

With this in place users would benefit from:

• Cloud for authoritative, 'multi-player capable' and durable storage (and meta—data)
• Local SSD and CPU cores for fast and price efficient execution

thus utilizing the strength of both, and making an architectural bet that align with expected near future improvements to costs for SSD storage and 'large core count' local compute.

Hoping to hear that this aligns with what you are already planning 😀

[ahawker]

I'd (also) love to see this functionality mainlined but there is https://duckdb.org/community_extensions/extensions/cache_httpfs.html which may fit your use case.

[shaneharter]

I can see this is a very old issue, but I wanted to add my endorsement.