Hot/cold storage tiering via metadata path rewrites — dangerous?

[ekkuleivonen]

TL;DR

Can I safely archive Parquet files by copying them to cold S3 and updating ducklake_data_file.path + path_is_relative=false so a single table spans hot + cold storage?

Context

Using DuckLake with:

• Postgres metadata
• S3 Parquet storage
• DuckDB queries

Observed path model:

data_path → schema.path → table.path → file.path

All currently path_is_relative = true.

Idea

Archive older data by:

1. Copy Parquet → cold S3
2. In one Postgres TX:
   • update ducklake_data_file.path → absolute path (same for delete files too i guess)
   • set path_is_relative = false
3. Commit

Result:

• hot + cold files coexist in one table
• no table rebuilds

Questions

• Is post-hoc mutation of path / path_is_relative supported?
• Any invariants this breaks (snapshots, delete files, cleanup)?
• Does DuckLake assume a single storage root?
• Any plans for native tiering / relocation?

Happy to share experiments if helpful!

Rock on,

• Ekku

[ekkuleivonen]

Update:

Thought I'd add the use-case that lead me to this idea:

What if I could archive certain partitions of a given table into cold storage?

Potential queries hitting the cold partitions should keep working like yesterday, but with degraded performance.

Same goes for whole tables, and schemas even.