Partition-aware data inlining

[IvensaMDH]

For partitioned tables, it would be useful if DuckLake could manage inlined data per partition rather than only at table level.

Example:

events PARTITIONED BY tenant_id

inline data:
tenant A: 150 MB
tenant B: 5 MB
tenant C: 20 KB

Ideally, DuckLake could flush only tenant A to Parquet while B and C remain inline until they have accumulated enough data. This would avoid creating small Parquet files for low-volume partitions.

The same partition awareness could also improve reads from the inline buffer. A query such as:

SELECT *
FROM events
WHERE tenant_id = 'B';

should ideally access only B's inline rows plus B's relevant Parquet files, rather than scanning inline rows for every partition.

Possible implementation approaches could include partition metadata, clustering, an internal index, or another partition-aware access structure. The exact API or implementation is less important than the behavior.

Would it make sense for DuckLake data inlining to support:

tracking accumulated inline data per partition;

selectively flushing partitions once they reach a target size;

keeping smaller partitions inline;

efficiently pruning inline data by the table's partition key?

This seems especially useful for multi-tenant streaming workloads where many tenants insert small batches at very different rates.