Feature Request: add a time filter to ducklake_merge_adjacent_files

[zhming0]

I run a scheduled compaction job for append only time series tables, and it gets slower as the tables grow. ducklake_merge_adjacent_files looks at every live file in the table on each run, but only files written since the last run can be new merge candidates in my use case. min_file_size/max_file_size help a bit, but they can't express "only recent files".

Would you be open to a newer_than => TIMESTAMPTZ option, so a scheduled job can do:

CALL ducklake_merge_adjacent_files('my_catalog', newer_than => now() - INTERVAL 1 DAY);

and only consider files written after that point?

For context, I'm building o11ylite, an observability app that stores logs, traces and metrics in DuckLake with frequent small ingests and periodic compaction. A time filter would drastically reduce my compaction cost on production, 🙏🏿 🙏🏿 .

I prepared a PR here: #1363