[RFC] Multi-catalog support

[zfarrell]

Proposal
We would like DuckLake to natively support multiple logical catalogs in a single metadata database and a single storage root, instead of requiring a separate set of metadata tables per catalog.

Today one set of DuckLake metadata tables is one catalog. You can already host more than one catalog in a single database by using the METADATA_SCHEMA option on ATTACH to put each catalog's metadata tables in its own SQL schema. That works, but it does not scale for us: it means maintaining N copies of the full metadata table set, one per catalog, with all the schema management and overhead that implies. For a multi-tenant service with many small catalogs that is a lot of duplicated structure to provision and operate.

We want the opposite: one shared set of metadata tables, with clean isolation between catalogs built into the model.

We have built and run this in production as a DataFusion extension on top of DuckLake, against Postgres. We built it in a way that can live separately from DuckLake (it's a superset of functionality), but our preference would be to roll this into the core DuckLake spec (and we genuinely think this is important for DuckLake to be competitive!).

How we built it
A thin, additive layer. The standard DuckLake tables keep their shape. We added a small registry of catalogs plus two map tables that record which snapshots and schemas belong to which catalog. Each catalog's head is the latest snapshot mapped to it, so one catalog committing never moves another's head. Files are segregated by path under the shared root, so two catalogs can both own a public schema without colliding. Because the base tables are untouched, the store is still valid single-catalog DuckLake and existing readers keep working.

Known issues
We kept ownership in map tables for one reason: we did not want to fork the schema. As mentioned above, the current design is a balance between the best option and one that allows us to not fork the DuckLake spec. The current map tables is probably not the right call for core. If you own the spec, denormalizing a catalog_id onto the metadata tables would be simpler and faster, especially for vacuum and expiration, which are our heaviest queries. We would lean that way for core and are happy to walk through the tradeoff.

Related
A few existing threads point at the same gap from the workaround side:

• Feature Request: Postgres catalog connection pooling across many attached catalogs #447 connection pooling across many attached catalogs. Attaching ~200 catalogs from one Postgres database, hitting connection limits because each attachment keeps its own pool. The same scaling wall, seen from the connection angle.
• Multiple catalog pointers when attaching to multiple ducklake databases with a single data catalog #833 multiple catalog pointers with a single data catalog. A Snowflake-stylADATA_SCHEMA, asking for a cleaner result than the duplicate metadata pointersthe per-schema approach produces.

Both are people working around the absence of shared-metadata multi-catalog. We think solving it in the model is the better fix.

[sandstrom]

This would be very useful. We are using vanilla DuckDB files, one per tenant, today.

But would like to try out Ducklake + Quack-based server instead, for various reasons.

And are running into the same issue that our current tenant isolation, one duckdb file per tenant, doesn't have an obvious equivalent in Ducklake.

[jdctinuiti]

+1 in interest for this! Would love to hear about how we can implement this for Quack catalogs as well