Ducklake spec clarifications needed

[apatrida]

Given I am about to start write support for our ducklake implementation, I have not checked everything, but some issues come up at first glance:

Spec vs actual column schema drift — ducklake_column has default_value_type and default_value_dialect in practice but these aren't in the spec docs. ducklake_schema_versions has table_id in practice. Implementers need to handle optional columns gracefully or they'll break on real catalogs.

snapshot_changes.changes_made format undocumented — values like inlined_insert:tablename, inline_flush:..., merge_adjacent:... aren't specified. If we want to write interoperable change entries, we're guessing at the format.

partial_max in ducklake_data_file — often populated in real catalogs, not well documented. Unclear semantics for implementers.

No schema for snapshot commit protocol — the spec describes the tables but not the write protocol (what order to insert rows, how to allocate IDs, atomicity requirements). Every implementer has to reverse-engineer this from DuckDB's behavior.

No guidance on concurrent writers — the spec doesn't address what happens when two engines write to the same catalog simultaneously. Optimistic locking? Last-writer-wins? Undefined.

[pdet]

Hi @apatrida can you share a bit more about what you mean with ducklake support here? Or what you are doing exactly?

[apatrida]

I am writing a plugin for Trino to support ducklake format, and have read only support mostly complete, and starting on the write side. Looking at how duckdb ducklake extension manages catalog and the ducklake spec for the catalog, there are some areas of confusion. I created other discussion topics for bigger items, and here was my leftover list of smaller things that were less clear to me.

[pdet]

Got it!

Just to give you a clear timeline, we are releasing DuckLake 1.0 in the next few weeks, and after the release we will update our spec and refactor it in order to facilitate independent DuckLake implementations like yours. So feel free to dump any issues you encounter with the spec in here, they will absolutely be valuable when we start that.

Regarding any current mismatches between the spec and the implementation, the implementation is the source of truth. Until 1.0, we have been focusing on moving the implementation faster, which sometimes means that the specs don't keep up. After the 1.0 release, this model will shift. (i.e., we will maintain the specs absolutely up to date with released DuckLake versions, and DuckLake itself will be BWC.)

I think even a PR to Trino with only read capabilities would already be an excellent start!

[apatrida]

Yes, the read side is a good start and most can clearly be implemented. I am far along with write as well as a starting point, at least minimally to have views for Trino which means a little more write infra in the code is necessary. It's moving along, bit by bit. I will plan to adjust things after 1.0 for sure, just trying to be "nearby the feature set and ready for that moment.

Looking at what I can share conceptually with Doris as well. That effort will start as well for the read side.

[apatrida]

by the way, my testing is in a matrix:

• duckdb write -> trino read
• trino write -> duckdb read
• trino write -> trino read

if that works, I'm fairly happy. I'm ignoring MYSQL catalog for a moment as the duckdb write side of that has too many issues for now, and will re-address it when my write side is done.

[apatrida]

@pdet 1.0 is out, congrats! where do we go from here. We have read/write Trino and are updating to any changes from 1.x and about to take our core catalog management and isolate it into a JVM package (and likely Kotlin mutli-platform for native/wasm/other platforms) for general use across plugins, and then add to Doris as well. Full read/write. But, there are issues.

Can we find a way to track and talk about all issues that other dialects/engines will face?

[apatrida]

see: https://github.com/brikk/ducklake-integrations/tree/main/jvm/trino-ducklake

[pdet]

Hi @apatrida, that sounds awesome!

I think the best way to move forward is to open issues in https://github.com/duckdb/ducklake-web/issues with either issues or clarification questions related to the spec, as pain points other developers face will most likely require us to patch the spec on the website.

[apatrida]

will do