Table-level conflict checks for concurrent insert and delete are too coarse

[tjnicolson]

Thank you for creating a wonderful piece of technology. DuckLake is fantastic ❤️.

#906

We are trying to parallelise ingestion of customer data into our DuckLake, deduping input data based on customer_id + some table specific fields. We are partitioning DuckLake by customer_id, and ensure that each customer is only ingested by one process at a time. However, the conflict detection in the above PR is kicking in, effectively forcing us to serialise the ingestion.

With our partitioning, if the conflict resolution was implemented at the file level we would be able to parallelise the ingestion.

Given the scale at which DuckLake is designed to operate at, serialising such writes per table is a significant limitation.

Would it be difficult to support file level checks?

(There are quite a few old discussing similar to this (e.g. #618), but I didn't find any since the PR above which introduced the table-level restriction and asked the titular question).

[tjnicolson]

Hey @pdet, I believe you established the file-level precedent here. What do you think?