deltalake/iceberg/hudi improvements

[kssenii]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Several improvements around data lakes:

• Make StorageIceberg work with non-partitioned data.
• Support Iceberg format version V2 (previously only V1 was supported)
• Support reading partitioned data for DeltaLake/Hudi
• Faster reading of DeltaLake metadata by using Delta's checkpoint files
• Fixed incorrect Hudi reads: previously it incorrectly chose which data to read and therefore was able to read correctly only small size tables
• Made these engines to pickup updates of changed data (previously the state was set on table creation)
• Make proper testing for Iceberg/DeltaLake/Hudi using spark

[AmosG]

can a test be added for partitioned delta tables?
i think there is an issue that clickhouse ignores the partitions

[amosdoublec]

what about support for deltalake timetravel
ie
select * from table@v2
union all
select * from table@v1

[kssenii]

what about support for deltalake timetravel
ie
select * from table@v2
union all
select * from table@v1

yes, but not in this PR, could you please create an issue feature request for this?

[ucasfl]

Also, it worth to support reading the table with cluster.

[antonio2368]

Initial review of the code only, didn't check tests.
Looks really clean, good job!

[kssenii]

Hi @kssenii
Thanks for this very valuable PR, it's amazing !
Can you please tell us what do you mean by "Make StorageIceberg work, previously it didn't." ?
Thanks

@alifirat, when I and some other people tested it - it did not work. I checked what changes I made to make it work and looks like it was able to work only if data was partitioned (because it incorrectly worked with paths).

[antonio2368]

LGTM

[kssenii]

Integration tests (tsan) [5/6] — fail: 2

test_tcp_handler_interserver_listen_host/test_case.py::test_request_to_node_with_interserver_listen_host
test_tcp_handler_interserver_listen_host/test_case.py::test_request_to_node_without_interserver_listen_host

Integration tests flaky check (asan)

test_storage_s3/test.py::test_select_columns

 DB::Exception: Message: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

not related to changes

Stress test (debug) — Killed by signal

#48777

[alifirat]

Hi @kssenii I just see that you have merged this PR. Do you think it's going to be available in your monthly release or backport to the existing releases ? Thanks !

[kssenii]

@alifirat, it will be available only in the next release, to the existing releases it will not be backported.

[ucasfl]

For Iceberg V2, it may have delete files, current implementation seems can not handle it.