Skip to content

Conversation

ianton-ru
Copy link

@ianton-ru ianton-ru commented Jul 11, 2025

Setting iceberg_metadata_file_path is sent to swarm node in distributed queries.

Solve #897.

For working with Iceberg table clickhouse needs Iceberg metadata file. When info about metadata absent in query, clickhouse tries to find it on storage. For this clickhouse uses listObject request.
https://github.com/Altinity/ClickHouse/blob/antalya-25.3/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp#L350

S3 Tables don't support listObject request. S3 tables works only with catalog, and in single-node variant clickhouse gets it from catalog and adds to settings:
https://github.com/Altinity/ClickHouse/blob/antalya-25.3/src/Databases/DataLake/DatabaseDataLake.cpp#L397
and later doesn't search anything, just read metadata by direct path. So this case works.

But in distributed requests swarm node does not have access to catalog, so tries to find metadata, and query failed with error

Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: Received from clickhouse2:9000. DB::Exception: Unable to parse ExceptionName: MethodNotAllowed Message: The specified method is not allowed against this resource. (Code: 100, S3 exception: 'MethodNotAllowed'). (S3_ERROR)

This PR adds iceberg_metadata_file_path to query when send it to swarm nodes.

Actual changes only in DataLakeConfiguration::createArgsWithAccessData.
https://github.com/Altinity/ClickHouse/pull/898/files#diff-589e6bce39764a9388b8e33f34dfa0d6b7c58d3874fd8b55a63ad02905cf4b90R219

But also I made refactoring, split file DataLakeConfiguration.h on h and cpp parts, all other methods moved as is, without changes.

@Enmk Enmk merged commit 143c05f into antalya-25.3 Jul 12, 2025
455 of 522 checks passed
@svb-alt svb-alt added port-antalya PRs to be ported to all new Antalya releases antalya-25.6 and removed antalya-25.6 labels Jul 14, 2025
Enmk added a commit that referenced this pull request Sep 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
antalya antalya-25.3 antalya-25.3.3 port-antalya PRs to be ported to all new Antalya releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants