Refactor code to support concurrent SELECT queries in Iceberg#86062
Merged
Refactor code to support concurrent SELECT queries in Iceberg#86062
Conversation
Contributor
divanik
changed the title
clickhouse-gh
bot
added
the
pr-bugfix
divanik
force-pushed
the
divanik/support_concurrent_queries_to_iceberg_table_3
branch
2 times, most recently
from
82dc20e to
914f93f
Compare
…k/support_concurrent_queries_to_iceberg_table_3
divanik
force-pushed
the
divanik/support_concurrent_queries_to_iceberg_table_3
branch
from
914f93f to
d10eb59
Compare
divanik
added
the
no-fast-tests
divanik
force-pushed
the
divanik/support_concurrent_queries_to_iceberg_table_3
branch
from
4c546d3 to
2995b4e
Compare
divanik
force-pushed
the
divanik/support_concurrent_queries_to_iceberg_table_3
branch
2 times, most recently
from
380c964 to
dc3bfbd
Compare
kssenii
reviewed
Sep 23, 2025
kssenii
reviewed
Sep 23, 2025
src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp
Outdated
Show resolved
Hide resolved
kssenii
reviewed
Sep 23, 2025
kssenii
reviewed
Sep 23, 2025
kssenii
reviewed
Sep 23, 2025
kssenii
reviewed
Sep 23, 2025
kssenii
reviewed
Sep 23, 2025
kssenii
reviewed
Sep 23, 2025
| /// This will update metadata for table function which contains specific information about table | ||
| /// state (e.g. for Iceberg). It is done because select queries for table functions are executed | ||
| /// in a different way and clickhouse can execute without calling updateExternalDynamicMetadataIfExists. | ||
| if (!do_lazy_init && is_table_function && configuration->needsUpdateForSchemaConsistency()) |
Member
There was a problem hiding this comment.
What if do_lazy_init == true? It currently is for cluster functions...
Member
Author
There was a problem hiding this comment.
Tests didn't break in a such scenario. Maybe we can add do_lazy_init = ... && !is_table_function and avoid this check here?
Member
There was a problem hiding this comment.
It used to be so initially, but the behaviour was changed on purpose #83052 🤔
kssenii
approved these changes
Sep 24, 2025
src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
Outdated
Show resolved
Hide resolved
kssenii
reviewed
Sep 25, 2025
src/Storages/ObjectStorage/DataLakes/DataLakeTableStateSnapshot.cpp
Outdated
Show resolved
Hide resolved
kssenii
reviewed
Sep 25, 2025
src/Storages/ObjectStorage/DataLakes/DataLakeTableStateSnapshot.cpp
Outdated
Show resolved
Hide resolved
src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergTableStateSnapshot.cpp
Outdated
Show resolved
Hide resolved
src/Storages/ObjectStorage/tests/gtest_datalake_table_state_serde.cpp
Outdated
Show resolved
Hide resolved
divanik
deleted the
divanik/support_concurrent_queries_to_iceberg_table_3
branch
robot-ch-test-poll4
added
the
pr-synced-to-cloud
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Iceberg table state is not stored in a storage object anymore. This should make Iceberg in ClickHouse usable with concurrent queries