[#11128] fix(iceberg): Merge server IRC config for dynamic provider#11153
Open
lasdf1234 wants to merge 24 commits into
Open
[#11128] fix(iceberg): Merge server IRC config for dynamic provider#11153lasdf1234 wants to merge 24 commits into
lasdf1234 wants to merge 24 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lasdf1234
changed the title
…ogConfigLoader Extract shared loading logic under a clearer name and preserve the default catalog merge key when resolving dynamic catalog configs. Co-authored-by: Cursor <cursoragent@cursor.com>
lasdf1234
changed the title
Restore original getCatalogName parsing in StaticIcebergConfigProvider while retaining initCatalogConfigs for dynamic provider reuse, and fix default catalog server config merge via serverCatalogKey. Co-authored-by: Cursor <cursoragent@cursor.com>
Increase the default table-metadata-cache-capacity from 200 to 1000 for IRC and lakehouse-iceberg catalogs to better match production workloads. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR enhances Iceberg REST server configuration behavior by introducing server-side static config parsing that can be merged into dynamically fetched Gravitino catalog configs, while also increasing the default table-metadata cache capacity and updating related docs/metadata.
Changes:
- Refactor static catalog config parsing into a reusable
initCatalogConfigsmethod and add tests around static/dynamic config behavior. - Add server-side config merging to
DynamicIcebergConfigProviderto fill missing catalog config keys from server defaults. - Increase default
table-metadata-cache-capacityfrom 200 to 1000 and update docs + catalog property metadata accordingly.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/provider/TestDynamicIcebergConfigProvider.java | Adds test coverage for server config loading and merge behavior for default vs named catalogs |
| iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/provider/StaticIcebergConfigProvider.java | Refactors static catalog config parsing into initCatalogConfigs for reuse/testing |
| iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/provider/DynamicIcebergConfigProvider.java | Merges server-side configs into dynamic catalog configs returned from Gravitino |
| iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/IcebergConfig.java | Raises default table metadata cache capacity |
| docs/lakehouse-iceberg-catalog.md | Updates documented default cache capacity |
| docs/iceberg-rest-service.md | Updates documented default cache capacity for REST service |
| catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogPropertiesMetadata.java | Updates catalog property metadata default value |
lasdf1234
changed the title
lasdf1234
changed the title
… 1000" This reverts commit 9b6fdba.
…t catalog When the resolved catalog name matches default-catalog-name, use the default_catalog server-side IcebergConfig entry in mergeServerConfig. Add integration test for dynamic provider server config merge. Co-authored-by: Cursor <cursoragent@cursor.com>
Focus integration coverage on root-level table-metadata-cache merge and loadTable cache initialization. Extend unit tests for default and resolved catalog names and drop redundant static initCatalogConfigs coverage. Co-authored-by: Cursor <cursoragent@cursor.com>
Use Map.of for mock catalog properties in testNamedDynamicCatalogMergesNamedStaticConfigOnly. Co-authored-by: Cursor <cursoragent@cursor.com>
Collaborator
Author
|
@roryqi If you have time, please review it for me. Thank you. |
Contributor
|
Dynamic provider shouldn't contain server irc config. It's not a good design. |
Root gravitino.iceberg-rest.* settings are stored under default_catalog in initCatalogConfigs; apply them only for the configured default dynamic catalog. Use a dedicated schema name in the merge IT to avoid colliding with authorization tests. Co-authored-by: Cursor <cursoragent@cursor.com>
…mic catalog" This reverts commit 5df682e.
Align unit and integration tests with DynamicIcebergConfigProvider behavior: only catalog.<name>.* server keys merge into dynamic catalogs. Co-authored-by: Cursor <cursoragent@cursor.com>
Code Coverage Report
Files
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aligns dynamic IRC catalog configuration with the static provider layering model.
Changes include:
StaticIcebergConfigProvider.initCatalogConfigs()so server-side catalog config parsing is shared.DynamicIcebergConfigProvider, load server configs at initialization and merge them into each dynamic catalog config viaputIfAbsentinmergeServerConfig().catalog.<name>.*server keys; the default catalog also merges rootgravitino.iceberg-rest.*settings.Why are the changes needed?
Authorization-enabled deployments use the dynamic config provider. Operators can set server-level IRC properties such as
gravitino.iceberg-rest.table-metadata-cache-*(visible in boot logs), but those settings were not applied to dynamic catalogs. The static provider path already worked; this change makes the dynamic path behave the same way.Fix: #11128
Does this PR introduce any user-facing change?
Yes.
gravitino.iceberg-rest.*andcatalog.<name>.*IRC properties now apply to dynamic catalogs when keys are missing from the Gravitino catalog definition.How was this patch tested?
Commands:
./gradlew :iceberg:iceberg-rest-server:test -PskipTests -PskipDockerTests=false -PskipWeb=true \ --tests "org.apache.gravitino.iceberg.integration.test.IcebergDynamicServerConfigMergeIT"