[#7836] feat(storage): Add partition statistics storage interface#7849
Closed
roryqi wants to merge 105 commits intoapache:mainfrom
Closed
[#7836] feat(storage): Add partition statistics storage interface#7849roryqi wants to merge 105 commits intoapache:mainfrom
roryqi wants to merge 105 commits intoapache:mainfrom
Conversation
### What changes were proposed in this pull request? Validate JDBC connection for unsafe operations. ### Why are the changes needed? To stop potentially unsafe operations. Fix: apache#7547 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Locally, plus added unit tests. --------- Co-authored-by: Mini Yu <yuqi@datastrato.com>
…expression (apache#7582) <!-- 1. Title: [#<issue>] <type>(<scope>): <subject> Examples: - "[apache#123] feat(operator): support xxx" - "[apache#233] fix: check null before access result in xxx" - "[MINOR] refactor: fix typo in variable name" - "[MINOR] docs: fix typo in README" - "[apache#255] test: fix flaky test NameOfTheTest" Reference: https://www.conventionalcommits.org/en/v1.0.0/ 2. If the PR is unfinished, please mark this PR as draft. --> ### What changes were proposed in this pull request? This is a refactor of builder method for `LiteralDTO`, `FieldReferenceDTO`, `FuncExpressionDTO`, and `UnparsedExpressionDTO` that is proposed in the [comments](apache#7498 (review)) of the previous PR. ### Why are the changes needed? We need to support Column and its default value in python client. apache#5202 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests --------- Signed-off-by: George T. C. Lai <tsungchih.hd@gmail.com>
### What changes were proposed in this pull request? unifed code style, the boolean type check use ` if( exp ) or if( !exp ) rather than if ( exp == true ), if (exp == false ) ` ### Why are the changes needed? close apache#7596 ### Does this PR introduce _any_ user-facing change? No, just code style change ### How was this patch tested? using existing tests
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
Examples:
- "[apache#123] feat(operator): support xxx"
- "[apache#233] fix: check null before access result in xxx"
- "[MINOR] refactor: fix typo in variable name"
- "[MINOR] docs: fix typo in README"
- "[apache#255] test: fix flaky test NameOfTheTest"
Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. If the PR is unfinished, please mark this PR as draft.
-->
### What changes were proposed in this pull request?
clean up unused method in `NameIdentifierUtil` class
### Why are the changes needed?
close issue apache#7588
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
using existing tests
…vent listener (apache#6485) ### What changes were proposed in this pull request? Adding `transformPreEvent` method to `EventListenerPlugin` to make event listener modify pre-events. ```java public interface EventListenerPlugin { default SupportsChangingPreEvent transformPreEvent(SupportsChangingPreEvent preEvent) { return preEvent; } } ``` ### Why are the changes needed? Fix: apache#6486 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? change the iceberg request and it take affects when creating table in local enviroment
### What changes were proposed in this pull request? update github issue template, update version dropdown list
…2) (apache#7361) ### What changes were proposed in this pull request? Add policy entity and POConverters ### Why are the changes needed? Fix: apache#7552 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? tests added
…apache#7578) ### What changes were proposed in this pull request? Support MySQL external type with Trino. ### Why are the changes needed? Fix: apache#7338 Fix: apache#7339 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? local integration-test
### What changes were proposed in this pull request? This PR proposes to add the Java and Python API for Job System. ### Why are the changes needed? This is the first work of apache#7154 Fix: apache#7503 ### Does this PR introduce _any_ user-facing change? Yes ### How was this patch tested? Tests will be added later on.
### What changes were proposed in this pull request? Fix catalog ### Why are the changes needed? So it correct Fix: #N/A ### Does this PR introduce _any_ user-facing change? None ### How was this patch tested? N/A
…7600) ### What changes were proposed in this pull request? The `EntityType ` class has short name variable, but it's duplicated. Table and Tag both used `ta`. And the short name is not used anymore. This pr is try to clean up related code. ### Why are the changes needed? Fix: apache#7597 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Use existing tests.
…#7651) ### What changes were proposed in this pull request? This PR proposes the removal of duplicated code that appears twice in the CLI module. ### Why are the changes needed? Fix: apache#7639 ### Does this PR introduce _any_ user-facing change? None ### How was this patch tested? I verified this patch by reviewing and running the existing test clinets:cli
### What changes were proposed in this pull request? Add a null check before entity type check. ### Why are the changes needed? Previously, `validateEntityHasIdentifier` only validates the entity type but doesn't check if the entity is null before accessing its properties, which may cause misleading error messages. Fix: apache#7636 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests.
…mplate (apache#7657) ### What changes were proposed in this pull request? Add a call of the existing `validation` method before object construction in `build`. ### Why are the changes needed? `validate` method exists with proper validation logic but is never called during object construction. Fix: apache#7640 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests.
### What changes were proposed in this pull request? Added 2 more precondition checks to `Namespace`: 1. leading single dot 2. contains double dot ### Why are the changes needed? To enforce a stricter namespace validation. Fix: apache#7635 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests.
### What changes were proposed in this pull request? Replace the second usage of `httpRequest.getHeader(FilesetAuditConstants.HTTP_HEADER_FILESET_DATA_OPERATION)` with `dataOperation` which was previously defined. ### Why are the changes needed? Cleaner and less code. Fix: apache#7632 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests.
…ashCode method of SparkJobTemplate (apache#7653) ### What changes were proposed in this pull request? Added Missing @OverRide annotation ### Why are the changes needed? Fix: apache#7641 ### Does this PR introduce _any_ user-facing change? None ### How was this patch tested? Execute existing api module test
…nMetadataObject (apache#7650) ### What changes were proposed in this pull request? Added a check for empty names during validation in validateAuthorizationMetadataObject Fix: apache#7592 ### Does this PR introduce any user-facing change? No ### How was this patch tested? Tested with a unit test
… in HTTPClient (apache#7601) ### What changes were proposed in this pull request? Added a null check for empty headers before adding request headers in HTTPClient. Closes: [apache#7589](apache#7589) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added a unit test.
### What changes were proposed in this pull request? fix index existence check ### Why are the changes needed? Fix: apache#7591 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? tests added. Co-authored-by: senlizishi <weixubin@gmail.com>
### What changes were proposed in this pull request? Add support for MODEL provider in CLI catalog creation command by: - Add MODEL constant definition - Add MODEL to valid providers set - Handle MODEL provider in internal() method - Set catalog type to MODEL for model provider ### Why are the changes needed? Currently, the CLI throws "Unsupported provider: model" error when trying to create a model catalog. This prevents users from using the CLI to manage model catalogs, even though the server supports model catalogs. Fix: CLI does not support create model catalog now ### Does this PR introduce *any* user-facing change? Yes, this PR introduces the following user-facing change: - Users can now use CLI command `gcli.sh catalog create --metalake <metalake> --name <name> --provider model` to create model catalogs - The CLI will no longer throw "Unsupported provider: model" error for model provider ### How was this patch tested? - Built the project successfully with `./gradlew build` - Ran CLI tests with `./gradlew :clients:cli:test` - all tests passed - **Manual testing**: CLI command `catalog create --metalake yedian --name model --provider model` now successfully creates model catalog instead of throwing "Unsupported provider: model" error - Verified the fix by starting local Gravitino server and testing the complete workflow
…che#7654) ### What changes were proposed in this pull request? Added DB name validation to the createDatabase method to secure DB creation. Fix: apache#7585 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested with unit tests to validate if invalid names are rejected and the names in the TestDatabaseName enum are accepted
### What changes were proposed in this pull request? Logging configuration moved to module level, handler added only once when module is imported. ### Why are the changes needed? The previous logger implementation in `http_client.py` creates a log handler every time a `Response` object is instantiated, which may cause log message duplication and consume unnecessary memory over time. Fix: apache#7583 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests for `client-python`.
…QL (apache#7733) ### What changes were proposed in this pull request? Support show index info with MySQL ### Why are the changes needed? `show create table` would display the index info of MySQL table. ```sql -- In MySQL CREATE TABLE gt_mysql_test_index.demo_with_primary_key_and_unique_key ( key1 INT, key2 INT, col1 INT, primary key (key1), unique key unique_key1 (key2) ); -- In Trino -- show create table CREATE TABLE gt_mysql.gt_mysql_test_index.demo_with_primary_key_and_unique_key ( key1 integer NOT NULL, key2 integer, col1 integer ) COMMENT '' WITH ( engine = 'InnoDB', primary_key = ARRAY['key1'], unique_key = ARRAY['unique_key1:key2'] ) ``` Fix: apache#7701 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? local tests
Signed-off-by: zacsun <zacsun@lilith.com>
Signed-off-by: zacsun <zacsun@lilith.com>
Signed-off-by: zacsun <zacsun@lilith.com>
Signed-off-by: zacsun <zacsun@lilith.com>
Signed-off-by: zacsun <zacsun@lilith.com>
### What changes were proposed in this pull request? Include useful info e.g. user, operation, metadata object in the error messages when auth fails. ### Why are the changes needed? Fix: apache#7753 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `org.apache.gravitino.server.web.filter.TestGravitinoInterceptionService`
### What changes were proposed in this pull request? change content type to 'application/x-www-form-urlencoded' for token api ### Why are the changes needed? login page has been blocked by CORS policy Fix: apache#7541 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? manually
…h multiple URIs (apache#7622) ### What changes were proposed in this pull request? This PR implements operations for model version with multiple URIs. ### Why are the changes needed? Fix: apache#7382 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT. --------- Co-authored-by: zhanghan <zhanghan18@xiaomi.com>
apache#7671) ### What changes were proposed in this pull request? Add interfaces and related classes for StarRocks, implementation will be in the next PR. ### Why are the changes needed? To support more catalogs. Fix: apache#7667 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? Test locally.
…ne. (apache#7785) ### What changes were proposed in this pull request? Increase the timeout time for build task from 30m to 60m. ### Why are the changes needed? 30m is not enough as the project introduces more modules. Fix: apache#7784 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? Exsiting CI.
…apache#7564) ### What changes were proposed in this pull request? add a basic framework for policy management ### Why are the changes needed? Fix: apache#7143 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Tests will be added after the logic is implemented.
…mns with precision (apache#6657) ### Why are the changes needed? Fix: apache#6656 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UT already added. --------- Signed-off-by: zacsun <zacsun@lilith.com>
…ts for empty strings (apache#7755) ### What changes were proposed in this pull request? Update strToSeq to return empty lists for empty strings ### Why are the changes needed? Fix: apache#7702 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added a unit test
…thorization (apache#7736) ### What changes were proposed in this pull request? Support user、role、permission、grant operation authorization ### Why are the changes needed? close apache#7737 ### Does this PR introduce _any_ user-facing change? None ### How was this patch tested? org.apache.gravitino.client.integration.test.authorization.UserAuthorizationIT org.apache.gravitino.client.integration.test.authorization.PermissionAuthorizationIT org.apache.gravitino.client.integration.test.authorization.RoleAuthorizationIT org.apache.gravitino.client.integration.test.authorization.GroupAuthorizationIT org.apache.gravitino.client.integration.test.authorization.OwnerAuthorizationIT
…part-2) (apache#7772) ### What changes were proposed in this pull request? This is the second part of the job framework PR to add the job executor interface and complete the job framework. ### Why are the changes needed? Fix: apache#7505 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add UTs to cover the code changes.
…r gravitino client (apache#7495) ### What changes were proposed in this pull request? Support pass custom http client config when using the Java Gravitino Client. ### Why are the changes needed? - We can config connectionTimeout or socketTimeout and so on. Fix: apache#7456 ### Does this PR introduce _any_ user-facing change? - support http client config |config | default | Description | | ----------- |----------- |----------- | |gravitino.client.connectionTimeoutMs | null | An optional http connection timeout in milliseconds| |gravitino.client.socketTimeoutMs | null | An optional http socket timeout in milliseconds | - add `withClientConfig` in `org.apache.gravitino.client.GravitinoClientBase.Builder` ``` Map<String, String> properties = ImmutableMap.of( "gravitino.client.connectionTimeoutMs", "10", "gravitino.client.socketTimeoutMs", "10" ); GravitinoClient.builder("uri") .withMetalake("metalake") .withClientConfig(properties) // add custom client config (optional) .builder(); GravitinoAdminClient.builder("uri") .withClientConfig(properties) // add custom client config (optional) .builder(); ``` ### How was this patch tested? original Uts Co-authored-by: Jerry Shao <jerryshao@datastrato.com>
### What changes were proposed in this pull request? add StarRocks Catalog Implement ### Why are the changes needed? To support StarRocks Catalog. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By E2E test, the test is in another pr apache#7792
### What changes were proposed in this pull request? Adding configs required for Azure based oAuth login These will not cause any problems with the current oAuth as these are just additional configs exposed in /configs endpoint ### Why are the changes needed? These changes will be used in future PRs for Azure MSAL based oAuth support in the front end code, through /configs endpoint Fix: apache#7538 ### Does this PR introduce _any_ user-facing change? This does not introduce user facing changes. Only changing the /configs API response for now ### How was this patch tested? (Please test your changes, and provide instructions on how to test it: Tested in local deployment. Enabled Azure oauth by ``` gravitino.authenticators = oauth gravitino.authenticator.oauth.provider = azure ``` and set gravitino.authenticator.oauth.azure.client-id , gravitino.authenticator.oauth.azure.authority, gravitino.authenticator.oauth.azure.redirect-uri, gravitino.authenticator.oauth.azure.jwks-uri, gravitino.authenticator.oauth.azure.scope along with existing oauth required parameters (we can modify this in future to make it such that if provider is set, the current oauth params need not be "required") Configs endpoint returns: <img width="1135" height="282" alt="Screenshot 2025-07-22 at 12 05 23 PM" src="https://github.com/user-attachments/assets/f0801088-5330-4e64-b2e3-ff9ad7bc02a3" />
…imitation (apache#7808) ### What changes were proposed in this pull request? - The `primary key` must be `not null` - The auto increment `unique key` must be `not null` ### Why are the changes needed? Fix: apache#7807 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? local tests
…etadata object. (apache#7830) ### What changes were proposed in this pull request? Use set to remove duplicated tags when list all tags for a metadata object. ### Why are the changes needed? We need to eliminate duplicates. Fix: apache#7688 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? UT
…info with MySQL (apache#7810) ### What changes were proposed in this pull request? Support create mysql table with index info in Trino like this: ```sql CREATE TABLE gt_mysql.gt_mysql_test_index.demo_with_primary_key_and_unique_key ( key1 integer NOT NULL, key2 integer, col1 integer ) COMMENT '' WITH ( engine = 'InnoDB', primary_key = ARRAY['key1'], unique_key = ARRAY['unique_key1:key2'] ) ``` ### Why are the changes needed? Fix: apache#7809 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? local tests
apache#7767) ### What changes were proposed in this pull request? Launch an init container named sqlfile to mount the scripts path to pod shared volumes. Launch another init container named init-sql to init the table that Gravitino required. Remove the scripts folder in chart. ### Why are the changes needed? Fix: apache#7766 ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? CI
### What changes were proposed in this pull request? Support check deny privilege ### Why are the changes needed? Fix: apache#7795 ### Does this PR introduce _any_ user-facing change? None ### How was this patch tested? Existing metadata authentication for IT and UT
…tions (apache#7818) ### What changes were proposed in this pull request? Just documents. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No need.
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
Examples:
- "[apache#123] feat(operator): support xxx"
- "[apache#233] fix: check null before access result in xxx"
- "[MINOR] refactor: fix typo in variable name"
- "[MINOR] docs: fix typo in README"
- "[apache#255] test: fix flaky test NameOfTheTest"
Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. If the PR is unfinished, please mark this PR as draft.
-->
### What changes were proposed in this pull request?
add StarRocks Catalog Implement
### Why are the changes needed?
To support StarRocks Catalog.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By E2E test, the test is in another pr
apache#7792
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add partition statistics storage interface
Why are the changes needed?
Fix: #7836
Does this PR introduce any user-facing change?
Developer API, not for users.
How was this patch tested?
Just an interface.