[#9710] refactor(flink-connector): add versioned Flink catalog entries#10501
Draft
FANNG1 wants to merge 12 commits intoapache:mainfrom
Draft
[#9710] refactor(flink-connector): add versioned Flink catalog entries#10501FANNG1 wants to merge 12 commits intoapache:mainfrom
FANNG1 wants to merge 12 commits intoapache:mainfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Refactors the Gravitino Flink connector into a Spark-like multi-version layout (Flink 1.18/1.19/1.20), moving version-specific catalog/factory/SPI entrypoints and integration-test entry classes into per-version modules while keeping shared logic in flink-common. It also adjusts integration-test infrastructure (aux services + host normalization), dependency/version coordinates, and CI workflows to run the new versioned modules.
Changes:
- Introduce
flink-commonplus versionedflink-1.18/flink-1.19/flink-1.20andflink-runtime-*modules with per-version SPI and compat hooks. - Add/adjust shared connector utilities, property converters, and integration test base logic to support version-specific behavior.
- Update Gradle settings, dependency versions, and GitHub workflows/scripts for the new module structure and test execution.
Reviewed changes
Copilot reviewed 105 out of 148 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| settings.gradle.kts | Includes and maps new versioned Flink modules + flink-common. |
| gradle/libs.versions.toml | Adds per-Flink-minor version coordinates (Flink/JDBC/Iceberg/Paimon). |
| integration-test-common/src/test/java/org/apache/gravitino/integration/test/util/BaseIT.java | Adds aux-service name composition + host normalization; updates Iceberg REST URI handling. |
| integration-test-common/src/test/java/org/apache/gravitino/integration/test/util/TestBaseIT.java | Adds unit tests for new BaseIT helpers. |
| flink-connector/flink-common/build.gradle.kts | Pins flink-common to Flink 1.18 coordinates; publishes test artifacts; cleanup tweaks. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/catalog/BaseCatalog.java | Adds versioned CatalogCompat hook to create catalog tables. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/catalog/BaseCatalogFactory.java | Introduces shared factory interface for Flink catalog factories. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/utils/CatalogCompat.java | Defines per-minor compatibility interface. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/utils/DefaultCatalogCompat.java | Default implementation of catalog compat behavior. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/utils/TableUtils.java | Adds column-position conversion helper. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/utils/PropertyUtils.java | Adds helper to extract Hadoop/Hive-prefixed properties. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/store/GravitinoCatalogStore.java | Makes catalog descriptor creation overridable; hardens factory discovery against ServiceConfigurationError/LinkageError. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/store/GravitinoCatalogStoreFactoryOptions.java | Adds catalog-store factory option keys. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/PartitionConverter.java | Introduces partition conversion interface. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/DefaultPartitionConverter.java | Default identity-only partition converter implementation. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/UnsupportPartitionConverter.java | Converter for catalogs that disallow partitions. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/SchemaAndTablePropertiesConverter.java | Adds shared schema/table properties conversion interface. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/hive/FlinkGenericTableUtil.java | Adds CatalogCompat-aware serialization/deserialization of generic tables. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalog.java | Routes generic-table conversions through versioned compat hook. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFactory.java | Refactors to allow versioned newCatalog(...) override. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFactoryOptions.java | Defines Hive factory options (metastore URIs). |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/hive/HiveCatalogPropertiesConverter.java | Adds Hive catalog property converter. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalog.java | Refactors Iceberg catalog construction to accept catalog options + iceberg properties. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFactory.java | Splits out newCatalog(...) and builds iceberg-specific options map. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFactoryOptions.java | Defines Iceberg factory options. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConverter.java | Adds Iceberg property conversion logic. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/iceberg/IcebergPropertiesConstants.java | Centralizes Iceberg property keys/constants. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/GravitinoJdbcCatalog.java | Refactors JDBC catalog to accept an injected underlying AbstractCatalog. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/GravitinoJdbcCatalogFactory.java | Adds required/optional options and versioned newCatalog(...) hook; validates options. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/GravitinoJdbcCatalogFactoryOptions.java | Defines JDBC factory options (base-url/username/password/default-db/driver). |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/JdbcPropertiesConstants.java | Centralizes JDBC option/property key mappings. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/mysql/MysqlPropertiesConverter.java | Adds MySQL JDBC property converter. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/mysql/GravitinoMysqlJdbcCatalogFactory.java | Adds MySQL JDBC catalog factory implementation. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/postgresql/PostgresqlPropertiesConverter.java | Adds Postgres JDBC property converter. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/postgresql/GravitinoPostgresJdbcCatalogFactory.java | Adds Postgres JDBC catalog factory implementation. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/paimon/PaimonPropertiesConverter.java | Adds Paimon property converter. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalog.java | Introduces Paimon catalog proxy implementation. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFactory.java | Adds versioned newCatalog(...) hook for Paimon. |
| flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFactoryOptions.java | Defines Paimon factory options. |
| flink-connector/flink-common/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory | Exposes shared catalog-store factory via SPI. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/utils/TestDefaultCatalogCompat.java | Adds unit tests for DefaultCatalogCompat. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/store/TestGravitinoFlinkConfig.java | Adds tests around catalog-store client config extraction. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/store/TestGravitinoCatalogStore.java | Adds test for skipping ServiceConfigurationError during SPI discovery. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/jdbc/TestMysqlPropertiesConverter.java | Adds MySQL properties converter test suite. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/jdbc/TestPostgresqlPropertiesConverter.java | Adds Postgres properties converter test suite. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/hive/TestHivePropertiesConverter.java | Adds Hive properties converter tests. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/hive/TestFlinkGenericTableUtil.java | Updates tests for compat-aware generic table conversions. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/catalog/TestBaseCatalog.java | Updates tests to use DefaultCatalogCompat table creation. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/FlinkCommonIT.java | Uses DefaultCatalogCompat for table creation where needed. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/catalog/GravitinoCatalogManagerIT.java | Makes base IT abstract for versioned subclasses. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/jdbc/FlinkJdbcMysqlCatalogIT.java | Makes base IT abstract for versioned subclasses. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT.java | Makes base IT abstract for versioned subclasses. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT.java | Makes base IT abstract for versioned subclasses. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonCatalogIT.java | Adds shared Paimon IT base. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonLocalFileSystemBackendIT.java | Makes backend IT abstract for versioned subclasses. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonJdbcBackendIT.java | Makes backend IT abstract for versioned subclasses. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonHiveBackendIT.java | Makes backend IT abstract for versioned subclasses. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveKerberosClientIT.java | Makes base IT abstract; resolves shared hive resources via classpath. |
| flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/utils/TestUtils.java | Adds shared Flink IT helper utilities. |
| flink-connector/flink-common/src/test/resources/log4j2.properties | Adds log4j2 test configuration for Flink connector tests. |
| flink-connector/flink-common/src/test/resources/flink-tests/hive-site.xml | Adds shared Hive test resources for versioned IT modules. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/utils/CatalogCompatFlink118.java | Adds Flink 1.18 catalog compat implementation. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/GravitinoJdbcCatalogFlink118.java | Adds Flink 1.18 JDBC catalog entry implementation. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFlink118.java | Adds Flink 1.18 Iceberg catalog entry implementation. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFactoryFlink118.java | Adds Flink 1.18 Iceberg factory entry. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFlink118.java | Adds Flink 1.18 Hive catalog entry implementation. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFactoryFlink118.java | Adds Flink 1.18 Hive factory entry. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFlink118.java | Adds Flink 1.18 Paimon catalog entry implementation. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFactoryFlink118.java | Adds Flink 1.18 Paimon factory entry. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/mysql/GravitinoMysqlJdbcCatalogFactoryFlink118.java | Adds Flink 1.18 MySQL JDBC factory entry. |
| flink-connector/v1.18/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/postgresql/GravitinoPostgresJdbcCatalogFactoryFlink118.java | Adds Flink 1.18 Postgres JDBC factory entry. |
| flink-connector/v1.18/flink/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory | Updates Flink 1.18 SPI to versioned factory entries. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/catalog/GravitinoCatalogManagerIT118.java | Adds Flink 1.18 IT entrypoint class. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/jdbc/FlinkJdbcMysqlCatalogIT118.java | Adds Flink 1.18 JDBC MySQL IT entrypoint. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT118.java | Adds Flink 1.18 Iceberg REST IT entrypoint. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT118.java | Adds Flink 1.18 Iceberg Hive IT entrypoint. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveCatalogIT118.java | Adds Flink 1.18 Hive IT entrypoint. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveKerberosClientIT118.java | Adds Flink 1.18 Hive Kerberos IT entrypoint. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonLocalFileSystemBackendIT118.java | Adds Flink 1.18 Paimon FS IT entrypoint. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonJdbcBackendIT118.java | Adds Flink 1.18 Paimon JDBC IT entrypoint. |
| flink-connector/v1.18/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonHiveBackendIT118.java | Adds Flink 1.18 Paimon Hive IT entrypoint. |
| flink-connector/v1.18/flink-runtime/build.gradle.kts | Points runtime module at flink-1.18; enables service file merge. |
| flink-connector/v1.18/flink-runtime/src/test/java/org/apache/gravitino/flink/runtime/TestRuntimeJarDependencies.java | Verifies runtime jar service-descriptor merge + no SLF4J shading. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/utils/CatalogCompatFlink119.java | Adds Flink 1.19 catalog compat implementation. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/GravitinoJdbcCatalogFlink119.java | Adds Flink 1.19 JDBC catalog entry implementation. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFlink119.java | Adds Flink 1.19 Iceberg catalog entry implementation. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFactoryFlink119.java | Adds Flink 1.19 Iceberg factory entry. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFlink119.java | Adds Flink 1.19 Hive catalog entry implementation. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFactoryFlink119.java | Adds Flink 1.19 Hive factory entry. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFlink119.java | Adds Flink 1.19 Paimon catalog entry implementation. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFactoryFlink119.java | Adds Flink 1.19 Paimon factory entry. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/mysql/GravitinoMysqlJdbcCatalogFactoryFlink119.java | Adds Flink 1.19 MySQL JDBC factory entry. |
| flink-connector/v1.19/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/postgresql/GravitinoPostgresJdbcCatalogFactoryFlink119.java | Adds Flink 1.19 Postgres JDBC factory entry. |
| flink-connector/v1.19/flink/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory | Adds Flink 1.19 SPI versioned factory entries. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/catalog/GravitinoCatalogManagerIT119.java | Adds Flink 1.19 IT entrypoint class. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/jdbc/FlinkJdbcMysqlCatalogIT119.java | Adds Flink 1.19 JDBC MySQL IT entrypoint. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT119.java | Adds Flink 1.19 Iceberg REST IT entrypoint. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT119.java | Adds Flink 1.19 Iceberg Hive IT entrypoint. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveCatalogIT119.java | Adds Flink 1.19 Hive IT entrypoint. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveKerberosClientIT119.java | Adds Flink 1.19 Hive Kerberos IT entrypoint. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonLocalFileSystemBackendIT119.java | Adds Flink 1.19 Paimon FS IT entrypoint. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonJdbcBackendIT119.java | Adds Flink 1.19 Paimon JDBC IT entrypoint. |
| flink-connector/v1.19/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonHiveBackendIT119.java | Adds Flink 1.19 Paimon Hive IT entrypoint. |
| flink-connector/v1.19/flink-runtime/build.gradle.kts | Adds Flink 1.19 runtime shadow-jar build + publish config. |
| flink-connector/v1.19/flink-runtime/src/test/java/org/apache/gravitino/flink/runtime/TestRuntimeJarDependencies.java | Verifies 1.19 runtime jar shading/service-descriptor behavior. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/utils/CatalogCompatFlink120.java | Adds Flink 1.20 compat using CatalogTable builder API. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/GravitinoJdbcCatalogFlink120.java | Adds Flink 1.20 JDBC catalog entry using newer Flink JDBC factory path. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFlink120.java | Adds Flink 1.20 Iceberg catalog entry implementation. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/iceberg/GravitinoIcebergCatalogFactoryFlink120.java | Adds Flink 1.20 Iceberg factory entry. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFlink120.java | Adds Flink 1.20 Hive catalog entry implementation. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/hive/GravitinoHiveCatalogFactoryFlink120.java | Adds Flink 1.20 Hive factory entry. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFlink120.java | Adds Flink 1.20 Paimon catalog entry implementation. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/paimon/GravitinoPaimonCatalogFactoryFlink120.java | Adds Flink 1.20 Paimon factory entry. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/mysql/GravitinoMysqlJdbcCatalogFactoryFlink120.java | Adds Flink 1.20 MySQL JDBC factory entry. |
| flink-connector/v1.20/flink/src/main/java/org/apache/gravitino/flink/connector/jdbc/postgresql/GravitinoPostgresJdbcCatalogFactoryFlink120.java | Adds Flink 1.20 Postgres JDBC factory entry. |
| flink-connector/v1.20/flink/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory | Adds Flink 1.20 SPI versioned factory entries. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/catalog/GravitinoCatalogManagerIT120.java | Adds Flink 1.20 IT entrypoint class. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/jdbc/FlinkJdbcMysqlCatalogIT120.java | Adds Flink 1.20 JDBC MySQL IT override for validation behavior. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergRestCatalogIT120.java | Adds Flink 1.20 Iceberg REST IT override. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/iceberg/FlinkIcebergHiveCatalogIT120.java | Adds Flink 1.20 Iceberg Hive IT entrypoint. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveCatalogIT120.java | Adds Flink 1.20 Hive IT override for exception-path differences. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveKerberosClientIT120.java | Adds Flink 1.20 Hive Kerberos IT entrypoint. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonLocalFileSystemBackendIT120.java | Adds Flink 1.20 Paimon FS IT entrypoint. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonJdbcBackendIT120.java | Adds Flink 1.20 Paimon JDBC IT entrypoint. |
| flink-connector/v1.20/flink/src/test/java/org/apache/gravitino/flink/connector/integration/test/paimon/FlinkPaimonHiveBackendIT120.java | Adds Flink 1.20 Paimon Hive IT entrypoint. |
| flink-connector/v1.20/flink-runtime/build.gradle.kts | Adds Flink 1.20 runtime shadow-jar build + publish config. |
| flink-connector/v1.20/flink-runtime/src/test/java/org/apache/gravitino/flink/runtime/TestRuntimeJarDependencies.java | Verifies 1.20 runtime jar shading/service-descriptor behavior. |
| dev/ci/run_flink_connector_smoke.sh | Adds helper script to run per-minor Flink smoke ITs in CI. |
| .github/workflows/flink-integration-test-action.yml | Updates workflow to run v1.18 full ITs + v1.19/v1.20 smoke tests. |
| .github/workflows/backend-integration-test-action.yml | Updates test exclusions to new Flink module names. |
Comments suppressed due to low confidence (2)
flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/integration/test/hive/FlinkHiveKerberosClientIT.java:253
- Avoid using fully-qualified class names inside the method body (e.g.,
java.nio.file.Paths,org.apache.flink.util.Preconditions). Import these types instead to match the project’s Java style and keep the code easier to read/maintain.
flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/jdbc/GravitinoJdbcCatalogFactory.java:48 GravitinoJdbcCatalogFactoryOptions.DRIVERis declared as an optional option, butcreateCatalog()unconditionally removes thedriverentry fromcontext.getOptions(). This makes the option impossible to set via SQL/config and is inconsistent withoptionalOptions(). Consider preserving the value for Gravitino (e.g., copy/rename it before stripping) and only remove it from the option map passed to the underlying Flink JDBC catalog factory, or drop the option fromoptionalOptions()if it is intentionally unsupported.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR refactors the Gravitino Flink connector multi-version layout to follow a Spark-like structure for Flink
1.18,1.19, and1.20.The main changes are:
v1.18,v1.19, andv1.20flink-commonThis PR also fixes provider-specific compatibility and validation gaps:
1.19and1.20now use the newer core JDBC catalog factory path1.20keeps a version-specific assertion for the different missing-option exception pathWhy are the changes needed?
The previous Flink multi-version layout still relied too heavily on shared catalog entry implementations.
That made future extension harder when provider behavior or Flink APIs drift by minor version. In practice, JDBC, Iceberg REST, and Hive validation paths already showed real version-specific differences.
This PR makes each Flink minor version own its catalog/factory/IT entry layer while still reusing shared base logic in
flink-common. That keeps the codebase closer to the Spark connector approach and makes future extensions, including Flink2.x, easier to isolate and review.Fix: #9710
Does this PR introduce any user-facing change?
Yes.
The produced Flink connector artifacts still target Flink
1.18,1.19, and1.20, but the internal implementation is now explicitly version-scoped:This does not change the user-facing SQL syntax, but it changes how the connector is organized and validated internally.
How was this patch tested?
The following validations were run:
:flink-connector:flink-1.18:test -PskipITs:flink-connector:flink-1.19:test -PskipITs:flink-connector:flink-1.20:test -PskipITsRepresentative version-specific integration tests were also run:
GravitinoCatalogManagerIT118GravitinoCatalogManagerIT119GravitinoCatalogManagerIT120Provider-specific versioned integration tests were run for all supported minors:
FlinkPaimonLocalFileSystemBackendIT118FlinkPaimonLocalFileSystemBackendIT119FlinkPaimonLocalFileSystemBackendIT120FlinkJdbcMysqlCatalogIT118FlinkJdbcMysqlCatalogIT119FlinkJdbcMysqlCatalogIT120FlinkIcebergRestCatalogIT118FlinkIcebergRestCatalogIT119FlinkIcebergRestCatalogIT120FlinkHiveCatalogIT118FlinkHiveCatalogIT119FlinkHiveCatalogIT120Additional test:
:integration-test-common:test --tests "org.apache.gravitino.integration.test.util.TestBaseIT" -PskipITs