[Draft] Master catalog spi review 17#65860
Closed
morningman wants to merge 71 commits into
Closed
Conversation
morningman
requested review from
924060929,
CalvinKirs,
Gabriel39,
dataroaring,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde and
starocean999
as code owners
July 22, 2026 01:08
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
This multi-month refactor needs persistent state for progress, decisions, risks, and cross-session agent handoff. Establishes a file-based tracking system including dashboard, ADR decision log, deviation log, risk register, per-stage task files, per-connector tracking, and an agent collaboration playbook covering context budget / subagent usage / handoff norms. Closes 18 design decisions (D-001..D-018) and registers 14 risks (R-001..R-014). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…T27) (apache#63582) ## Summary Lands the P0 SPI baseline for the catalog-SPI migration (master plan §3.1 / RFC §2.1), with zero impact on the already-migrated JDBC + ES connectors. - **Batch 0** (commits 1-2): SPI types + fe-core bridges — `ConnectorMetaInvalidator`, `ConnectorTransaction`, `ConnectorMvccSnapshot`, `ExternalMetaCacheInvalidator`, `ConnectorMvccSnapshotAdapter`, `PluginDrivenTransactionManager` generalization. - **Batch 1** (commit 3): DDL + Partition SPI — `ConnectorCreateTableRequest` + 4 spec POJOs, 4 new defaults on `ConnectorTableOps`, 3 new fields on `ConnectorPartitionInfo`, fe-core converter, `PluginDrivenExternalCatalog.createTable` routing. - **Batch 2** (commit 4): Import-gate + unit tests — `tools/check-connector-imports.sh` wired through exec-maven-plugin; `FakeConnectorPlugin` covering every default fall-through; routing tests for the invalidator; converter tests for all 4 partition styles + 2 bucket flavors. ## Commits - `[feat](connector) add P0 batch 0 SPI baseline: MetaInvalidator / Transaction / MvccSnapshot` (T03-T08) - `[feat](connector) wire P0 batch 0 SPI into fe-core` (T09-T12) - `[feat](connector) add P0 batch 1 SPI: CreateTableRequest + listPartitions` (T13-T20) - `[feat](connector) add P0 batch 2 gate + unit tests` (T21-T23, T26-T27) ## Test plan - [x] `mvn -pl fe-connector/fe-connector-api,fe-connector/fe-connector-spi -am compile` — SPI modules compile - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` — fe-core compile - [x] `mvn -pl fe-core checkstyle:check` — 0 violations - [x] `mvn -pl fe-connector validate` — import gate runs and passes (baseline clean) - [x] `mvn -pl fe-core -am test -Dtest='FakeConnectorPluginTest,ExternalMetaCacheInvalidatorTest,CreateTableInfoToConnectorRequestConverterTest,ConnectorPluginManagerTest,ConnectorSessionImplTest'` — 39/39 green - [x] `mvn -pl fe-connector/fe-connector-jdbc,fe-connector/fe-connector-es -am compile` — downstream connectors compile unchanged - [ ] JDBC regression-test suite (T24) — to be exercised by this PR's CI pipeline - [ ] ES regression-test suite (T25) — to be exercised by this PR's CI pipeline ## Tracking Full plan, decisions, and risk log live under `plan-doc/` in the repo (introduced by 6315983, already on the base branch). Per-task status: `plan-doc/tasks/P0-spi-foundation.md`. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pache#63641) P1 batch A — close out scan-node SPI consolidation while keeping migration-period fallbacks in place. Three surgical changes route `PluginDrivenExternalTable` first in the nereids translator hot paths so already-migrated SPI connectors (JDBC, ES) take the SPI route, while the existing `instanceof XExternalTable` chains remain as fallbacks for connectors still pending migration (P3–P7). - **T3** — `PhysicalPlanTranslator.visitPhysicalFileScan`: move the existing `PluginDrivenExternalTable` branch from position 8 to position 1; the 7 connector-specific branches (HMS / Iceberg / Paimon / Trino / MaxCompute / LakeSoul / RemoteDoris) stay in place as migration-period fallbacks - **T4** — `PhysicalPlanTranslator.visitPhysicalHudiScan`: add a `PluginDrivenExternalTable` branch routed to `PluginDrivenScanNode.create(...)`, threading `tableSnapshot` + `scanParams` through `FileQueryScanNode` setters; `incrementalRelation` flagged as a P3 Hudi SPI extension TODO. The new branch is unreachable today (`PhysicalHudiScan` is only built for `HMSExternalTable + DLAType.HUDI`), so this is groundwork for P3 with zero current-day runtime impact - **T5** — `LogicalFileScan`: in `computeOutput()`, add a `PluginDrivenExternalTable` branch calling new helper `computePluginDrivenOutput()` — same shape as `computeIcebergOutput`, using `getFullSchema()` + virtualColumns; in `supportPruneNestedColumn()`, add an explicit `PluginDrivenExternalTable → false` branch. Both behaviorally equivalent for JDBC/ES today since they have no hidden cols and no virtualColumns P1 batch B (T1 — delete 13 legacy `Jdbc*Client` + `JdbcFieldSchema`) is deferred to P8 because the 3 fe-core callers — `PostgresResourceValidator`, `StreamingJobUtils`, `CdcStreamTableValuedFunction` — are live CDC streaming code that requires SPI extension for `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables`, which is out of P1 surgical scope. Background and tracking docs live in `plan-doc/` (Master Plan §3.2 P1, tasks/P1-scan-node-cleanup.md, decisions log). - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` → BUILD SUCCESS - [x] `mvn -pl fe-core checkstyle:check` → 0 violations - [x] JDBC + ES regression-test passing — baseline established in P0 / PR apache#63582 - [ ] PR CI green on this PR - [ ] Manual scan-node smoke for an SPI connector — JDBC `SELECT *` should fall into the new `PluginDrivenExternalTable` branch first 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…apache#64096) ### What problem does this PR solve? Related PR: apache#63582 (P0 — SPI baseline), apache#63641 (P1 — nereids plugin-driven routing) Problem Summary: This is **P2** of the catalog SPI migration and targets the `branch-catalog-spi` feature branch (continuing P0 apache#63582 and P1 apache#63641). It fully migrates `trino-connector` off the legacy in-tree `fe-core/datasource/trinoconnector/` implementation and onto the connector SPI module `fe-connector-trino`, making `trino-connector` the first connector to complete the SPI consumption playbook that later connectors will reuse as a template. All five batches land together so there is no intermediate state where a newly-created trino catalog cannot be serialized. **Batch A — complete the SPI surface (`fe-connector-trino` only, no fe-core changes)** - `TrinoConnectorProvider.validateProperties`: enforce the required `trino.connector.name` property at `CREATE CATALOG` time (ported from the legacy `checkProperties`). - `TrinoDorisConnector.preCreateValidation`: call `ensureInitialized()` so plugin loading + connector-factory resolution happen at catalog creation instead of being deferred to the first `SELECT`. - `TrinoConnectorDorisMetadata.applyFilter` / `applyProjection`: bridge Trino native filter/projection pushdown, reusing `TrinoPredicateConverter` to translate a Doris `ConnectorExpression` into a Trino `TupleDomain`. `remainingFilter` is conservatively returned as the original expression to match legacy behavior (conjuncts are not stripped; BE re-evaluates them). **Batch B — fe-core bridge for image compatibility** - `GsonUtils`: atomically replace the three legacy `registerSubtype` entries (`TrinoConnectorExternalCatalog` / `Database` / `Table`) with `registerCompatibleSubtype` redirects onto the `PluginDrivenExternal*` hierarchy. This must be atomic — `RuntimeTypeAdapterFactory` rejects duplicate labels, so keeping both bindings would throw at static init. Mirrors what ES/JDBC already did. - `PluginDrivenExternalCatalog.gsonPostProcess`: extract a `legacyLogTypeToCatalogType()` helper that maps `Type.TRINO_CONNECTOR` → `"trino-connector"`; the generic `name().toLowerCase()` would otherwise produce the wrong `"trino_connector"` (underscore) that `CatalogFactory` does not recognize. - `PluginDrivenExternalTable.getEngine()` / `getEngineTableTypeName()`: add `trino-connector` branches that preserve the legacy engine-name / table-type display across `SHOW TABLE STATUS` and `information_schema`. **Batch C — flip the switch** - Add `"trino-connector"` to `CatalogFactory.SPI_READY_TYPES` so catalog creation routes through the SPI path. **Batch D — remove legacy code** - Drop the `instanceof TrinoConnectorExternalTable` scan branch in `PhysicalPlanTranslator` (the `PluginDrivenExternalTable` SPI branch already handles it). - Drop `case "trino-connector"` in `CatalogFactory`. - Delete `fe-core/datasource/trinoconnector/` (10 files) and the now-dead legacy `TrinoConnectorPredicateTest`. - Route the `TRINO_CONNECTOR` db-build case in `ExternalCatalog` to `PluginDrivenExternalDatabase` (mirrors the migrated JDBC case). - **Retained for image compatibility**: the `InitCatalogLog.Type.TRINO_CONNECTOR` and `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` enums, the GsonUtils redirects, and the `MetastoreProperties` trino-connector entry. **Batch E — tests + tracking docs** - 29 JUnit 5 unit tests over the plugin-free converters: - `TrinoPredicateConverterTest` — `ConnectorExpression` pushdown trees → Trino `TupleDomain` (EQ / range / NE / IN / IS [NOT] NULL / AND / OR, Slice encoding), plus graceful degradation to `TupleDomain.all()` on null/unsupported input. - `TrinoTypeMappingTest` — Trino SPI type → Doris `ConnectorType` (scalars, decimal precision/scale, timestamp precision clamp, array/map/struct, unsupported-type failure). - `TrinoConnectorProviderTest` — `validateProperties` fast-fails when `trino.connector.name` is missing/empty. - No Trino plugin/cluster required; plugin-dependent paths remain covered by the existing `external_table_p0/p2` `trino_connector` regression suites. - Sync the migration tracking docs under `plan-doc/` (already carried on this feature branch since P0). **Net effect**: 28 files, +1025 / −2681 (~1656 LOC net removed). Old FE images holding legacy trino catalogs / databases / tables deserialize onto the `PluginDrivenExternal*` hierarchy through the GsonUtils string-name redirect, with engine-name display preserved. **Deferred (follow-ups, not in this PR)**: - `trino_connector_migration_compat` regression test (old-image deserialization) — requires a running cluster + Trino plugin + docker, unavailable in this dev environment; tracked as a CI/cluster follow-up. - The plugin-install documentation update lives in the `doris-website` repo and is handled separately. ### Release note None ### Check List (For Author) - Test - [x] Unit Test — 29 new tests in `fe-connector-trino` (predicate converter / type mapping / property validation). - [ ] Regression test — existing `trino_connector` suites cover plugin paths; the new old-image compat regression is deferred to a CI/cluster follow-up. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. Internal routing moves from the legacy fe-core path to the SPI path; image compatibility, engine-name display, and pushdown semantics all mirror the legacy behavior. All batches land together, so there is no serialization-gap window. - Does this need documentation? - [x] Yes. The trino-connector plugin-install doc update is a follow-up in the `doris-website` repo. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
…tch design (hybrid, T02-T08) (apache#64143) ## Proposed changes testing with apache#64146 P3 of the catalog-SPI migration (base: `branch-catalog-spi`). Migrates the **hudi** connector following the **hybrid** strategy (D-019): harden the dormant HMS-over-SPI hudi connector to correctness parity, build a test baseline, and write the per-table dispatch design — **all behind the closed gate** (`SPI_READY_TYPES` unchanged). >⚠️ **No user-visible behavior change.** The SPI hudi path stays dormant (gate closed); hudi queries continue to use the legacy `HMSExternalTable.dlaType=HUDI` path. This PR removes correctness blockers ahead of the live cutover (deferred to P7 / batch E). ### What's included **Correctness fixes (hardening dormant code, behind gate):** - **T02** — fix hudi JNI `column_types` double bug: emit full Hive type strings (was Doris bare type names, losing precision/scale/subtypes) and send `column_names`/`column_types`/`delta_logs` as typed lists end-to-end (was comma join/split, which shattered `decimal(10,2)` / `struct<...>`). Matches the BE `hudi_jni_reader.cpp` contract (names `,` / types `#` / delta `,`). - **T04** — fail loud on time-travel / incremental read in the SPI `visitPhysicalHudiScan` branch (was silently returning the latest snapshot / silently full-scanning). - **T05** — real EQ/IN partition pruning in `HudiConnectorMetadata.applyFilter` (was a placeholder that ignored predicates and unconditionally switched the partition source from Hudi-metadata to HMS); faithfully mirrors `HiveConnectorMetadata.applyFilter`. - **T07** — column-name casing fix in `avroSchemaToColumns` (top-level lowercase, mirroring legacy `HMSExternalTable`). **Test baseline (all three connector modules started P3 with 0 tests):** - `fe-connector-hudi` (33): type-mapping / schema-parity (COW/MOR golden) / table-type / partition-pruning / scan-range. - `fe-connector-hms` (12): shared Hive-type-string parser tests. - `fe-connector-hive` (14): file-format / partition-pruning (mirrors T05). - COW/MOR schema is **type-agnostic** (golden parity vs legacy `initHudiSchema`); table type only affects scan planning. **Decisions / design (code-grounded, design-only):** - **T03** — defer `schema_id`/`history_schema_info` field-id evolution to batch E (DV-006; not a model-agnostic SPI fix). - **T06** — keep MVCC/snapshot SPI defaults (opt-out) + document (DV-007). - **T08** — `tableFormatType` dispatch design memo + **D-020**: single `hms` catalog per-table routing via a new backward-compatible `ConnectorMetadata.getScanPlanProvider(handle)` (per-table provider seam); refines D-005. The keystone gap is split into M1 (identity consumption, fe-core reads `tableFormatType` as an opaque string) and M2 (scan routing). ### Deferred to batch E / P7 (not in this PR) Gate flip (`SPI_READY_TYPES += hms/hudi`), fe-core `tableFormatType` consumption (M1+M2 implementation), live cutover, delete legacy `datasource/hudi/`, full incremental/time-travel/MVCC, Iceberg-on-hms via SPI (needs P6 `IcebergScanPlanProvider`), cluster/runtime validation. ### Verification Per task tracking, each code batch landed with: per-module compile + checkstyle 0 (incl. test sources) + connector import-gate pass + new unit tests green. The two most recent commits are docs-only (`plan-doc/`); the code is unchanged since the last green batch. Gate stays closed → the dormant SPI path is unreachable at runtime → zero live-path risk. CI re-verifies. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…core + make fe-core odps-free (T07-T09) (apache#64300) Follow-up to apache#64253 (the MaxCompute catalog-SPI cutover). After the cutover a `max_compute` catalog deserializes to `PluginDrivenExternalCatalog` and no legacy `MaxComputeExternal*` object is ever instantiated, so the legacy MaxCompute subsystem in fe-core is dead code. This removes it and makes fe-core's dependency tree fully odps-free. **1. Remove legacy subsystem** (`7a4db351100`) - Delete 20 fe-core files: `datasource/maxcompute/*` (incl. `MCTransaction`, `MaxComputeScanNode`/`Split`), the MaxCompute sink/insert/txn plumbing, and 2 legacy-only tests. - Clean ~21 reverse-reference sites (imports + dead `instanceof`/visitor/rule branches), keeping every `PluginDriven`/connector sibling branch and the image/replay keep-set (GsonUtils compat strings; `TableType`/`TransactionType`/`TableFormatType`/`InitCatalogLog.Type` `MAX_COMPUTE` enums; block-id thrift). - Rewire 3 tests; e.g. `FrontendServiceImplTest`'s block-id RPC test now mocks the generic `Transaction` SPI, since `getMaxComputeBlockIdRange` reads the PluginDriven connector transaction. **2. Make fe-core odps-free** (`409300a75b8`) - Drop the two odps deps from `fe-core/pom.xml`. - Move `MCUtils` from fe-common into `be-java-extensions/max-compute-connector` (its only consumer after the removal); keep `MCProperties` (odps-free constants) in fe-common. - Drop `odps-sdk-core` from fe-common — it was also leaking netty/protobuf transitively to fe-common's own `DorisHttpException`/`GsonUtilsBase`, so declare `netty-all` + `protobuf-java` directly (proper dependency hygiene). **3. Doc-sync** (`f8c305765e8`) — plan-doc PROGRESS/HANDOFF/deviations/design tracking notes. - `mvn -pl :fe-core -am test-compile` (main+test) passes; checkstyle 0 violations; connector import-gate passes. - `grep -rn com.aliyun.odps fe/fe-core/src` → empty. - `mvn -pl :fe-core dependency:tree | grep odps` → empty (no odps, direct or transitive). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he#64446) (apache#64446) Migrates the legacy in-tree Paimon catalog (fe-core `datasource/paimon`, `metacache/paimon`, `property/metastore/*Paimon*` and their reverse references) onto the catalog SPI as a self-contained `fe-connector-paimon` plugin, following the maxcompute full-adopter + cutover template. `paimon` is added to `SPI_READY_TYPES`, so a Paimon catalog now deserializes to `PluginDrivenExternalCatalog` and all five functional areas — normal-table read, system-table read, DDL, MTMV/MVCC/time-travel (procedures are a doc-only no-op: fe-core has none) — flow through the generic `PluginDriven*` bridge with no Paimon-specific branch in fe-core. Paimon is the first lakehouse full-adopter to exercise the MVCC / sys-table / MTMV SPI surfaces, which become the reusable template for the upcoming iceberg/hudi migrations, so the cutover is held to byte-parity with the legacy path. The legacy `datasource/paimon/*` classes are left in place (now dead on the cutover path) and removed in a follow-up, mirroring the maxcompute cutover -> P4 removal. DefaultConnectorContext) Source-agnostic seams so any connector can express these capabilities; every new method is a default no-op/identity, so hive/iceberg/jdbc/maxcompute/ trino/es are byte-for-byte unaffected. - MVCC/time-travel: `resolveTimeTravel(session, handle, ConnectorTimeTravelSpec)` + `applySnapshot(...)` (threads a pinned snapshot into the handle before planScan); new immutable `ConnectorTimeTravelSpec` (SNAPSHOT_ID / TIMESTAMP / TAG / BRANCH / INCREMENTAL) that fe-core extracts from SQL; `ConnectorMvccSnapshot` carries `schemaId` for schema-evolution-aware time travel. - System tables: `listSupportedSysTables` / `getSysTableHandle`; fe-core composes the `{base}${sys}` reference name. - Scan: proportional split-weight (`getSelfSplitWeight`/`getTargetSplitSize`), COUNT(*) pushdown (`getPushDownRowCount`, 7-arg `planScan`), native-read accounting (`isNativeReadRange`), `getDeleteFiles` (VERBOSE merge-on-read), `ignorePartitionPruneShortCircuit` (predicate-driven connectors keep genuine-null partitions). - Metadata: `ConnectorPartitionInfo.fileCount` + `SUPPORTS_PARTITION_STATS` (rich SHOW PARTITIONS); `ConnectorColumn.withTimeZone` (WITH_TIMEZONE DESC marker, independent of the timestamp_tz mapping flag); connector cache control (`invalidateTable`/`invalidateAll`, `schemaCacheTtlSecondOverride`). - `ConnectorContext`: six engine-side hooks the connector must not implement itself (`loadHiveConfResources`, `vendStorageCredentials`, `getBackendStorageProperties`, `normalizeStorageUri` x2, `getStorageProperties`), bound in `DefaultConnectorContext` over fe-core's single-source-of-truth credential/URI/HiveConf utilities (no re-ported logic that could drift). The metastore-side counterpart of the fe-filesystem StorageProperties SPI: a Paimon catalog resolves its backend (hive/dlf/rest/jdbc/filesystem) by ServiceLoader dispatch on `paimon.catalog.type` instead of a fe-core switch. - `fe-connector-metastore-api`: hadoop/SDK-free `MetaStoreProperties` contracts (HMS/DLF/REST/JDBC/FileSystem) carrying only Map/scalar facts. - `fe-connector-metastore-spi`: `MetaStoreProvider<P>` + first-hit `MetaStoreProviders` dispatcher + five impls, registered via META-INF/services. `HmsMetaStorePropertiesImpl` reproduces the legacy `buildHmsHiveConf` key set and parity-critical ordering (storage overlay before the kerberos block; username/sasl last). Adding a backend = one provider class + one services line (no central enum/switch). - ServiceLoader is loaded with the SPI interface's own (plugin) classloader, NOT the thread-context CL: at CREATE CATALOG the static init first fires on an FE worker thread whose TCCL is the FE app loader, so a 1-arg load would cache an empty provider list process-wide. - fe-core bridges add `initExecutionAuthenticator(List<StorageProperties>)` so HDFS-Kerberos doAs is wired on the plugin path (legacy `initializeCatalog` is dead). Full read + DDL + partitions + statistics + system tables + MVCC/time-travel across all five metastore flavors, planning native (ORC/Parquet) vs JNI reads. - `PaimonCatalogFactory` (pure flavor switch: Options + Hadoop Configuration + HiveConf, classloader-pinned for child-first loading), `PaimonCatalogOps` (injection seam over the remote Catalog for offline-testable metadata), `PaimonConnectorMetadata` (~9 -> 28 methods: schema latest/at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor). - `PaimonScanPlanProvider` (~6 -> 30 methods): predicate/projection pushdown, native-vs-JNI routing, large-file sub-splitting with deletion vectors, COUNT(*) collapse, schema-evolution field-id dictionary, `$ro` unwrap to base FileStoreTable. - New: `PaimonSchemaBuilder` (CREATE TABLE), `PaimonIncrementalScanParams` (@incr), `PaimonTableResolver` (sys/branch-aware handle -> Table), `PaimonLatestSnapshotCache` + `PaimonSchemaAtMemo` (per-catalog caches on the long-lived connector). - `PaimonTypeMapping` gains the write direction (Doris -> Paimon) for CREATE TABLE. - NEW `PluginDrivenMvccExternalTable` (generic MvccTable + MTMV host: the runtime class for paimon tables), `PluginDrivenMvccSnapshot`, `PluginDrivenSysExternalTable` + `systable/PluginDrivenSysTable`. Behavior is selected by `ConnectorCapability` (SUPPORTS_MVCC_SNAPSHOT, SUPPORTS_PARTITION_STATS) — no paimon branch in fe-core. - `PluginDrivenScanNode`: MVCC pin threaded at every handle-consumption site, sys-table time-travel guard, COUNT pushdown, connector-agnostic EXPLAIN re-emission. - Cutover: `CatalogFactory` adds `paimon` to `SPI_READY_TYPES` and drops the built-in case; `GsonUtils` removes the 7 built-in `Paimon*` subtype registrations and adds replay compat so persisted `PaimonExternalCatalog` (all 5 flavors) / `...Database` -> `PluginDriven*` and `PaimonExternalTable` -> `PluginDrivenMvccExternalTable` on deserialization (existing clusters upgrade without losing catalogs). - Nereids/DDL: `PhysicalPlanTranslator` drops the `PaimonScanNode` branch; `UserAuthentication`, `CreateTableInfo`, `ShowPartitionsCommand` generalized; `Env.getDdlStmt` renders LOCATION/PROPERTIES for SHOW CREATE (paimon engine, no credential leak). - fe-filesystem: NEW `HdfsFileSystemProperties` / `HdfsConfigFileLoader` (typed HDFS; defaults-laden BE map vs defaults-free FE Hadoop-config map so it cannot clobber a co-bound object store's fs.s3a.* during a multi-backend merge); S3 MinIO support + legacy tuning defaults; OSS/OBS/COS ANONYMOUS provider-type; new `FileSystemFactory.bindAllStorageProperties` / `FileSystemPluginManager.bindAll`. - NEW `fe-kerberos` leaf module: dependency-free `AuthType` / `KerberosAuthSpec` facts shared across modules without pulling in Hadoop. - Packaging: NEW `fe-connector-paimon-hive-shade` relocates `org.apache.thrift` for HMS 2.3.7; the paimon plugin-zip excludes fe-thrift/libthrift (provided parent-first) to avoid a split `TBase` across loaders; self-contained bundle (hadoop-aws, AWS-SDK, hadoop-hdfs-client). `paimon.version` (1.3.1) pinned as the single property across FE connector / BE paimon-scanner / preload-extensions for FE<->BE Table/Split serde. - BE `PaimonJniScanner.getPredicates()` null-predicate backstop (no-filter scan; pairs with the FE producer now always emitting an empty predicate list). Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Storage/credentials: canonical `s3.*`/`oss.*` keys -> `fs.s3a.*`/`fs.oss.*` (STORAGE-CREDS); canonical `AWS_*` BE creds (STATIC-CREDS-BE); per-table vended DLF/STS token (REST-VENDED); `oss/cos/obs/s3a` -> `s3://` for data + deletion-vector paths (URI-NORMALIZE); external `hive.conf.resources` hive-site.xml (HMS-CONFRES); real doAs on filesystem/jdbc + wrapped HMS read RPCs (KERBEROS-DOAS). - Reads/types: paimon columns forced nullable (READ-NOTNULL); `VARCHAR(65533)` not widened to STRING (VARCHAR-BOUNDARY); dotted `enable.mapping.varbinary` / `.timestamp_tz` keys (MAPPING-FLAG-KEYS); per-type partition-value rendering (NATIVE-PARTVAL); genuine `\N` not coerced to NULL (PARTITION-NULL-SENTINEL); field-id history dict for rename-safe native reads (SCHEMA-EVOLUTION); paimon-native split encode under `enable_paimon_cpp_reader` (CPP-READER). - Planning/perf: restore the `force_jni_scanner` escape hatch (FORCE-JNI-SCANNER); intra-file native sub-splitting (NATIVE-SUBSPLIT); precomputed COUNT(*) row count (COUNT-PUSHDOWN); table row count for CBO / SHOW TABLE STATUS (TABLE-STATS). - Time travel / DDL: `FOR TIME AS OF` under CST/PST session zones (TZ-ALIAS); resolved + validated `driver_url` for jdbc catalogs (JDBC-DRIVER-URL); reject a local-only name conflict on CREATE TABLE (CREATE-TABLE-LOCAL-CONFLICT). plan-doc P5 design / task-list / HANDOFF / deviation tracking notes. - ~70 new FE unit suites: paimon connector (offline recording fakes: `RecordingPaimonCatalogOps` / `FakePaimonTable`), fe-core PluginDriven* MVCC / sys-table / scan-node / split-weight, metastore-spi dispatch + per-backend properties, fe-filesystem HDFS/MinIO/anonymous, fe-kerberos. - checkstyle 0 violations; connector import-gate passes; FE main+test compile clean. - End-to-end paimon behavior is docker-gated (`enablePaimonTest=true`); the five functional areas are covered by the `external_table_p0/paimon` regression suites, green on the latest CI run for this branch.
… make fe-core paimon-SDK-free (T29) (apache#64653) Follow-up to apache#64446 (the Paimon catalog-SPI cutover). After the cutover a `paimon` catalog deserializes to `PluginDrivenExternalCatalog` and its tables to `PluginDrivenMvccExternalTable`, so no legacy `Paimon*` catalog / table / scan-node object is ever instantiated — the entire legacy Paimon subsystem in fe-core is dead code. This PR removes it and makes fe-core's dependency tree fully paimon-SDK-free (zero `org.apache.paimon` imports, the 5 paimon maven deps dropped). Mirrors the P4 maxcompute cutover → legacy-removal (apache#64256). **1. Remove legacy subsystem + sever reverse-refs** (`7632a074e4b`) - Delete the dead fe-core paimon files: `datasource/paimon/*` (all 5 flavor catalogs, `PaimonExternalTable`, `PaimonMetadataOps`, `PaimonUtil`, `source/PaimonScanNode` / `PaimonSplit` / `PaimonPredicateConverter` / `PaimonSource` / `PaimonValueConverter`, `profile/*`), `datasource/metacache/paimon/*`, `systable/PaimonSysTable`, and the legacy-only tests. - Sever the live reverse-references to the deleted classes, keeping every `PluginDriven` / connector sibling branch and the image/replay keep-set: drop the `PAIMON` db-creation switch arm in `ExternalCatalog`; the paimon engine routing in `ExternalMetaCacheMgr` / `ExternalMetaCacheRouteResolver`; the dead `PAIMON_EXTERNAL_TABLE` branch in `Env.getDdlStmt` (keep the LIVE `PLUGIN_EXTERNAL_TABLE` SHOW CREATE LOCATION/PROPERTIES rendering); the `PaimonSysExternalTable` branch in `UserAuthentication`; the legacy clauses + `handleShowPaimonTablePartitions()` in `ShowPartitionsCommand` (keep `hasPartitionStatsCapability()` and the `TableType.PAIMON_EXTERNAL_TABLE` enum). Landed as one compiling unit (mirrors the P4 apache#64300 precedent: `PaimonUtils` still called the removed `ExternalMetaCacheMgr.paimon()`). - Inline the `getPaimonCatalogType()` string literals (`hms`/`filesystem`/`dlf`/`rest`/ `jdbc`) to decouple the still-consumed thin metastore-property classes from the deleted `PaimonExternalCatalog`. **2. Make fe-core paimon-SDK-free** (`32de7d16702`) - Strip the dead paimon-SDK catalog-building cluster from `AbstractPaimonProperties` + the 5 flavors (`initializeCatalog` / `buildCatalogOptions` / `appendCatalogOptions` / `getMetastoreType` / `normalizeS3Config` / … — 0 live callers; the plugin path builds catalogs connector-side). Keep every LIVE SDK-free duty: `warehouse` `@ConnectorProperty`, the Kerberos `executionAuthenticator` doAs wiring (read by `PluginDrivenExternalCatalog`), property normalization/validation, `getPaimonCatalogType`, `Type.PAIMON`. - Vended credentials: delete `PaimonVendedCredentialsProvider` + its test + the `VendedCredentialsFactory` `case PAIMON`. Its SDK methods were dead — reachable only via the iceberg-only `getStoragePropertiesMapWithVendedCredentials`; the real paimon vended path is the connector's `PaimonScanPlanProvider.extractVendedToken` (moved off fe-core at cutover). Relocate its one LIVE duty (the REST "skip static storage map" gate) to a new SDK-free `MetastoreProperties.isVendedCredentialsEnabled()` (base=false, `PaimonRestMetaStoreProperties`=true); `CatalogProperty` routes iceberg through its provider (byte-identical) and everything else through the metastore-props method. - pom: drop `paimon-core` / `-common` / `-format` / `-s3` / `-jindo` from `fe-core/pom.xml`. `fe/pom.xml` `paimon.version` is kept (fe-connector-paimon + BE paimon-scanner still consume it). **3. Doc-sync** (`58160cbcb56`, `895e214b2bd`) — plan-doc HANDOFF / PROGRESS / connectors / P5-T29 legacy-removal design tracking notes. - `mvn -pl :fe-core -am test-compile` (main + test) passes; checkstyle 0 violations; connector import-gate (`tools/check-connector-imports.sh`) passes. - `grep -rn org.apache.paimon fe/fe-core/src` → empty (main and test). - `mvn -pl :fe-core dependency:tree -Dincludes=org.apache.paimon` → empty (no paimon dep, direct or transitive). - End-to-end paimon regression (`external_table_p0/paimon`, docker `enablePaimonTest=true`) passed on this branch.
…kerberos (apache#64655) **P3b: consolidate the drifted Kerberos/Hadoop authentication implementations into the new top-level neutral leaf module `fe-kerberos`** as the single source of truth. Done as 3 commits: 1. **trino → JDK** (`4a740e1`) — replace the only external dependency in the auth path, trino's `KerberosTicketUtils`, with a JDK-only (`javax.security.auth.kerberos`) byte-for-byte equivalent, so the kerberos path is trino-free. 2. **relocate** (`8898e15`) — move the 13 `fe-common` `security.authentication.*` classes to `org.apache.doris.kerberos.*` in `fe-kerberos`; retarget all consumer imports (fe-core + 3 be-java-extensions scanners); merge the duplicate `AuthType`. 3. **unify interface** (`5e3e896`) — merge the two competing `HadoopAuthenticator` interfaces (fe-common's `PrivilegedExceptionAction` variant vs fe-filesystem-spi's `IOCallable` variant) into the single fe-kerberos one, and delete fe-filesystem-hdfs's own `KerberosHadoopAuthenticator`/`SimpleHadoopAuthenticator` copies (which had drifted from the canonical impls). `DFSFileSystem` now routes through the shared authenticators. `fe-kerberos` remains a top-level neutral leaf (no dependency cycle). HDFS filesystem access now uses the same authenticators as the HMS path (restoring parity). Two intentional behavior changes in fe-filesystem-hdfs: simple / no-`hadoop.username` now runs as remote user `hadoop` (was: FE process user, direct); kerberos uses the shared `LoginContext` + 80%-lifetime refresh. fe-filesystem-hdfs 79/0/0 (+fe-kerberos/spi), checkstyle 0, connector import-gate clean, whole-repo grep for the removed symbols = 0. >⚠️ docker kerberos e2e (HDFS kerberized + HMS) NOT yet run — the real gate; UGI login can't be exercised in unit tests.
…move legacy fe-core subsystem (apache#64688) Migrates the legacy in-tree Iceberg catalog (fe-core `datasource/iceberg`, its `property/metastore/*Iceberg*` clusters, and every reverse `instanceof IcebergExternal*` / `case ICEBERG` coupling scattered across nereids / planner / alter) onto the catalog SPI as a self-contained `fe-connector-iceberg` plugin, following the paimon full-adopter + cutover template. `iceberg` is added to `SPI_READY_TYPES`, so an Iceberg catalog now deserializes to `PluginDrivenExternalCatalog` and every functional area — read, write, row-level DML, `ALTER TABLE EXECUTE` procedures, system tables, DDL and MVCC/time-travel — flows through the generic `PluginDriven*` bridge with no Iceberg-specific branch in fe-core. (Iceberg tables surfaced *through* a `hive`/HMS catalog stay on the hive path until P7; this PR covers the dedicated `type=iceberg` catalog.) Iceberg is the widest lakehouse adopter migrated so far — 7 catalog flavors, merge-on-read row-level DML, 9 stored procedures, v3 row-lineage / deletion vectors — so the cutover is held to behavior-parity with the legacy path. Unlike the paimon migration (a cutover PR + a separate removal PR), this PR does the build-out, the live cutover **and** the full legacy-subsystem removal on one branch: fe-core `datasource/iceberg` and its property clusters are deleted, not just left dead. Part of the catalog-SPI migration tracked in apache#65185 (phase **P6**). Full read + write + row-level DML + procedures + system tables + DDL + MVCC/time-travel, planning native (Parquet/ORC) vs JNI reads. Key seams: - `IcebergCatalogFactory` (pure flavor switch, classloader-pinned for child-first loading), `IcebergCatalogOps` (metadata / DDL ops), `IcebergConnectorMetadata` (schema latest / at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor), `IcebergConnectorProvider` (ServiceLoader entry). - `IcebergScanPlanProvider` + `IcebergScanRange`: predicate/projection pushdown (`IcebergPredicateConverter` → Iceberg `Expression`), FileScanTask byte-offset sub-splitting, COUNT(\*) collapse, merge-on-read delete attachment (position delete / Puffin DV / equality delete), `IcebergColumnHandle` field-id column pruning, schema-evolution-safe reads (`IcebergSchemaUtils` field-id dictionary), synthetic rowid + v3 row-lineage metadata columns. - Write path: `IcebergWritePlanProvider` / `IcebergWriteContext` / `IcebergWriterHelper` / `IcebergConnectorTransaction` — INSERT / INSERT OVERWRITE (dynamic + static partition), branch-targeted writes, WRITE ORDERED BY, per-spec distribution, optimistic-concurrency conflict detection, one SDK transaction per statement, BE commit-fragment → DataFile/DeleteFile conversion. - Row-level DML: DELETE / UPDATE / MERGE INTO over position-delete (v2) and deletion-vector (v3) merge-on-read, baseSnapshot-anchored read. - `action/`: the 9 `ALTER TABLE EXECUTE` procedures (rollback_to_snapshot, rollback_to_timestamp, set_current_snapshot, cherrypick_snapshot, fast_forward, expire_snapshots, publish_changes, rewrite_manifests, rewrite_data_files) dispatched by `IcebergExecuteActionFactory`; `rewrite/` hosts the distributed bin-pack `rewrite_data_files` engine. - `IcebergSchemaBuilder` (CREATE TABLE), `IcebergTypeMapping` (both directions), `IcebergColumnChange` / `IcebergComplexTypeDiff` (schema + partition evolution), `IcebergLatestSnapshotCache` / `IcebergManifestCache` (per-catalog caches on the long-lived connector), `dlf/` (Aliyun DLF client). (`fe-connector-metastore-iceberg`) An Iceberg catalog resolves its backend by `MetaStoreProvider` ServiceLoader dispatch on `iceberg.catalog.type` (registered via `META-INF/services`), instead of a fe-core switch — **7 flavors**: REST, Hive Metastore, AWS Glue, Hadoop / filesystem, JDBC, Aliyun DLF, AWS S3Tables. Adding a backend = one provider class + one services line. The connector self-builds the HMS-metastore Kerberos authenticator, so metastore auth is owned entirely by the plugin. - `CatalogFactory` adds `iceberg` to `SPI_READY_TYPES` and drops the built-in case. - `GsonUtils` replay-compat: persisted `IcebergExternalCatalog` → `PluginDrivenExternalCatalog`, `IcebergExternalDatabase` → `PluginDrivenExternalDatabase`, `IcebergExternalTable` → `PluginDrivenMvccExternalTable` on deserialization, so existing clusters upgrade without losing catalogs (metadata backward-compat invariant). - Nereids/DDL/alter generalized off the concrete Iceberg types; SHOW CREATE renders LOCATION / PARTITION BY / ORDER BY / PROPERTIES via the generic path (iceberg engine name preserved, no credential leak). Per the 2026-07 architecture decision (fe-core holds **no** property parsing for migrated connectors; storage parsing → `fe-filesystem`, metastore parsing → `fe-connector`, both plugin-side; the plugin assembles → BE thrift → hands back to fe-core), iceberg's property/credential/auth resolution moves entirely into the connector: - Storage credentials bound to `fe-filesystem` on both scan and write paths (one source), retiring the second fe-core `StorageProperties.createAll` parse. - Vended temporary credentials, `warehouse` → `fs.defaultFS` derivation (hadoop flavor only, via the neutral `Connector.deriveStorageProperties` SPI), and Kerberos doAs are all connector-owned; the fe-core pre-execution authenticator is a no-op for plugin catalogs. - fe-core's shared static property/metastore helpers are kept alive only for the connectors still on the legacy path (Hive / Hudi / LakeSoul / HMS-iceberg). Net **−21.7k** lines from fe-core. Removes the 4 dead entity classes (`IcebergExternalTable` / `IcebergExternalDatabase` / `IcebergSysExternalTable` / `IcebergExternalCatalog`), the fe-core Iceberg `MetastoreProperties` cluster + its credential helpers, the 5 fe-core Iceberg connectivity probes and the coordinator branches, and un-registers the Iceberg property factories (fail-loud on any stray call). The now-dead paimon `MetastoreProperties` cluster left behind by P5 is swept in the same pass. `GsonUtils` retains only the three string compat labels for old-image upgrade. Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Classloader: TCCL pinned to the plugin classloader at all three loci where fe-core crosses into bundled Iceberg (scan call thread, write/DDL/commit engine thread via `TcclPinningConnectorContext`, and Iceberg's own manifest-writer worker pool), preventing a duplicate-copy `ClassCastException`. - Nested schema dictionary field names lowercased so a mixed-case nested struct (e.g. Iceberg `DROP_AND_ADD`) can't crash BE with `std::out_of_range`. - Session time zone: lazy `ZoneId` resolution tolerant of Doris CST/PST-style aliases in `FOR TIME AS OF` and datetime partition predicates. - Storage / URI normalization (`oss/cos/obs/s3a` → `s3`), SHOW CREATE secret redaction, and `ALTER TABLE EXECUTE ... WHERE` rejection kept at legacy wording. - ~68 FE unit test classes: iceberg connector (offline recording fakes for the remote Catalog / Table), metastore-iceberg per-flavor dispatch + properties, and the fe-core `PluginDriven*` MVCC / sys-table / scan-node / GSON-replay surfaces; many carry mutation checks. - checkstyle 0 violations; connector import-gate passes (no `fe-connector-iceberg → fe-core` import); FE main + test compile clean. - End-to-end iceberg behavior is docker-gated (`enableIcebergTest=true`); read, write, row-level DML, procedures, system tables, DDL and time-travel are covered by the `external_table_p0/iceberg` (+ `external_table_p2/iceberg`) regression suites. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etaspace leak JdbcConnectorClient cached JDBC-driver classloaders in a ref-counted map that evicted the entry when refCount reached 0. A JDBC driver self-registers into the static java.sql.DriverManager on class load, and close() never deregisters it, so an evicted classloader stays pinned by DriverManager for the life of the process. Serial CREATE -> DROP -> CREATE churn of the same driver URL therefore rebuilt a fresh, separately-pinned classloader every cycle, leaking one driver's worth of Metaspace per cycle. Live FE evidence (external regression build 986696 OOM): only 4 jdbc catalogs yet 38 FactoryURLClassLoader instances survived a full GC (jmap -histo:live), tracking the DriverInfo count -> ~34 leaked, DriverManager-pinned driver classloaders. FE Metaspace grew 165MB -> 1565MB over the single-concurrency 5.5h run (multi-concurrency did not OOM: overlapping same-URL catalogs keep refCount > 0 so the loader was reused instead of evicted-and-rebuilt). Fix: revert to a keep-alive Map<URL, ClassLoader> (one loader per distinct driver URL, shared across catalogs, never evicted), mirroring the pre-SPI fe-core JdbcClient and the pattern IcebergConnector / PaimonConnector already use. Bounded by the number of distinct driver jars. Adds a regression test asserting repeated create/recreate cycles for one URL add exactly one cached classloader. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XKf5EzHW75qzgrXnkoBZcF
…TableTest after rebase Rebase-integration fix, belongs with P4 (apache#64300 "remove legacy maxcompute subsystem"); squash into it when finalizing that PR. Upstream apache#64937 added MaxComputeExternalTableTest as a unit test for the legacy MaxComputeExternalTable.parsePartitionValues. P4 deletes that class, but the new test file rode into the rebase base with no modify/delete conflict and then dangled: fe-core test-compile failed with "cannot find symbol: MaxComputeExternalTable". No coverage lost: the new connector's MaxComputeConnectorMetadata.listPartitionValues already resolves partition values by name via ODPS PartitionSpec.get(column) (structurally immune to the out-of-order/invalid-spec bug apache#64937 fixed in the legacy path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…es into SPI connectors After rebasing branch-catalog-spi onto upstream master (4dfcb4f), an audit of the 3 upstream commits touching fe-core datasource/ found behaviour that was dropped when P5/P6 removed the legacy fe-core paimon/iceberg subsystems. Port the bounded correctness gaps into the new fe-connector SPI code (main + tests): - apache#65332 (paimon JNI IOManager): PaimonScanPlanProvider.getBackendPaimonOptions returned emptyMap() for non-jdbc catalogs, so the three JNI IOManager options (paimon.doris.enable_jni_io_manager / .tmp_dir / .impl_class) never reached BE and Paimon primary-key merge reads could not spill (OOM risk). Collect the three options (strip the "paimon." prefix) before the metastore-type check and forward them for all catalog flavours; jdbc-only driver logic stays in the jdbc branch. - apache#65094 tier1 (build-time correctness): IcebergSchemaBuilder buildPartitionSpec / buildSortOrder and PaimonSchemaBuilder primary/partition keys used bare column names, so a partition/sort/key referenced with a different case than the (now case-preserving) schema column failed the engine's case-sensitive lookup. Add case-insensitive resolution back to the canonical column name (iceberg via Schema.caseInsensitiveFindField; paimon via a lower->canonical map), mirroring upstream getIcebergColumnName / getPaimonColumnNames. - apache#65094 tier2 (consistency, worse than upstream): PaimonConnectorMetadata emitted "partition_columns" with case-preserving keys while column names are lowercased, so PluginDrivenExternalTable's case-sensitive byName lookup silently dropped mixed-case paimon partition columns (table treated as non-partitioned). Lower-case partition_columns with the same bare toLowerCase() the columns use. Verified: mvn package -pl :fe-connector-paimon -am (95 paimon tests) + iceberg 23 tests, 0 failures. Read-path column-name case preservation (tier3) and the apache#63068 OIDC session-credential port are tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…column-name case on the read path Align the iceberg + paimon catalog-SPI connectors with upstream apache#65094: external table top-level column names now keep their remote case on the read path instead of being lowercased. FE-only (BE untouched): the byte-match invariant "Doris scan slot name == TSchema top-level TField name" is preserved by flipping BOTH the scan slot names AND the field-id-dict / current-TSchema top-level names to case-preserving together, so BE (which matches top-level by field-id) needs no change. Nested struct children stay as-is (iceberg lowercase for BE StructNode.children.at; paimon paimon-cased). Reverts tier2's lowercasing of paimon partition_columns (bce28b2) now that the paimon columns are themselves case-preserving. iceberg (IcebergConnectorMetadata): partition_columns / getColumnHandles / parseSchema. iceberg (IcebergPartitionUtils): getIdentityPartitionColumns (path_partition_keys), getIdentityPartitionInfoMap (scan partition-value map) and generateRawPartition -- the listPartitions value-map key that PluginDrivenExternalTable.getNameToPartitionItems looks up by the case-preserved partition_columns remote name; omitting it would silently drop a mixed-case partition column's value in MTMV / partition pruning. paimon (PaimonConnectorMetadata): partition_columns / getColumnHandles / mapFields. paimon (PaimonScanPlanProvider): path_partition_keys / getPartitionInfoMap and the current (-1) TSchema entry (buildSchemaInfo lowercaseTopLevelNames -> false at the sole current-schema call site; the historical entries and nested names are unchanged). Tests: flip the connector tests that pinned mixed-case -> lowercase output to assert case preservation; add an IcebergSchemaUtils mixed-case byte-match case and an IcebergPartitionUtils.listPartitions regression pinning the case-preserved partition-value-map key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…l DTO + user-session capability Re-migrate the Iceberg REST OIDC per-user session-credential feature (upstream apache#63068 / e545f1a, dropped by the 2026-07-09 P6 rebase) onto the catalog SPI. This is the SPI layer (fe-connector-api): - ConnectorDelegatedCredential: neutral, immutable DTO carrying a user's per-connection OIDC/JWT/SAML token across the fe-core -> connector boundary so the connector never imports a fe-core type. The token is connection-scoped and in-memory only; toString() redacts it (never edit-logged / persisted / SHOWn). - ConnectorSession.getDelegatedCredential() (default empty) + getSessionId() (default = getQueryId): a connector reads the credential and the stable AuthSession key (preserved across FE observer->master forwarding) off the session. - ConnectorCapability.SUPPORTS_USER_SESSION: gates FE credential injection (least-privilege -- a connector that never consumes the token never receives it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
… deps dropped Record the iceberg-connector test relocation (user chose migrate over delete) and the fe-core iceberg dependency-cluster removal, plus the deferred aws-json-protocol / avro / parquet-avro follow-up and why it needs a -Dverbose dependency:tree (nearest- wins pruning hides transitive supply while the direct declarations remain). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…> parquet-hadoop+parquet-column
The Batch-3 tail-cleanups coupled to the iceberg removal, each verified against the
resolved -am dependency:tree (direct declarations removed first so nearest-wins no
longer masks the transitive supply):
- aws-json-protocol: 0 occurrences anywhere in fe-core's resolved tree once the
iceberg cluster (its only consumers: glue / s3tables / iceberg-aws) is gone. The
retained AWS SDK v2 clients use other protocols (sts=query, s3=xml). Removed.
- avro (explicit decl): fe-core has zero org.apache.avro references; the declaration
only existed "for iceberg". After removal avro is still on the classpath via
hadoop-client -> hadoop-common -> avro:1.12.1:compile, so nothing is lost.
- parquet-avro -> parquet-hadoop + parquet-column: the real consumer is the HTTP
import-sampling ParquetReader, which uses org.apache.parquet.{hadoop,column,schema,
example.data,io} and never parquet.avro. parquet-avro already pulled parquet-hadoop
+ parquet-column at the same 1.17.0, so the loaded classes are unchanged; declaring
them directly just drops the unused avro bridge. Fixed the stale "For Iceberg"
comment accordingly.
Verified: fe-core -am test-compile green (validate gates pass); dependency:tree
confirms aws-json-protocol gone, avro still supplied transitively, parquet-avro gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
Record the verified removal of the coupled aws-json-protocol / avro / parquet-avro tail, with the resolved-dependency-tree evidence for each, closing out Batch 3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…k deps All three had zero references in fe/**/src and zero transitive consumers (dependency:tree shows dynamodb/logs as fe-core direct-only leaves): - aws-java-sdk-dynamodb / aws-java-sdk-logs (v1): hadoop-aws 3.4.2 no longer ships S3Guard (Hadoop 3.4.0 removed it; the jar contains no DynamoDB class), and the CloudWatch audit destination that the stale "only for apache ranger audit" comment refers to (AmazonCloudWatchAuditDestination) lives in ranger-plugins-audit, which is not on fe-core's classpath (fe-core only pulls ranger-audit-core:2.8.0, whose only destinations are File/base). - bce-java-sdk: BOS object storage is handled via the S3-compatible path (ObjectInfoAdapter 'case BOS' -> S3Properties), so the native Baidu SDK is unused; fe-filesystem does not declare it either. Removing bce-java-sdk also drops its transitive javax.validation:validation-api, which was the only classpath source for the single decorative @NotNull on ExternalMetaIdMgr.replayMetaIdMappingsLog. That annotation is redundant with the Preconditions.checkNotNull(log) on the next line (and is fe-core's only javax.validation usage), so it is removed rather than pulling in a new dependency, keeping fe-core source shrinking only. Parent-pom cleanup of the now-orphaned entries: the mqtt (org.eclipse.paho) management block + mqtt.version property (mqtt was a bce-only transitive), the validation-api management block + property, and the dynamodb/logs version pins. Verified: `test-compile -pl fe-core -am` BUILD SUCCESS (gates pass); resolved dependency:tree confirms the five jars (incl. orphaned mqtt/validation-api) are gone while kept AWS deps (aws-java-sdk-s3 -> kms/core/jmespath) remain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…oved) Backfill the delete verdicts and evidence for aws-java-sdk-dynamodb, aws-java-sdk-logs and bce-java-sdk into the analysis doc / TASKLIST / HANDOFF, and record the adversarial cross-check (4 refutation agents, all refuted=false). Code change: 278d3f4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
Per user request, defer the iceberg row-level DML cluster (~15 files, the largest and highest-risk migration) to the end of the LIVE source-specific logic migration, tackling the smaller, self-contained items first (hudi TVF, ranger-hive authorizer, engine=hive, scattered source branches). The Elasticsearch compat stub stays a long-term-retention candidate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…ersistence stubs The legacy internal-catalog engine=hive cluster is a deprecated/dead feature, not a live one to migrate: engine=hive table creation already throws (InternalCatalog "Cannot create hive table in internal catalog"), catalog.HiveTable is constructed only in a test, the broker "LOAD ... FROM TABLE" path is Spark-Load-only and Spark Load is disabled, and CREATE RESOURCE type=hms is creatable but has no consumer. External Hive access is fully served by the HMS multi-catalog connector, so there is no live capability to delegate. The only remaining obligation is old-image Gson deserialization, so the classes are reduced to thin persistence stubs (mirroring the existing EsTable/EsResource precedent) rather than deleted. Changes: - HiveTable / HMSResource: reduced to @deprecated Gson persistence stubs. Dropped the fe-core property parsing (validate/setProperties), toThrift, the 4-arg ctor and the getters. Kept the @SerializedName fields, no-arg ctors, and (crucially) their GsonUtils registerSubtype registrations + Resource.getLegacyClazz HMS mapping, so an old image/edit-log containing these still deserializes (an unregistered "clazz" tag throws JsonParseException and fails FE startup). - Resource.getResourceInstance: CREATE RESOURCE type=hms now throws "no longer supported" (mirrors the ES case). - Env SHOW CREATE: the two TableType.HIVE arms collapse to a deprecation comment (mirrors the ELASTICSEARCH/JDBC arms), removing the property render from fe-core. - BrokerFileGroup / NereidsBrokerFileGroup: the load-from-table branch (which did `instanceof HiveTable`) collapses to a deprecation throw; behavior is unchanged for every reachable input (it already always threw, since no HiveTable can exist). - MaterializeProbeVisitor: drop the dead HiveTable.class entry (drift left by an earlier connector-table removal). - Remove the obsolete HiveTableTest; add LegacyHiveMetaGsonCompatTest, which guards both invariants (subtype registration kept, and @SerializedName labels unchanged). - LoadCommand: refresh a stale comment about the removed load-from-table capability. - regression-test drop_resource: create/drop an HDFS resource instead of an HMS one (HMS create now throws), preserving the DROP RESOURCE coverage. Verified: fe-core test-compile is green (checkstyle gates pass); the compat guard test passes 2/2; an adversarial clean-room review across 5 dimensions found no blockers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…eep in fe-core The cleanup analysis (B.2) flagged catalog/authorizer/ranger/hive/* as "source-specific behavior" to migrate into a connector. A clean-room adversarial review (5 recon arms + 3 refutation agents, all refuted=false) plus a Trino architecture reference shows this is the same over-classification pattern seen with the legacy engine=hive cluster -- except this time the code is LIVE, not dead. Findings recorded in HANDOFF / TASKLIST / analysis doc: - "hive" names Apache Ranger's built-in "hive" service-definition model (database/table/column/udf), NOT the Doris hive/HMS connector. The 9 files import only org.apache.ranger.* + the fe-core auth framework; zero hive/HMS/metastore imports (the one hadoop import is Configuration for audit). - The controller discards the catalog name (ctl), never branches on data-source type, and is the sole external-catalog Ranger authorizer -- iceberg/paimon/ jdbc/es can all wire it via access_controller.class. It is the sibling of the internal-table ranger-doris authorizer and shares the RangerAccessController base. - It is live (a regression test wires it, reflection-reachable, no disablement gate, no @deprecated) -- the decisive contrast with the dead engine=hive cluster. - Moving it to a connector is infeasible and would violate the fe-core "only out, not in" rule (it would require exporting ~10 auth-framework types and relocating a base class co-owned with ranger-doris that references the internal DorisAccessType). Trino keeps Ranger as one engine-level SystemAccessControl plugin over a generic tabular model, never per-connector. Disposition (signed off): keep as-is in fe-core, no code change. The misleading "hive" package name is the recurrence root cause; a compat-managed rename is deferred as a separate change (access_controller.class persists the FQCN and would need dual-name aliases). Doc-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
hudi_meta was the last datasource-specific "xxx_meta" table function left in fe-core (iceberg/paimon migrated their metadata inspection to native $-suffix system tables). It is deleted outright with no replacement: - fe-core: drop HudiTableValuedFunction + HudiMeta, their registration and dispatch/visitor arms, the MetadataGenerator HUDI arm, and the hudi-exclusive generic seams PluginDrivenExternalTable.supportsMetadataTable / getMetadataTableRows (verified zero non-hudi callers). - connector: drop HudiConnectorMetadata.getMetadataTableRows, HudiConnector's SUPPORTS_METADATA_TABLE capability, the hive-gateway sibling delegation, ConnectorCapability.SUPPORTS_METADATA_TABLE, and the ConnectorMetadata.getMetadataTableRows SPI default. - thrift: deprecate-in-place (keep + mark // deprecated) THudiMetadataParams, THudiQueryType and TMetaScanRange.hudi_params, mirroring the retained-but- deprecated TIcebergMetadataParams / TPaimonMetadataParams. No thrift regen. - BE: delete the now-dead meta_scanner HUDI arm + _build_hudi_metadata_request. - tests: delete test_hudi_meta; the timetravel/incremental/partition-prune suites that used hudi_meta only to enumerate commit timestamps now read the _hoodie_commit_time meta column instead. The hudi data-read path (THudiFileDesc, HUDI_TABLE, HudiScanPlanProvider, HadoopHudiJniScanner, hudi_jni_reader) is a separate feature and untouched. FE test-compile (fe-core + connector modules) is green; the edited connector unit tests pass (15/3/1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…ctors Move the per-source CREATE TABLE DDL validation out of fe-core CreateTableInfo/PartitionTableInfo into the target connectors, mirroring the existing MaxComputeConnectorMetadata.validateColumns precedent and Trino's beginCreateTable model. fe-core only shrinks: the neutral ConnectorCreateTableRequest already carries every field the moved checks need, and the session-variable / capability plumbing already exists, so no new fe-core machinery is added. - iceberg: reject DISTRIBUTE BY, and validate the ORDER BY write sort order (column existence / sortable type / duplicates) in IcebergConnectorMetadata.createTable. - paimon: reject DISTRIBUTE BY in PaimonConnectorMetadata.createTable. - hive: reject NOT NULL columns and enforce the external partition rules (key exists / no float or complex partition column / nullable gated on allow_partition_column_nullable / no duplicate / not-all-columns / fields at the end / order consistent) in HiveConnectorMetadata.createTable. - the cross-engine "only iceberg supports sort order" reject becomes a connector-agnostic gate on the new ConnectorCapability.SUPPORTS_SORT_ORDER (declared by iceberg only), checked in CreateTableInfo.validate; it covers both the SPI connectors and the legacy internal-catalog engines and no longer names a data source. PartitionTableInfo.validatePartitionInfo loses its dead engineName==hive block and the now-unused engineName/columns parameters (callers updated). Each connector gains a *CreateTableValidationTest (mirroring MaxComputeValidateColumnsTest). The fe-core CreateTableCommandTest assertions for the now-delegated checks are removed (covered connector-side). All moved messages are byte-identical, so the e2e suites (test_hive_ddl, test_iceberg_create_table) are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…tubs
The deprecated legacy internal-catalog ES metadata classes EsTable
(engine=elasticsearch) and EsResource (CREATE RESOURCE type=es) are no
longer creatable and have no live callers, so they exist only as thin
Gson persistence stubs kept for reading older FE images / edit logs.
Two invariants must hold or an FE replaying such an image fails to start:
1. the Gson subtype registrations must stay (registerSubtype(EsTable)
/registerSubtype(EsResource) in GsonUtils, plus the getLegacyClazz
ES mapping in Resource) -- the polymorphic "clazz" discriminator
throws JsonParseException for an unregistered tag; and
2. the @SerializedName field labels must stay (pi/tc for EsTable,
properties for EsResource) -- a rename silently drops persisted
field values.
These stubs were the reference implementation the legacy hive stubs
were later modeled on, yet -- unlike the hive stubs, which are covered
by LegacyHiveMetaGsonCompatTest -- they had no guard test. Add a mirror
LegacyEsMetaGsonCompatTest that round-trips an empty stub (invariant 1)
and injects old-image bytes carrying real field values (invariant 2);
the structural partitionInfo field, which a fresh stub leaves null, is
guarded by a direct @SerializedName("pi") reflection assertion.
No production source changes: fe-core source stays reduce-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
Record the T5.5 outcome in the clean-up-deps HANDOFF/TASKLIST: the ES EsTable/EsResource stubs are dead (creation blocked in InternalCatalog and Resource.getResourceInstance) but cannot be safely deleted, because they are registered RuntimeTypeAdapterFactory subtypes that old metadata images / edit logs may still contain -- dropping the registration throws JsonParseException on load and prevents FE startup, with no metadata version escape hatch. Disposition: retain in place with zero source changes; the only action is closing the consistency gap with the hive stubs by adding LegacyEsMetaGsonCompatTest. Next: T5.6 iceberg row-level DML cluster. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…entity from the connector
The position-delete synthesized row-id column (__DORIS_ICEBERG_ROWID_COL__, a
STRUCT{file_path,row_position,partition_spec_id,partition_data}) was defined in
two fe-core classes -- IcebergRowId and IcebergMetadataColumn -- duplicating the
identical column the iceberg connector already declares as a synthetic write
column (IcebergWritePlanProvider.getSyntheticWriteColumns). Delete both classes
and source the identity from the connector, aligning with Trino (the connector
supplies the row-id column; the engine holds the generic row-level DML machine):
- RowLevelDmlRowIdUtils.getRowIdColumn resolves the row-id column from the
table's full schema (already connector-appended during row-level DML), falling
back to a new ungated PluginDrivenExternalTable.getSyntheticWriteColumns()
accessor, and fails loud if the connector declares none (was: reconstruct the
STRUCT in fe-core).
- IcebergMergeCommand's not-matched INSERT-branch NULL-literal type comes from
that column's type instead of the deleted IcebergRowId.getRowIdType().
- IcebergRowLevelDmlTransform's write-constraint exclusion uses an inline
constant set of the four $-prefixed position-delete metadata names.
The FE<->BE wire name __DORIS_ICEBERG_ROWID_COL__ (resolved by name in the BE)
and the STRUCT shape are unchanged, so DML behavior and the BE contract stay
byte-identical. Net fe-core source: +40/-194.
Tests: strengthen the ConnectorColumnConverter contract pin to assert the exact
STRUCT type; add getRowIdColumn branch coverage (full-schema / connector-fallback
/ throw) and the getSyntheticWriteColumns accessor coverage; drop the tests of the
deleted classes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
… (workflow 2 done) Mark T5.6 workflow 2 complete (commit 3cb3d01): the row-id column identity is now sourced from the connector; IcebergRowId/IcebergMetadataColumn deleted. Record the recon correction (the connector already declares the row-id synthetic write column, so the originally-planned new SPI method was redundant) and that mechanical renaming (workflow 1) is the remaining next step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…names
The row-level DELETE/UPDATE/MERGE plan machinery in fe-core (sink nodes,
implementation rules, PlanType/RuleType constants, visitor methods, the merge
partition-field carrier, and the three plan-synthesis helpers) was named Iceberg*
but its content is connector-agnostic engine machinery: it builds the generic
PluginDrivenTableSink, and the iceberg-format thrift is emitted by the connector's
planWrite. Rename it so fe-core reads connector-agnostic:
- Logical/PhysicalIcebergDeleteSink -> Logical/PhysicalExternalRowLevelDeleteSink
- Logical/PhysicalIcebergMergeSink -> Logical/PhysicalExternalRowLevelMergeSink
(+ the two implementation rules, the PlanType and RuleType constants, the
SinkVisitor and PhysicalPlanTranslator visit methods, the ExpressionRewrite
inner rule, and the RequestPropertyDeriver overrides)
- DataPartition/DistributionSpecMerge.IcebergPartitionField -> MergePartitionField
- IcebergDeleteCommand/UpdateCommand/MergeCommand ->
ExternalRowLevel{Delete,Update,Merge}PlanBuilder (they are plan synthesizers, not
Nereids Command subclasses, so the *Command suffix was factually wrong)
IcebergRowLevelDmlTransform keeps its name: it is the genuinely iceberg-specific
strategy (it holds the position-delete $-prefixed metadata column names and the
frozen iceberg_* profile/txn label prefixes).
Pure rename, no behavior change. The FE<->BE contract is untouched: thrift types
(TIceberg*Sink / TIcebergPartitionField), the thrift enum values
TDataSinkType.ICEBERG_{DELETE,MERGE}_SINK, TPartitionType.MERGE_PARTITIONED, the
row-id wire constant Column.ICEBERG_ROWID_COL, the iceberg_* label strings, and the
merge branch-label value all stay byte-identical. EXPLAIN output is unaffected (no
regression .out pins these node names; regular EXPLAIN DELETE/MERGE renders the
generic PluginDrivenTableSink).
Caveat: the four renamed RuleType constants are reachable by string through the
persistable disable_nereids_rules / enable_nereids_rules session variables. A global
that literally names an old rule name would fail after upgrade (uncaught
RuleType.valueOf) -- near-zero probability (these are sink binding/implementation
rules no reasonable user disables), and consistent with the project's standing
no-alias policy for RuleType renames.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…anup task complete Mark T5.6 workflow 1 done (commit f63aecf): the iceberg-named-but-generic row-level DML plan machinery is renamed to neutral names; IcebergRowLevelDmlTransform kept. With T5.6 complete, Batch 5 and the whole fe-core datasource dependency / residual-code cleanup are done. Note the one optional follow-up (DeleteCommandContext dead code + stale migration comments) and the RuleType.valueOf config-string caveat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…igration comments
DeleteCommandContext carried a single field -- a single-valued DeleteFileType enum
(POSITION_DELETE) -- plus a dead toTFileContent() (zero production callers). No branch
ever discriminated on the enum; the value was only threaded through the row-level DML
plan builders, args, and sink nodes and displayed in toString(). Delete the whole class
and un-thread it from RowLevelDmlArgs, DeleteFromCommand, UpdateCommand, the three
ExternalRowLevel*PlanBuilder synthesizers, and the four
Logical/PhysicalExternalRowLevel{Delete,Merge}Sink plan nodes (dropping it from their
constructors, equals/hashCode, and toString). The real FE->BE delete-type is emitted by
the iceberg connector's planWrite (TFileContent.POSITION_DELETES), unaffected.
Also drop the stale internal migration-phase codenames from the row-level DML cluster
comments (P6.6 / T07c / O5-2 / post-flip / post-cutover / pre-cutover) and fix the
RowLevelDmlTransform.handles javadoc that still described the retired table-type check
(it is a connector-capability probe today).
Pure cleanup, no behavior change: compiles clean and the affected unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
… complete) Mark the DeleteCommandContext removal + stale-comment cleanup done (commit 71cfc08). T5.6 and the whole fe-core datasource dependency / residual-code cleanup are now complete with no remaining items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
…ster base) Rebased catalog-spi-review-17 onto the force-pushed upstream branch-catalog-spi via `git pull --rebase upstream-apache branch-catalog-spi`. Fork-point replayed only the 24 review-17 cleanup commits (the lower 41 are old-upstream content the new upstream redid with fresh hashes). One real conflict — StatisticsUtil.getIcebergColumnStats: my dead-code deletion vs master apache#65782's mode=none fix. Resolved by keeping the deletion: the method is dead in upstream too (no production caller after the P6 migration), and apache#65782's write-path fix already lives in the connector IcebergWriterHelper with no read-path equivalent to port. Also removed the two orphaned read-path tests that upstream a2a6f69 added to StatisticsUtilTest (folded into the conflict commit) — git does not flag them as a conflict, so they would otherwise be a latent compile break. Verified: full fe test-compile BUILD SUCCESS + 0 checkstyle violations; 152 affected unit tests green (69 iceberg-connector + 83 fe-core). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
morningman
force-pushed
the
master-catalog-spi-review-17
branch
from
July 22, 2026 03:39
267a094 to
3551b53
Compare
Contributor
Author
|
run buildall |
…in (-54MB) hudi-common transitively bundles org.rocksdb:rocksdbjni (54MB, ~1/4 of the whole connector plugin lib). RocksDB is only used by Hudi's OPT-IN RocksDbBasedFileSystemView / RocksDbDiskMap. The FE hudi connector builds an in-memory file-system view (FileSystemViewManager.createInMemoryFileSystemView in HudiScanPlanProvider) and never selects the RocksDB view; the metadata-table external spillable map defaults to BITCASK (pure-Java). The connector code has zero rocksdb references and the createInMemory path has no static reference to the RocksDB classes, so there is no class-init hazard - the jar is dead weight on the FE metadata read path. Trims the hudi connector plugin lib from 216MB to 162MB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e hudi connector plugin (-27MB)
hudi-common depends on hbase-server for the HFile-format metadata-table reader.
hbase-server transitively drags the HBase RegionServer web-UI stack (hbase-http ->
hbase-shaded-jetty/jersey) and the Hadoop YARN + MapReduce + HDFS-daemon job stack
(hadoop-mapreduce-client-core -> yarn-api/common/client + jline; hadoop-hdfs the
NameNode/DataNode server jar -> leveldbjni; hadoop-distcp) - ~27MB the FE metadata
read path never uses (it never submits a YARN/MR job, starts a RegionServer, or
serves the HBase info UI).
Exclude that job/server/UI infra at hudi-common while KEEPING the HFile reader core
(hbase-server + hbase-common + hbase-client + hbase-protocol*). hadoop-hdfs-CLIENT
(needed for hdfs:// warehouses) previously arrived under the excluded mapreduce
subtree, so re-declare it directly - runtime scope, ${hadoop.version}, hadoop-common
excluded to keep a single copy - mirroring fe-connector-iceberg/paimon and the same
trims fe-connector-hms-hive-shade already applies.
Trims the hudi connector plugin lib to 135MB (216MB -> 135MB together with the
preceding rocksdbjni removal; 230 -> 192 jars). Build + assembly verified GREEN;
runtime Hudi read regression (COW/MOR x S3/HDFS x metadata-table enabled) still
pending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29558 ms |
…cit partition values The catalog-SPI hive refactor (apache#65473) moved external hive CREATE TABLE partition validation out of fe-core into HiveConnectorMetadata.createTable. That merge reordered two checks which previously ran in different phases: column-existence was validated in fe-core PartitionTableInfo during analysis (first), while the explicit-partition-values rejection lived in HiveMetadataOps.createTable during execution (later). In the connector both now run together, but the explicit-values rejection was placed ahead of the validatePartition column-existence check. Consequently a CREATE TABLE that both references a non-existent partition column and specifies explicit partition values, e.g. PARTITION BY LIST (pt000) (PARTITION pp VALUES IN ('2014-01-01')) reported "Partition values expressions is not supported in hive catalog." instead of the more specific "partition key pt000 is not exists", regressing the user-facing diagnostic for a typo'd partition column and breaking external_table_p0/hive/ddl/test_hive_ddl (line 649). Move validatePartition() ahead of the hasExplicitPartitionValues() rejection to restore the pre-migration precedence. The validation stays in the connector (no fe-core validation is re-added). allow_partition_column_nullable defaults to true, so a valid nullable partition column still reaches and triggers the explicit-values rejection; validatePartition early-returns for non-partitioned tables. Sibling sub-tests at lines 620 (bad column, no values) and 634 (valid column, explicit values) keep their existing error messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
Contributor
TPC-DS: Total hot run time: 177709 ms |
Contributor
ClickBench: Total hot run time: 24.93 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.