[Draft] Master catalog spi review 22 - #66212
Closed
morningman wants to merge 87 commits into
Closed
Conversation
This multi-month refactor needs persistent state for progress, decisions, risks, and cross-session agent handoff. Establishes a file-based tracking system including dashboard, ADR decision log, deviation log, risk register, per-stage task files, per-connector tracking, and an agent collaboration playbook covering context budget / subagent usage / handoff norms. Closes 18 design decisions (D-001..D-018) and registers 14 risks (R-001..R-014). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…T27) (apache#63582) ## Summary Lands the P0 SPI baseline for the catalog-SPI migration (master plan §3.1 / RFC §2.1), with zero impact on the already-migrated JDBC + ES connectors. - **Batch 0** (commits 1-2): SPI types + fe-core bridges — `ConnectorMetaInvalidator`, `ConnectorTransaction`, `ConnectorMvccSnapshot`, `ExternalMetaCacheInvalidator`, `ConnectorMvccSnapshotAdapter`, `PluginDrivenTransactionManager` generalization. - **Batch 1** (commit 3): DDL + Partition SPI — `ConnectorCreateTableRequest` + 4 spec POJOs, 4 new defaults on `ConnectorTableOps`, 3 new fields on `ConnectorPartitionInfo`, fe-core converter, `PluginDrivenExternalCatalog.createTable` routing. - **Batch 2** (commit 4): Import-gate + unit tests — `tools/check-connector-imports.sh` wired through exec-maven-plugin; `FakeConnectorPlugin` covering every default fall-through; routing tests for the invalidator; converter tests for all 4 partition styles + 2 bucket flavors. ## Commits - `[feat](connector) add P0 batch 0 SPI baseline: MetaInvalidator / Transaction / MvccSnapshot` (T03-T08) - `[feat](connector) wire P0 batch 0 SPI into fe-core` (T09-T12) - `[feat](connector) add P0 batch 1 SPI: CreateTableRequest + listPartitions` (T13-T20) - `[feat](connector) add P0 batch 2 gate + unit tests` (T21-T23, T26-T27) ## Test plan - [x] `mvn -pl fe-connector/fe-connector-api,fe-connector/fe-connector-spi -am compile` — SPI modules compile - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` — fe-core compile - [x] `mvn -pl fe-core checkstyle:check` — 0 violations - [x] `mvn -pl fe-connector validate` — import gate runs and passes (baseline clean) - [x] `mvn -pl fe-core -am test -Dtest='FakeConnectorPluginTest,ExternalMetaCacheInvalidatorTest,CreateTableInfoToConnectorRequestConverterTest,ConnectorPluginManagerTest,ConnectorSessionImplTest'` — 39/39 green - [x] `mvn -pl fe-connector/fe-connector-jdbc,fe-connector/fe-connector-es -am compile` — downstream connectors compile unchanged - [ ] JDBC regression-test suite (T24) — to be exercised by this PR's CI pipeline - [ ] ES regression-test suite (T25) — to be exercised by this PR's CI pipeline ## Tracking Full plan, decisions, and risk log live under `plan-doc/` in the repo (introduced by 6315983, already on the base branch). Per-task status: `plan-doc/tasks/P0-spi-foundation.md`. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pache#63641) P1 batch A — close out scan-node SPI consolidation while keeping migration-period fallbacks in place. Three surgical changes route `PluginDrivenExternalTable` first in the nereids translator hot paths so already-migrated SPI connectors (JDBC, ES) take the SPI route, while the existing `instanceof XExternalTable` chains remain as fallbacks for connectors still pending migration (P3–P7). - **T3** — `PhysicalPlanTranslator.visitPhysicalFileScan`: move the existing `PluginDrivenExternalTable` branch from position 8 to position 1; the 7 connector-specific branches (HMS / Iceberg / Paimon / Trino / MaxCompute / LakeSoul / RemoteDoris) stay in place as migration-period fallbacks - **T4** — `PhysicalPlanTranslator.visitPhysicalHudiScan`: add a `PluginDrivenExternalTable` branch routed to `PluginDrivenScanNode.create(...)`, threading `tableSnapshot` + `scanParams` through `FileQueryScanNode` setters; `incrementalRelation` flagged as a P3 Hudi SPI extension TODO. The new branch is unreachable today (`PhysicalHudiScan` is only built for `HMSExternalTable + DLAType.HUDI`), so this is groundwork for P3 with zero current-day runtime impact - **T5** — `LogicalFileScan`: in `computeOutput()`, add a `PluginDrivenExternalTable` branch calling new helper `computePluginDrivenOutput()` — same shape as `computeIcebergOutput`, using `getFullSchema()` + virtualColumns; in `supportPruneNestedColumn()`, add an explicit `PluginDrivenExternalTable → false` branch. Both behaviorally equivalent for JDBC/ES today since they have no hidden cols and no virtualColumns P1 batch B (T1 — delete 13 legacy `Jdbc*Client` + `JdbcFieldSchema`) is deferred to P8 because the 3 fe-core callers — `PostgresResourceValidator`, `StreamingJobUtils`, `CdcStreamTableValuedFunction` — are live CDC streaming code that requires SPI extension for `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables`, which is out of P1 surgical scope. Background and tracking docs live in `plan-doc/` (Master Plan §3.2 P1, tasks/P1-scan-node-cleanup.md, decisions log). - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` → BUILD SUCCESS - [x] `mvn -pl fe-core checkstyle:check` → 0 violations - [x] JDBC + ES regression-test passing — baseline established in P0 / PR apache#63582 - [ ] PR CI green on this PR - [ ] Manual scan-node smoke for an SPI connector — JDBC `SELECT *` should fall into the new `PluginDrivenExternalTable` branch first 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…apache#64096) ### What problem does this PR solve? Related PR: apache#63582 (P0 — SPI baseline), apache#63641 (P1 — nereids plugin-driven routing) Problem Summary: This is **P2** of the catalog SPI migration and targets the `branch-catalog-spi` feature branch (continuing P0 apache#63582 and P1 apache#63641). It fully migrates `trino-connector` off the legacy in-tree `fe-core/datasource/trinoconnector/` implementation and onto the connector SPI module `fe-connector-trino`, making `trino-connector` the first connector to complete the SPI consumption playbook that later connectors will reuse as a template. All five batches land together so there is no intermediate state where a newly-created trino catalog cannot be serialized. **Batch A — complete the SPI surface (`fe-connector-trino` only, no fe-core changes)** - `TrinoConnectorProvider.validateProperties`: enforce the required `trino.connector.name` property at `CREATE CATALOG` time (ported from the legacy `checkProperties`). - `TrinoDorisConnector.preCreateValidation`: call `ensureInitialized()` so plugin loading + connector-factory resolution happen at catalog creation instead of being deferred to the first `SELECT`. - `TrinoConnectorDorisMetadata.applyFilter` / `applyProjection`: bridge Trino native filter/projection pushdown, reusing `TrinoPredicateConverter` to translate a Doris `ConnectorExpression` into a Trino `TupleDomain`. `remainingFilter` is conservatively returned as the original expression to match legacy behavior (conjuncts are not stripped; BE re-evaluates them). **Batch B — fe-core bridge for image compatibility** - `GsonUtils`: atomically replace the three legacy `registerSubtype` entries (`TrinoConnectorExternalCatalog` / `Database` / `Table`) with `registerCompatibleSubtype` redirects onto the `PluginDrivenExternal*` hierarchy. This must be atomic — `RuntimeTypeAdapterFactory` rejects duplicate labels, so keeping both bindings would throw at static init. Mirrors what ES/JDBC already did. - `PluginDrivenExternalCatalog.gsonPostProcess`: extract a `legacyLogTypeToCatalogType()` helper that maps `Type.TRINO_CONNECTOR` → `"trino-connector"`; the generic `name().toLowerCase()` would otherwise produce the wrong `"trino_connector"` (underscore) that `CatalogFactory` does not recognize. - `PluginDrivenExternalTable.getEngine()` / `getEngineTableTypeName()`: add `trino-connector` branches that preserve the legacy engine-name / table-type display across `SHOW TABLE STATUS` and `information_schema`. **Batch C — flip the switch** - Add `"trino-connector"` to `CatalogFactory.SPI_READY_TYPES` so catalog creation routes through the SPI path. **Batch D — remove legacy code** - Drop the `instanceof TrinoConnectorExternalTable` scan branch in `PhysicalPlanTranslator` (the `PluginDrivenExternalTable` SPI branch already handles it). - Drop `case "trino-connector"` in `CatalogFactory`. - Delete `fe-core/datasource/trinoconnector/` (10 files) and the now-dead legacy `TrinoConnectorPredicateTest`. - Route the `TRINO_CONNECTOR` db-build case in `ExternalCatalog` to `PluginDrivenExternalDatabase` (mirrors the migrated JDBC case). - **Retained for image compatibility**: the `InitCatalogLog.Type.TRINO_CONNECTOR` and `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` enums, the GsonUtils redirects, and the `MetastoreProperties` trino-connector entry. **Batch E — tests + tracking docs** - 29 JUnit 5 unit tests over the plugin-free converters: - `TrinoPredicateConverterTest` — `ConnectorExpression` pushdown trees → Trino `TupleDomain` (EQ / range / NE / IN / IS [NOT] NULL / AND / OR, Slice encoding), plus graceful degradation to `TupleDomain.all()` on null/unsupported input. - `TrinoTypeMappingTest` — Trino SPI type → Doris `ConnectorType` (scalars, decimal precision/scale, timestamp precision clamp, array/map/struct, unsupported-type failure). - `TrinoConnectorProviderTest` — `validateProperties` fast-fails when `trino.connector.name` is missing/empty. - No Trino plugin/cluster required; plugin-dependent paths remain covered by the existing `external_table_p0/p2` `trino_connector` regression suites. - Sync the migration tracking docs under `plan-doc/` (already carried on this feature branch since P0). **Net effect**: 28 files, +1025 / −2681 (~1656 LOC net removed). Old FE images holding legacy trino catalogs / databases / tables deserialize onto the `PluginDrivenExternal*` hierarchy through the GsonUtils string-name redirect, with engine-name display preserved. **Deferred (follow-ups, not in this PR)**: - `trino_connector_migration_compat` regression test (old-image deserialization) — requires a running cluster + Trino plugin + docker, unavailable in this dev environment; tracked as a CI/cluster follow-up. - The plugin-install documentation update lives in the `doris-website` repo and is handled separately. ### Release note None ### Check List (For Author) - Test - [x] Unit Test — 29 new tests in `fe-connector-trino` (predicate converter / type mapping / property validation). - [ ] Regression test — existing `trino_connector` suites cover plugin paths; the new old-image compat regression is deferred to a CI/cluster follow-up. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. Internal routing moves from the legacy fe-core path to the SPI path; image compatibility, engine-name display, and pushdown semantics all mirror the legacy behavior. All batches land together, so there is no serialization-gap window. - Does this need documentation? - [x] Yes. The trino-connector plugin-install doc update is a follow-up in the `doris-website` repo. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
…tch design (hybrid, T02-T08) (apache#64143) ## Proposed changes testing with apache#64146 P3 of the catalog-SPI migration (base: `branch-catalog-spi`). Migrates the **hudi** connector following the **hybrid** strategy (D-019): harden the dormant HMS-over-SPI hudi connector to correctness parity, build a test baseline, and write the per-table dispatch design — **all behind the closed gate** (`SPI_READY_TYPES` unchanged). >⚠️ **No user-visible behavior change.** The SPI hudi path stays dormant (gate closed); hudi queries continue to use the legacy `HMSExternalTable.dlaType=HUDI` path. This PR removes correctness blockers ahead of the live cutover (deferred to P7 / batch E). ### What's included **Correctness fixes (hardening dormant code, behind gate):** - **T02** — fix hudi JNI `column_types` double bug: emit full Hive type strings (was Doris bare type names, losing precision/scale/subtypes) and send `column_names`/`column_types`/`delta_logs` as typed lists end-to-end (was comma join/split, which shattered `decimal(10,2)` / `struct<...>`). Matches the BE `hudi_jni_reader.cpp` contract (names `,` / types `#` / delta `,`). - **T04** — fail loud on time-travel / incremental read in the SPI `visitPhysicalHudiScan` branch (was silently returning the latest snapshot / silently full-scanning). - **T05** — real EQ/IN partition pruning in `HudiConnectorMetadata.applyFilter` (was a placeholder that ignored predicates and unconditionally switched the partition source from Hudi-metadata to HMS); faithfully mirrors `HiveConnectorMetadata.applyFilter`. - **T07** — column-name casing fix in `avroSchemaToColumns` (top-level lowercase, mirroring legacy `HMSExternalTable`). **Test baseline (all three connector modules started P3 with 0 tests):** - `fe-connector-hudi` (33): type-mapping / schema-parity (COW/MOR golden) / table-type / partition-pruning / scan-range. - `fe-connector-hms` (12): shared Hive-type-string parser tests. - `fe-connector-hive` (14): file-format / partition-pruning (mirrors T05). - COW/MOR schema is **type-agnostic** (golden parity vs legacy `initHudiSchema`); table type only affects scan planning. **Decisions / design (code-grounded, design-only):** - **T03** — defer `schema_id`/`history_schema_info` field-id evolution to batch E (DV-006; not a model-agnostic SPI fix). - **T06** — keep MVCC/snapshot SPI defaults (opt-out) + document (DV-007). - **T08** — `tableFormatType` dispatch design memo + **D-020**: single `hms` catalog per-table routing via a new backward-compatible `ConnectorMetadata.getScanPlanProvider(handle)` (per-table provider seam); refines D-005. The keystone gap is split into M1 (identity consumption, fe-core reads `tableFormatType` as an opaque string) and M2 (scan routing). ### Deferred to batch E / P7 (not in this PR) Gate flip (`SPI_READY_TYPES += hms/hudi`), fe-core `tableFormatType` consumption (M1+M2 implementation), live cutover, delete legacy `datasource/hudi/`, full incremental/time-travel/MVCC, Iceberg-on-hms via SPI (needs P6 `IcebergScanPlanProvider`), cluster/runtime validation. ### Verification Per task tracking, each code batch landed with: per-module compile + checkstyle 0 (incl. test sources) + connector import-gate pass + new unit tests green. The two most recent commits are docs-only (`plan-doc/`); the code is unchanged since the last green batch. Gate stays closed → the dormant SPI path is unreachable at runtime → zero live-path risk. CI re-verifies. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…core + make fe-core odps-free (T07-T09) (apache#64300) Follow-up to apache#64253 (the MaxCompute catalog-SPI cutover). After the cutover a `max_compute` catalog deserializes to `PluginDrivenExternalCatalog` and no legacy `MaxComputeExternal*` object is ever instantiated, so the legacy MaxCompute subsystem in fe-core is dead code. This removes it and makes fe-core's dependency tree fully odps-free. **1. Remove legacy subsystem** (`7a4db351100`) - Delete 20 fe-core files: `datasource/maxcompute/*` (incl. `MCTransaction`, `MaxComputeScanNode`/`Split`), the MaxCompute sink/insert/txn plumbing, and 2 legacy-only tests. - Clean ~21 reverse-reference sites (imports + dead `instanceof`/visitor/rule branches), keeping every `PluginDriven`/connector sibling branch and the image/replay keep-set (GsonUtils compat strings; `TableType`/`TransactionType`/`TableFormatType`/`InitCatalogLog.Type` `MAX_COMPUTE` enums; block-id thrift). - Rewire 3 tests; e.g. `FrontendServiceImplTest`'s block-id RPC test now mocks the generic `Transaction` SPI, since `getMaxComputeBlockIdRange` reads the PluginDriven connector transaction. **2. Make fe-core odps-free** (`409300a75b8`) - Drop the two odps deps from `fe-core/pom.xml`. - Move `MCUtils` from fe-common into `be-java-extensions/max-compute-connector` (its only consumer after the removal); keep `MCProperties` (odps-free constants) in fe-common. - Drop `odps-sdk-core` from fe-common — it was also leaking netty/protobuf transitively to fe-common's own `DorisHttpException`/`GsonUtilsBase`, so declare `netty-all` + `protobuf-java` directly (proper dependency hygiene). **3. Doc-sync** (`f8c305765e8`) — plan-doc PROGRESS/HANDOFF/deviations/design tracking notes. - `mvn -pl :fe-core -am test-compile` (main+test) passes; checkstyle 0 violations; connector import-gate passes. - `grep -rn com.aliyun.odps fe/fe-core/src` → empty. - `mvn -pl :fe-core dependency:tree | grep odps` → empty (no odps, direct or transitive). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he#64446) (apache#64446) Migrates the legacy in-tree Paimon catalog (fe-core `datasource/paimon`, `metacache/paimon`, `property/metastore/*Paimon*` and their reverse references) onto the catalog SPI as a self-contained `fe-connector-paimon` plugin, following the maxcompute full-adopter + cutover template. `paimon` is added to `SPI_READY_TYPES`, so a Paimon catalog now deserializes to `PluginDrivenExternalCatalog` and all five functional areas — normal-table read, system-table read, DDL, MTMV/MVCC/time-travel (procedures are a doc-only no-op: fe-core has none) — flow through the generic `PluginDriven*` bridge with no Paimon-specific branch in fe-core. Paimon is the first lakehouse full-adopter to exercise the MVCC / sys-table / MTMV SPI surfaces, which become the reusable template for the upcoming iceberg/hudi migrations, so the cutover is held to byte-parity with the legacy path. The legacy `datasource/paimon/*` classes are left in place (now dead on the cutover path) and removed in a follow-up, mirroring the maxcompute cutover -> P4 removal. DefaultConnectorContext) Source-agnostic seams so any connector can express these capabilities; every new method is a default no-op/identity, so hive/iceberg/jdbc/maxcompute/ trino/es are byte-for-byte unaffected. - MVCC/time-travel: `resolveTimeTravel(session, handle, ConnectorTimeTravelSpec)` + `applySnapshot(...)` (threads a pinned snapshot into the handle before planScan); new immutable `ConnectorTimeTravelSpec` (SNAPSHOT_ID / TIMESTAMP / TAG / BRANCH / INCREMENTAL) that fe-core extracts from SQL; `ConnectorMvccSnapshot` carries `schemaId` for schema-evolution-aware time travel. - System tables: `listSupportedSysTables` / `getSysTableHandle`; fe-core composes the `{base}${sys}` reference name. - Scan: proportional split-weight (`getSelfSplitWeight`/`getTargetSplitSize`), COUNT(*) pushdown (`getPushDownRowCount`, 7-arg `planScan`), native-read accounting (`isNativeReadRange`), `getDeleteFiles` (VERBOSE merge-on-read), `ignorePartitionPruneShortCircuit` (predicate-driven connectors keep genuine-null partitions). - Metadata: `ConnectorPartitionInfo.fileCount` + `SUPPORTS_PARTITION_STATS` (rich SHOW PARTITIONS); `ConnectorColumn.withTimeZone` (WITH_TIMEZONE DESC marker, independent of the timestamp_tz mapping flag); connector cache control (`invalidateTable`/`invalidateAll`, `schemaCacheTtlSecondOverride`). - `ConnectorContext`: six engine-side hooks the connector must not implement itself (`loadHiveConfResources`, `vendStorageCredentials`, `getBackendStorageProperties`, `normalizeStorageUri` x2, `getStorageProperties`), bound in `DefaultConnectorContext` over fe-core's single-source-of-truth credential/URI/HiveConf utilities (no re-ported logic that could drift). The metastore-side counterpart of the fe-filesystem StorageProperties SPI: a Paimon catalog resolves its backend (hive/dlf/rest/jdbc/filesystem) by ServiceLoader dispatch on `paimon.catalog.type` instead of a fe-core switch. - `fe-connector-metastore-api`: hadoop/SDK-free `MetaStoreProperties` contracts (HMS/DLF/REST/JDBC/FileSystem) carrying only Map/scalar facts. - `fe-connector-metastore-spi`: `MetaStoreProvider<P>` + first-hit `MetaStoreProviders` dispatcher + five impls, registered via META-INF/services. `HmsMetaStorePropertiesImpl` reproduces the legacy `buildHmsHiveConf` key set and parity-critical ordering (storage overlay before the kerberos block; username/sasl last). Adding a backend = one provider class + one services line (no central enum/switch). - ServiceLoader is loaded with the SPI interface's own (plugin) classloader, NOT the thread-context CL: at CREATE CATALOG the static init first fires on an FE worker thread whose TCCL is the FE app loader, so a 1-arg load would cache an empty provider list process-wide. - fe-core bridges add `initExecutionAuthenticator(List<StorageProperties>)` so HDFS-Kerberos doAs is wired on the plugin path (legacy `initializeCatalog` is dead). Full read + DDL + partitions + statistics + system tables + MVCC/time-travel across all five metastore flavors, planning native (ORC/Parquet) vs JNI reads. - `PaimonCatalogFactory` (pure flavor switch: Options + Hadoop Configuration + HiveConf, classloader-pinned for child-first loading), `PaimonCatalogOps` (injection seam over the remote Catalog for offline-testable metadata), `PaimonConnectorMetadata` (~9 -> 28 methods: schema latest/at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor). - `PaimonScanPlanProvider` (~6 -> 30 methods): predicate/projection pushdown, native-vs-JNI routing, large-file sub-splitting with deletion vectors, COUNT(*) collapse, schema-evolution field-id dictionary, `$ro` unwrap to base FileStoreTable. - New: `PaimonSchemaBuilder` (CREATE TABLE), `PaimonIncrementalScanParams` (@incr), `PaimonTableResolver` (sys/branch-aware handle -> Table), `PaimonLatestSnapshotCache` + `PaimonSchemaAtMemo` (per-catalog caches on the long-lived connector). - `PaimonTypeMapping` gains the write direction (Doris -> Paimon) for CREATE TABLE. - NEW `PluginDrivenMvccExternalTable` (generic MvccTable + MTMV host: the runtime class for paimon tables), `PluginDrivenMvccSnapshot`, `PluginDrivenSysExternalTable` + `systable/PluginDrivenSysTable`. Behavior is selected by `ConnectorCapability` (SUPPORTS_MVCC_SNAPSHOT, SUPPORTS_PARTITION_STATS) — no paimon branch in fe-core. - `PluginDrivenScanNode`: MVCC pin threaded at every handle-consumption site, sys-table time-travel guard, COUNT pushdown, connector-agnostic EXPLAIN re-emission. - Cutover: `CatalogFactory` adds `paimon` to `SPI_READY_TYPES` and drops the built-in case; `GsonUtils` removes the 7 built-in `Paimon*` subtype registrations and adds replay compat so persisted `PaimonExternalCatalog` (all 5 flavors) / `...Database` -> `PluginDriven*` and `PaimonExternalTable` -> `PluginDrivenMvccExternalTable` on deserialization (existing clusters upgrade without losing catalogs). - Nereids/DDL: `PhysicalPlanTranslator` drops the `PaimonScanNode` branch; `UserAuthentication`, `CreateTableInfo`, `ShowPartitionsCommand` generalized; `Env.getDdlStmt` renders LOCATION/PROPERTIES for SHOW CREATE (paimon engine, no credential leak). - fe-filesystem: NEW `HdfsFileSystemProperties` / `HdfsConfigFileLoader` (typed HDFS; defaults-laden BE map vs defaults-free FE Hadoop-config map so it cannot clobber a co-bound object store's fs.s3a.* during a multi-backend merge); S3 MinIO support + legacy tuning defaults; OSS/OBS/COS ANONYMOUS provider-type; new `FileSystemFactory.bindAllStorageProperties` / `FileSystemPluginManager.bindAll`. - NEW `fe-kerberos` leaf module: dependency-free `AuthType` / `KerberosAuthSpec` facts shared across modules without pulling in Hadoop. - Packaging: NEW `fe-connector-paimon-hive-shade` relocates `org.apache.thrift` for HMS 2.3.7; the paimon plugin-zip excludes fe-thrift/libthrift (provided parent-first) to avoid a split `TBase` across loaders; self-contained bundle (hadoop-aws, AWS-SDK, hadoop-hdfs-client). `paimon.version` (1.3.1) pinned as the single property across FE connector / BE paimon-scanner / preload-extensions for FE<->BE Table/Split serde. - BE `PaimonJniScanner.getPredicates()` null-predicate backstop (no-filter scan; pairs with the FE producer now always emitting an empty predicate list). Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Storage/credentials: canonical `s3.*`/`oss.*` keys -> `fs.s3a.*`/`fs.oss.*` (STORAGE-CREDS); canonical `AWS_*` BE creds (STATIC-CREDS-BE); per-table vended DLF/STS token (REST-VENDED); `oss/cos/obs/s3a` -> `s3://` for data + deletion-vector paths (URI-NORMALIZE); external `hive.conf.resources` hive-site.xml (HMS-CONFRES); real doAs on filesystem/jdbc + wrapped HMS read RPCs (KERBEROS-DOAS). - Reads/types: paimon columns forced nullable (READ-NOTNULL); `VARCHAR(65533)` not widened to STRING (VARCHAR-BOUNDARY); dotted `enable.mapping.varbinary` / `.timestamp_tz` keys (MAPPING-FLAG-KEYS); per-type partition-value rendering (NATIVE-PARTVAL); genuine `\N` not coerced to NULL (PARTITION-NULL-SENTINEL); field-id history dict for rename-safe native reads (SCHEMA-EVOLUTION); paimon-native split encode under `enable_paimon_cpp_reader` (CPP-READER). - Planning/perf: restore the `force_jni_scanner` escape hatch (FORCE-JNI-SCANNER); intra-file native sub-splitting (NATIVE-SUBSPLIT); precomputed COUNT(*) row count (COUNT-PUSHDOWN); table row count for CBO / SHOW TABLE STATUS (TABLE-STATS). - Time travel / DDL: `FOR TIME AS OF` under CST/PST session zones (TZ-ALIAS); resolved + validated `driver_url` for jdbc catalogs (JDBC-DRIVER-URL); reject a local-only name conflict on CREATE TABLE (CREATE-TABLE-LOCAL-CONFLICT). plan-doc P5 design / task-list / HANDOFF / deviation tracking notes. - ~70 new FE unit suites: paimon connector (offline recording fakes: `RecordingPaimonCatalogOps` / `FakePaimonTable`), fe-core PluginDriven* MVCC / sys-table / scan-node / split-weight, metastore-spi dispatch + per-backend properties, fe-filesystem HDFS/MinIO/anonymous, fe-kerberos. - checkstyle 0 violations; connector import-gate passes; FE main+test compile clean. - End-to-end paimon behavior is docker-gated (`enablePaimonTest=true`); the five functional areas are covered by the `external_table_p0/paimon` regression suites, green on the latest CI run for this branch.
… make fe-core paimon-SDK-free (T29) (apache#64653) Follow-up to apache#64446 (the Paimon catalog-SPI cutover). After the cutover a `paimon` catalog deserializes to `PluginDrivenExternalCatalog` and its tables to `PluginDrivenMvccExternalTable`, so no legacy `Paimon*` catalog / table / scan-node object is ever instantiated — the entire legacy Paimon subsystem in fe-core is dead code. This PR removes it and makes fe-core's dependency tree fully paimon-SDK-free (zero `org.apache.paimon` imports, the 5 paimon maven deps dropped). Mirrors the P4 maxcompute cutover → legacy-removal (apache#64256). **1. Remove legacy subsystem + sever reverse-refs** (`7632a074e4b`) - Delete the dead fe-core paimon files: `datasource/paimon/*` (all 5 flavor catalogs, `PaimonExternalTable`, `PaimonMetadataOps`, `PaimonUtil`, `source/PaimonScanNode` / `PaimonSplit` / `PaimonPredicateConverter` / `PaimonSource` / `PaimonValueConverter`, `profile/*`), `datasource/metacache/paimon/*`, `systable/PaimonSysTable`, and the legacy-only tests. - Sever the live reverse-references to the deleted classes, keeping every `PluginDriven` / connector sibling branch and the image/replay keep-set: drop the `PAIMON` db-creation switch arm in `ExternalCatalog`; the paimon engine routing in `ExternalMetaCacheMgr` / `ExternalMetaCacheRouteResolver`; the dead `PAIMON_EXTERNAL_TABLE` branch in `Env.getDdlStmt` (keep the LIVE `PLUGIN_EXTERNAL_TABLE` SHOW CREATE LOCATION/PROPERTIES rendering); the `PaimonSysExternalTable` branch in `UserAuthentication`; the legacy clauses + `handleShowPaimonTablePartitions()` in `ShowPartitionsCommand` (keep `hasPartitionStatsCapability()` and the `TableType.PAIMON_EXTERNAL_TABLE` enum). Landed as one compiling unit (mirrors the P4 apache#64300 precedent: `PaimonUtils` still called the removed `ExternalMetaCacheMgr.paimon()`). - Inline the `getPaimonCatalogType()` string literals (`hms`/`filesystem`/`dlf`/`rest`/ `jdbc`) to decouple the still-consumed thin metastore-property classes from the deleted `PaimonExternalCatalog`. **2. Make fe-core paimon-SDK-free** (`32de7d16702`) - Strip the dead paimon-SDK catalog-building cluster from `AbstractPaimonProperties` + the 5 flavors (`initializeCatalog` / `buildCatalogOptions` / `appendCatalogOptions` / `getMetastoreType` / `normalizeS3Config` / … — 0 live callers; the plugin path builds catalogs connector-side). Keep every LIVE SDK-free duty: `warehouse` `@ConnectorProperty`, the Kerberos `executionAuthenticator` doAs wiring (read by `PluginDrivenExternalCatalog`), property normalization/validation, `getPaimonCatalogType`, `Type.PAIMON`. - Vended credentials: delete `PaimonVendedCredentialsProvider` + its test + the `VendedCredentialsFactory` `case PAIMON`. Its SDK methods were dead — reachable only via the iceberg-only `getStoragePropertiesMapWithVendedCredentials`; the real paimon vended path is the connector's `PaimonScanPlanProvider.extractVendedToken` (moved off fe-core at cutover). Relocate its one LIVE duty (the REST "skip static storage map" gate) to a new SDK-free `MetastoreProperties.isVendedCredentialsEnabled()` (base=false, `PaimonRestMetaStoreProperties`=true); `CatalogProperty` routes iceberg through its provider (byte-identical) and everything else through the metastore-props method. - pom: drop `paimon-core` / `-common` / `-format` / `-s3` / `-jindo` from `fe-core/pom.xml`. `fe/pom.xml` `paimon.version` is kept (fe-connector-paimon + BE paimon-scanner still consume it). **3. Doc-sync** (`58160cbcb56`, `895e214b2bd`) — plan-doc HANDOFF / PROGRESS / connectors / P5-T29 legacy-removal design tracking notes. - `mvn -pl :fe-core -am test-compile` (main + test) passes; checkstyle 0 violations; connector import-gate (`tools/check-connector-imports.sh`) passes. - `grep -rn org.apache.paimon fe/fe-core/src` → empty (main and test). - `mvn -pl :fe-core dependency:tree -Dincludes=org.apache.paimon` → empty (no paimon dep, direct or transitive). - End-to-end paimon regression (`external_table_p0/paimon`, docker `enablePaimonTest=true`) passed on this branch.
…kerberos (apache#64655) **P3b: consolidate the drifted Kerberos/Hadoop authentication implementations into the new top-level neutral leaf module `fe-kerberos`** as the single source of truth. Done as 3 commits: 1. **trino → JDK** (`4a740e1`) — replace the only external dependency in the auth path, trino's `KerberosTicketUtils`, with a JDK-only (`javax.security.auth.kerberos`) byte-for-byte equivalent, so the kerberos path is trino-free. 2. **relocate** (`8898e15`) — move the 13 `fe-common` `security.authentication.*` classes to `org.apache.doris.kerberos.*` in `fe-kerberos`; retarget all consumer imports (fe-core + 3 be-java-extensions scanners); merge the duplicate `AuthType`. 3. **unify interface** (`5e3e896`) — merge the two competing `HadoopAuthenticator` interfaces (fe-common's `PrivilegedExceptionAction` variant vs fe-filesystem-spi's `IOCallable` variant) into the single fe-kerberos one, and delete fe-filesystem-hdfs's own `KerberosHadoopAuthenticator`/`SimpleHadoopAuthenticator` copies (which had drifted from the canonical impls). `DFSFileSystem` now routes through the shared authenticators. `fe-kerberos` remains a top-level neutral leaf (no dependency cycle). HDFS filesystem access now uses the same authenticators as the HMS path (restoring parity). Two intentional behavior changes in fe-filesystem-hdfs: simple / no-`hadoop.username` now runs as remote user `hadoop` (was: FE process user, direct); kerberos uses the shared `LoginContext` + 80%-lifetime refresh. fe-filesystem-hdfs 79/0/0 (+fe-kerberos/spi), checkstyle 0, connector import-gate clean, whole-repo grep for the removed symbols = 0. >⚠️ docker kerberos e2e (HDFS kerberized + HMS) NOT yet run — the real gate; UGI login can't be exercised in unit tests.
…move legacy fe-core subsystem (apache#64688) Migrates the legacy in-tree Iceberg catalog (fe-core `datasource/iceberg`, its `property/metastore/*Iceberg*` clusters, and every reverse `instanceof IcebergExternal*` / `case ICEBERG` coupling scattered across nereids / planner / alter) onto the catalog SPI as a self-contained `fe-connector-iceberg` plugin, following the paimon full-adopter + cutover template. `iceberg` is added to `SPI_READY_TYPES`, so an Iceberg catalog now deserializes to `PluginDrivenExternalCatalog` and every functional area — read, write, row-level DML, `ALTER TABLE EXECUTE` procedures, system tables, DDL and MVCC/time-travel — flows through the generic `PluginDriven*` bridge with no Iceberg-specific branch in fe-core. (Iceberg tables surfaced *through* a `hive`/HMS catalog stay on the hive path until P7; this PR covers the dedicated `type=iceberg` catalog.) Iceberg is the widest lakehouse adopter migrated so far — 7 catalog flavors, merge-on-read row-level DML, 9 stored procedures, v3 row-lineage / deletion vectors — so the cutover is held to behavior-parity with the legacy path. Unlike the paimon migration (a cutover PR + a separate removal PR), this PR does the build-out, the live cutover **and** the full legacy-subsystem removal on one branch: fe-core `datasource/iceberg` and its property clusters are deleted, not just left dead. Part of the catalog-SPI migration tracked in apache#65185 (phase **P6**). Full read + write + row-level DML + procedures + system tables + DDL + MVCC/time-travel, planning native (Parquet/ORC) vs JNI reads. Key seams: - `IcebergCatalogFactory` (pure flavor switch, classloader-pinned for child-first loading), `IcebergCatalogOps` (metadata / DDL ops), `IcebergConnectorMetadata` (schema latest / at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor), `IcebergConnectorProvider` (ServiceLoader entry). - `IcebergScanPlanProvider` + `IcebergScanRange`: predicate/projection pushdown (`IcebergPredicateConverter` → Iceberg `Expression`), FileScanTask byte-offset sub-splitting, COUNT(\*) collapse, merge-on-read delete attachment (position delete / Puffin DV / equality delete), `IcebergColumnHandle` field-id column pruning, schema-evolution-safe reads (`IcebergSchemaUtils` field-id dictionary), synthetic rowid + v3 row-lineage metadata columns. - Write path: `IcebergWritePlanProvider` / `IcebergWriteContext` / `IcebergWriterHelper` / `IcebergConnectorTransaction` — INSERT / INSERT OVERWRITE (dynamic + static partition), branch-targeted writes, WRITE ORDERED BY, per-spec distribution, optimistic-concurrency conflict detection, one SDK transaction per statement, BE commit-fragment → DataFile/DeleteFile conversion. - Row-level DML: DELETE / UPDATE / MERGE INTO over position-delete (v2) and deletion-vector (v3) merge-on-read, baseSnapshot-anchored read. - `action/`: the 9 `ALTER TABLE EXECUTE` procedures (rollback_to_snapshot, rollback_to_timestamp, set_current_snapshot, cherrypick_snapshot, fast_forward, expire_snapshots, publish_changes, rewrite_manifests, rewrite_data_files) dispatched by `IcebergExecuteActionFactory`; `rewrite/` hosts the distributed bin-pack `rewrite_data_files` engine. - `IcebergSchemaBuilder` (CREATE TABLE), `IcebergTypeMapping` (both directions), `IcebergColumnChange` / `IcebergComplexTypeDiff` (schema + partition evolution), `IcebergLatestSnapshotCache` / `IcebergManifestCache` (per-catalog caches on the long-lived connector), `dlf/` (Aliyun DLF client). (`fe-connector-metastore-iceberg`) An Iceberg catalog resolves its backend by `MetaStoreProvider` ServiceLoader dispatch on `iceberg.catalog.type` (registered via `META-INF/services`), instead of a fe-core switch — **7 flavors**: REST, Hive Metastore, AWS Glue, Hadoop / filesystem, JDBC, Aliyun DLF, AWS S3Tables. Adding a backend = one provider class + one services line. The connector self-builds the HMS-metastore Kerberos authenticator, so metastore auth is owned entirely by the plugin. - `CatalogFactory` adds `iceberg` to `SPI_READY_TYPES` and drops the built-in case. - `GsonUtils` replay-compat: persisted `IcebergExternalCatalog` → `PluginDrivenExternalCatalog`, `IcebergExternalDatabase` → `PluginDrivenExternalDatabase`, `IcebergExternalTable` → `PluginDrivenMvccExternalTable` on deserialization, so existing clusters upgrade without losing catalogs (metadata backward-compat invariant). - Nereids/DDL/alter generalized off the concrete Iceberg types; SHOW CREATE renders LOCATION / PARTITION BY / ORDER BY / PROPERTIES via the generic path (iceberg engine name preserved, no credential leak). Per the 2026-07 architecture decision (fe-core holds **no** property parsing for migrated connectors; storage parsing → `fe-filesystem`, metastore parsing → `fe-connector`, both plugin-side; the plugin assembles → BE thrift → hands back to fe-core), iceberg's property/credential/auth resolution moves entirely into the connector: - Storage credentials bound to `fe-filesystem` on both scan and write paths (one source), retiring the second fe-core `StorageProperties.createAll` parse. - Vended temporary credentials, `warehouse` → `fs.defaultFS` derivation (hadoop flavor only, via the neutral `Connector.deriveStorageProperties` SPI), and Kerberos doAs are all connector-owned; the fe-core pre-execution authenticator is a no-op for plugin catalogs. - fe-core's shared static property/metastore helpers are kept alive only for the connectors still on the legacy path (Hive / Hudi / LakeSoul / HMS-iceberg). Net **−21.7k** lines from fe-core. Removes the 4 dead entity classes (`IcebergExternalTable` / `IcebergExternalDatabase` / `IcebergSysExternalTable` / `IcebergExternalCatalog`), the fe-core Iceberg `MetastoreProperties` cluster + its credential helpers, the 5 fe-core Iceberg connectivity probes and the coordinator branches, and un-registers the Iceberg property factories (fail-loud on any stray call). The now-dead paimon `MetastoreProperties` cluster left behind by P5 is swept in the same pass. `GsonUtils` retains only the three string compat labels for old-image upgrade. Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Classloader: TCCL pinned to the plugin classloader at all three loci where fe-core crosses into bundled Iceberg (scan call thread, write/DDL/commit engine thread via `TcclPinningConnectorContext`, and Iceberg's own manifest-writer worker pool), preventing a duplicate-copy `ClassCastException`. - Nested schema dictionary field names lowercased so a mixed-case nested struct (e.g. Iceberg `DROP_AND_ADD`) can't crash BE with `std::out_of_range`. - Session time zone: lazy `ZoneId` resolution tolerant of Doris CST/PST-style aliases in `FOR TIME AS OF` and datetime partition predicates. - Storage / URI normalization (`oss/cos/obs/s3a` → `s3`), SHOW CREATE secret redaction, and `ALTER TABLE EXECUTE ... WHERE` rejection kept at legacy wording. - ~68 FE unit test classes: iceberg connector (offline recording fakes for the remote Catalog / Table), metastore-iceberg per-flavor dispatch + properties, and the fe-core `PluginDriven*` MVCC / sys-table / scan-node / GSON-replay surfaces; many carry mutation checks. - checkstyle 0 violations; connector import-gate passes (no `fe-connector-iceberg → fe-core` import); FE main + test compile clean. - End-to-end iceberg behavior is docker-gated (`enableIcebergTest=true`); read, write, row-level DML, procedures, system tables, DDL and time-travel are covered by the `external_table_p0/iceberg` (+ `external_table_p2/iceberg`) regression suites. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etaspace leak JdbcConnectorClient cached JDBC-driver classloaders in a ref-counted map that evicted the entry when refCount reached 0. A JDBC driver self-registers into the static java.sql.DriverManager on class load, and close() never deregisters it, so an evicted classloader stays pinned by DriverManager for the life of the process. Serial CREATE -> DROP -> CREATE churn of the same driver URL therefore rebuilt a fresh, separately-pinned classloader every cycle, leaking one driver's worth of Metaspace per cycle. Live FE evidence (external regression build 986696 OOM): only 4 jdbc catalogs yet 38 FactoryURLClassLoader instances survived a full GC (jmap -histo:live), tracking the DriverInfo count -> ~34 leaked, DriverManager-pinned driver classloaders. FE Metaspace grew 165MB -> 1565MB over the single-concurrency 5.5h run (multi-concurrency did not OOM: overlapping same-URL catalogs keep refCount > 0 so the loader was reused instead of evicted-and-rebuilt). Fix: revert to a keep-alive Map<URL, ClassLoader> (one loader per distinct driver URL, shared across catalogs, never evicted), mirroring the pre-SPI fe-core JdbcClient and the pattern IcebergConnector / PaimonConnector already use. Bounded by the number of distinct driver jars. Adds a regression test asserting repeated create/recreate cycles for one URL add exactly one cached classloader. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XKf5EzHW75qzgrXnkoBZcF
…TableTest after rebase Rebase-integration fix, belongs with P4 (apache#64300 "remove legacy maxcompute subsystem"); squash into it when finalizing that PR. Upstream apache#64937 added MaxComputeExternalTableTest as a unit test for the legacy MaxComputeExternalTable.parsePartitionValues. P4 deletes that class, but the new test file rode into the rebase base with no modify/delete conflict and then dangled: fe-core test-compile failed with "cannot find symbol: MaxComputeExternalTable". No coverage lost: the new connector's MaxComputeConnectorMetadata.listPartitionValues already resolves partition values by name via ODPS PartitionSpec.get(column) (structurally immune to the out-of-order/invalid-spec bug apache#64937 fixed in the legacy path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…es into SPI connectors After rebasing branch-catalog-spi onto upstream master (4dfcb4f), an audit of the 3 upstream commits touching fe-core datasource/ found behaviour that was dropped when P5/P6 removed the legacy fe-core paimon/iceberg subsystems. Port the bounded correctness gaps into the new fe-connector SPI code (main + tests): - apache#65332 (paimon JNI IOManager): PaimonScanPlanProvider.getBackendPaimonOptions returned emptyMap() for non-jdbc catalogs, so the three JNI IOManager options (paimon.doris.enable_jni_io_manager / .tmp_dir / .impl_class) never reached BE and Paimon primary-key merge reads could not spill (OOM risk). Collect the three options (strip the "paimon." prefix) before the metastore-type check and forward them for all catalog flavours; jdbc-only driver logic stays in the jdbc branch. - apache#65094 tier1 (build-time correctness): IcebergSchemaBuilder buildPartitionSpec / buildSortOrder and PaimonSchemaBuilder primary/partition keys used bare column names, so a partition/sort/key referenced with a different case than the (now case-preserving) schema column failed the engine's case-sensitive lookup. Add case-insensitive resolution back to the canonical column name (iceberg via Schema.caseInsensitiveFindField; paimon via a lower->canonical map), mirroring upstream getIcebergColumnName / getPaimonColumnNames. - apache#65094 tier2 (consistency, worse than upstream): PaimonConnectorMetadata emitted "partition_columns" with case-preserving keys while column names are lowercased, so PluginDrivenExternalTable's case-sensitive byName lookup silently dropped mixed-case paimon partition columns (table treated as non-partitioned). Lower-case partition_columns with the same bare toLowerCase() the columns use. Verified: mvn package -pl :fe-connector-paimon -am (95 paimon tests) + iceberg 23 tests, 0 failures. Read-path column-name case preservation (tier3) and the apache#63068 OIDC session-credential port are tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…column-name case on the read path Align the iceberg + paimon catalog-SPI connectors with upstream apache#65094: external table top-level column names now keep their remote case on the read path instead of being lowercased. FE-only (BE untouched): the byte-match invariant "Doris scan slot name == TSchema top-level TField name" is preserved by flipping BOTH the scan slot names AND the field-id-dict / current-TSchema top-level names to case-preserving together, so BE (which matches top-level by field-id) needs no change. Nested struct children stay as-is (iceberg lowercase for BE StructNode.children.at; paimon paimon-cased). Reverts tier2's lowercasing of paimon partition_columns (bce28b2) now that the paimon columns are themselves case-preserving. iceberg (IcebergConnectorMetadata): partition_columns / getColumnHandles / parseSchema. iceberg (IcebergPartitionUtils): getIdentityPartitionColumns (path_partition_keys), getIdentityPartitionInfoMap (scan partition-value map) and generateRawPartition -- the listPartitions value-map key that PluginDrivenExternalTable.getNameToPartitionItems looks up by the case-preserved partition_columns remote name; omitting it would silently drop a mixed-case partition column's value in MTMV / partition pruning. paimon (PaimonConnectorMetadata): partition_columns / getColumnHandles / mapFields. paimon (PaimonScanPlanProvider): path_partition_keys / getPartitionInfoMap and the current (-1) TSchema entry (buildSchemaInfo lowercaseTopLevelNames -> false at the sole current-schema call site; the historical entries and nested names are unchanged). Tests: flip the connector tests that pinned mixed-case -> lowercase output to assert case preservation; add an IcebergSchemaUtils mixed-case byte-match case and an IcebergPartitionUtils.listPartitions regression pinning the case-preserved partition-value-map key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…l DTO + user-session capability Re-migrate the Iceberg REST OIDC per-user session-credential feature (upstream apache#63068 / e545f1a, dropped by the 2026-07-09 P6 rebase) onto the catalog SPI. This is the SPI layer (fe-connector-api): - ConnectorDelegatedCredential: neutral, immutable DTO carrying a user's per-connection OIDC/JWT/SAML token across the fe-core -> connector boundary so the connector never imports a fe-core type. The token is connection-scoped and in-memory only; toString() redacts it (never edit-logged / persisted / SHOWn). - ConnectorSession.getDelegatedCredential() (default empty) + getSessionId() (default = getQueryId): a connector reads the credential and the stable AuthSession key (preserved across FE observer->master forwarding) off the session. - ConnectorCapability.SUPPORTS_USER_SESSION: gates FE credential injection (least-privilege -- a connector that never consumes the token never receives it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…injection + per-user cache bypass FE bridge for the Iceberg REST per-user session feature (builds on the SPI DTO): - ConnectorSessionBuilder.from(ctx): copy the retained per-connection DelegatedCredential + sessionId onto the ConnectorSession ONLY when the target connector declares SUPPORTS_USER_SESSION (least-privilege); map fe-core DelegatedCredential -> neutral ConnectorDelegatedCredential by enum name so an added-but-unmapped type fails loud rather than being silently dropped. - PluginDrivenExternalCatalog.shouldBypassTableNameCache / shouldBypassDbNameCache = supportsUserSession() && ctx.hasDelegatedCredential(). Under session=user the REST server returns PER-USER metadata, so the shared (catalog+name-keyed, NOT user-keyed) db/table-name caches are bypassed to prevent cross-user leakage; a session with no credential keeps the shared cache (the fail-closed rejection then happens connector-side on the actual read, never by serving/poisoning a shared entry). - ExternalCatalog/ExternalDatabase: db-level live-path bypass (getFilteredDatabaseNames(false)/getDbNullableWithoutCache/matchesLocalDbName), still re-appending information_schema+mysql and never mutating the shared lowerCaseTo*Name lookup when bypassing (updateLookup=false). Tests: ConnectorSessionImplTest (capability-gated inject / non-capable skip / no-cred), PluginDrivenExternalCatalogSessionBypassTest (the bypass decision across capability + credential + null-context). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…outing (metadata + scan/write/procedure) Iceberg connector consumer for the per-user REST session feature: - IcebergConnectorProperties + IcebergRestMetaStoreProperties: iceberg.rest.session (none|user) + iceberg.rest.oauth2.delegated-token-mode (access_token|token_exchange) + optional session-timeout, with validation (session=user requires security.type=oauth2; relax the "oauth2 requires static token/credential" rule for user-session, whose identity is the per-request user token). - IcebergDelegatedCredentialUtils + IcebergSessionCatalogAdapter: bridge the neutral ConnectorDelegatedCredential to an iceberg SessionCatalog.SessionContext (access_token = verbatim OAuth2 bearer; token_exchange = the typed token-type key) over a SINGLE shared RESTSessionCatalog. delegatedCatalog/delegatedViewCatalog fail closed on a tokenless session=user request (never borrow a shared identity). - IcebergConnector: build a RESTSessionCatalog for a session=user REST catalog; newCatalogBackedOps(session) routes per-request metadata through the querying user's delegated catalog; declare SUPPORTS_USER_SESSION only for that config. - scan/write/procedure providers take a Function<ConnectorSession, IcebergCatalogOps> resolver (connector passes this::newCatalogBackedOps); the legacy ops constructors bind a constant s->ops so existing tests/behaviour stay byte-identical. This makes SELECT / INSERT-DELETE-MERGE / ALTER TABLE ... EXECUTE run per-user. The write COMMIT was already per-user (IcebergConnectorTransaction is opened by beginTransaction over the session-aware metadata ops). Tests: IcebergProviderSessionRoutingTest (each provider resolves ops with the call session + fail-closed), IcebergSessionCatalogAdapterTest (credential-key mapping per mode + fail-closed + fromString), IcebergConnectorValidatePropertiesTest (+session validation). IcebergConnectorTest/IcebergProcedureOpsTest adjusted for the resolver constructor (reflection reads catalogOpsResolver; a bare-null ctor arg is cast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…research notes + HANDOFF (Tasks 0-7 DONE) - Add the authoritative design doc (Trino-aligned, SPI-retargeted) and the source-verified research notes (Trino reference + apache#63068 as-built + current SPI). - HANDOFF: mark Tasks 0-7 DONE with the 3-commit stack (SPI 55c991d / FE 930e477 / connector a07216e), the clean-room review result (routing completeness = zero gaps; cache-leakage = none), and the two follow-ups (background log-noise reduction; two minor case/consistency nits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…essionContextTest (cross-user cache-leakage data-flow) Closes the one remaining apache#63068 test gap: the DATA-FLOW proof (not just the bypass decision, which PluginDrivenExternalCatalogSessionBypassTest already pins) that a session=user catalog serves per-user database metadata LIVE and never through the shared (catalog+name-keyed, not user-keyed) name cache -- the cross-user leakage guard (Trino CVE-2026-34214). Retargeted off the deleted IcebergRestExternalCatalog onto a PluginDrivenExternalCatalog test subclass: mockStatic(SessionContext.current()) drives the per-token identity, the catalog overrides the remote listing to return each user's own databases and record the listing token. Two disjoint per-user results + a live re-list on EVERY read (including a repeat of an earlier token) prove no user's database set leaks to another and nothing was served from a shared cache; information_schema+mysql stay visible. apache#63068 asserted this via a no-token "bootstrap" read, which on this branch fail-closes (a session=user catalog has no shared identity to bootstrap) -- the per-read live token record is the architecture-correct equivalent observable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…a-flow test + Task 0 retained-base verify Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KKXx3btbQc6w3MbZbEoRf
…er rebase to 4f48ccf Rebase branch-catalog-spi onto upstream-apache/master (10 upstream commits, 21 local commits replayed). Two files were touched by both sides; both are paimon + apache#65365 fallout: 1. Legacy fe-core PaimonScanNode.java: modify/delete conflict — upstream apache#65365 added "jni.enable_file_reader_async" to its BACKEND_PAIMON_OPTIONS forwarding list, while P5 (T29) deleted the whole legacy fe-core paimon subsystem. Resolution keeps the deletion and ports the new key to the connector-side mirror list PaimonScanPlanProvider.BACKEND_PAIMON_JNI_OPTIONS (+ intent test): the SPI path is the only forwarding path left, so without the port the new catalog property would silently never reach BE's PaimonJniScanner. 2. PaimonJniScanner.java: silent semantic conflict — apache#65365 removed the then-unused java.util.Collections import upstream, while this branch's null-predicate backstop uses Collections.emptyList(). git auto-merged the non-overlapping hunks without complaint, breaking compilation. Restore the import. Verified: fe-connector-paimon package GREEN (PaimonScanPlanProviderTest 65/0 incl. new backendOptionsForwardFileReaderAsyncOptOut); paimon-scanner Paimon* tests 11/0; full build.sh --fe BUILD SUCCESS (build cache disabled). Pre-existing local-env JVM crash in java-common JniScannerTest (StubRoutines::jbyte_disjoint_arraycopy_avx3, byte-identical module before and after rebase) is unrelated and left as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hade Under a parallel reactor (build.sh's default -T 1C), the package-phase maven-shade-plugin goal can start before this no-source module's main jar is created and attached, failing with "Failed to create shaded artifact, project main artifact does not exist". build.sh's single-threaded retry regex does not match this error string, so it surfaces as BUILD FAILURE instead of auto-retrying. Pin maven-jar-plugin's default-jar execution to the package phase with forceCreation=true, declared before the shade plugin, so the module's main artifact is guaranteed present before shade runs -- eliminating the race while keeping -T parallelism (no fallback to single-threaded). Verified: bash build.sh --fe --clean (default -T 1C, all 61 modules) BUILD SUCCESS, the race error did not recur, jar:jar (default-jar) runs before shade:shade, and the shaded-jar relocation output is unchanged (the diff is jar-plugin-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on_deletes connector gap as flip blocker Rebase: 23 upstream commits / 23 ours replayed, behind0/ahead23. All 5 conflicts came from a single upstream commit 0814e49 (apache#65135, iceberg $position_deletes system table), which lands in fe-core code P6 deleted/migrated to the connector SPI. Records the per-file adjudication, the two hazards git did not flag as conflicts (IcebergScanNodeTest hard-depends on a method the P6 side removes; the groovy suite's auto-merged out-of-hunk lines), and the resulting flip blocker: the iceberg connector does not expose position_deletes, so the feature is absent vs upstream master while its ~1200 lines of BE readers land orphaned. Two p0 suites are intentionally left red rather than editing the tests to hide the gap. Verified: FE_MAVEN_THREADS=1 -Dmaven.build.cache.enabled=false bash build.sh --fe = BUILD SUCCESS, 61 modules, 0 checkstyle violations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ design (T0 evidence in, ready to implement) Research (workflow wf_583cf18a-7ae, 5 lenses) + T0 empirical prerequisites (workflow wf_2f84859d-a52, 3 checks that actually compiled and ran code). T0 falsified two design assumptions, both now recorded: - Jackson vs Gson: Float/Double are byte-identical (the suspected risk did not reproduce); BigDecimal differs in BOTH directions (1.50 -> 1.5, and 10 -> 1E+1, a syntax change). Root cause is JsonNodeFactory bigDecimalExact=false under valueToTree, not the writer. Fix: a hoisted mapper copy with withExactBigDecimals(true) -- never mutate iceberg's shared JsonUtil.mapper(). - partition_data_json is NOT parsed as JSON: it is fed to Doris's STRUCT text serde, and DataTypeNullableSerDe::from_string swallows every parse error into a NULL partition while returning OK. A wrong shape is silent wrong data, not an error. The connector's existing renderer emits a JSON array and would hit this. Also found a blocker-shaped gap: [D-065] skips the schema-evolution dict for all sys handles on the rationale that the metadata-table schema rides inside the serialized FileScanTask -- true only for the JNI path. position_deletes is the first sys table reaching a native BE reader, which needs history_schema_info to resolve the `row` column (v1 hard-errors, v2 silently degrades to name matching). Design decisions: D1 smooth-upgrade guard NOT ported (user ruling: implement the final form, no compat) => fe-core and SPI both unchanged, the port is entirely connector-local. D2 third range shape on IcebergScanRange. D3/D5/D6 per T0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… design (T0 evidence in) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ble on the connector SPI Ports upstream apache#65135 (0814e49) onto this branch's connector SPI. Upstream implemented $position_deletes in the fe-core iceberg subsystem that P6 deleted, so after the rebase the BE readers, the thrift contract and the two regression suites were present while no FE ever produced the ranges -- the feature was absent vs master and its ~1200 lines of BE C++ were orphaned. fe-core and the SPI are UNCHANGED: the smooth-upgrade backend guard is deliberately not ported (user ruling: implement the final form), which was the only piece needing engine-side state. Everything lands inside fe-connector-iceberg. - IcebergConnectorMetadata: expose position_deletes (drop the two Q2-era filters). The exclusion mirrored legacy's "not supported yet"; apache#65135 invalidated that premise. - IcebergScanRange: a THIRD range shape. BE routes into its native reader only on top-level TIcebergFileDesc.content in {1,3} + a PARQUET/ORC range format, and asserts exactly one delete descriptor. The data path must therefore never set top-level content to 1 or 3 -- now a load-bearing invariant. - IcebergPartitionUtils: getPartitionDataObjectJson, a SECOND renderer. The existing one emits a JSON array of strings; this path needs a type-native object keyed by the metadata table's partition field NAMES resolved by iceberg field ID. Values ride a mapper copy with withExactBigDecimals(true): stock Jackson renders BigDecimal("10") as 1E+1, a JSON syntax change vs legacy Gson (verified empirically). - IcebergScanPlanProvider: the planning branch (newBatchScan -- PositionDeletesTable .newScan() throws -- pin, predicate, split sizing, PUFFIN-vs-parquet/orc, AVRO fails loud), and a narrowed [D-065] gate: every other sys table's schema rides inside the serialized FileScanTask, but this one reads natively and BE needs history_schema_info to resolve `row` (v1 hard-errors, v2 silently degrades to name matching). Two defects found by adversarial review and fixed here: the dict branch called resolveSysTable, which carries no auth wrap by contract and would have failed a kerberized catalog at plan time (now built from the already-loaded base table, also saving a round-trip); and the shared split-size heuristic summed fileSizeInBytes, which over-counts puffin DVs ~N-fold (now ScanTaskUtil.contentSizeInBytes, byte-equal for data files -- the comment claiming iceberg 1.10.1 lacks that method was false). Verified: connector suite 977/977 (+11 new), 0 checkstyle violations, and FE_MAVEN_THREADS=1 -Dmaven.build.cache.enabled=false bash build.sh --fe = BUILD SUCCESS, 61 modules. The two p0 regression suites are NOT run -- no local iceberg REST+MinIO+Spark env; they remain the flip gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…remaining flip gate Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refs apache#65185. Part of **P7 — hive (+HMS)** in the Catalog SPI migration tracking issue. This PR starts the hive/HMS connector migration on `branch-catalog-spi`, covering the HMS metadata path, metastore-event pipeline, ACID transaction write path, and the hudi live cutover dependency tracked in P7. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… a slim HMS metastore-client shade (apache#65721) ## Proposed changes Part of the catalog SPI migration umbrella: apache#65185. The `fe/fe-connector` closure depended on `org.apache.doris:hive-catalog-shade`, a ~122 MB "kitchen-sink" shade whose Hive metastore-client content is under 2% of its bytes (the rest is a full Hadoop, the paimon bundle, an ancient fastutil, DLF, etc.). This PR replaces it, for the fe-connector modules, with a new plugin-private slim shade `fe-connector-hms-hive-shade` (~15 MB) that bundles **only** the Hive metastore-**client** closure and relocates `org.apache.thrift` to `shade.doris.hive.org.apache.thrift`, mirroring the existing `fe-connector-paimon-hive-shade`. ### What changed - **New module `fe-connector-hms-hive-shade`**: hive-standalone-metastore (api + client), hive-common (`HiveConf`), hive-serde, hive-storage-api, iceberg-hive-metastore (`HiveCatalog`), libthrift/libfb303 0.9.3, codehaus jackson 1.9.2 (HMS JSON event messaging) and commons-lang 2.x (`HiveConf`). It uses an artifactSet `<includes>` **whitelist** so the hadoop-yarn/curator/jersey/kerby tail dragged in by hive-shims stays out; hadoop-common and guava/slf4j/log4j come from each plugin's own deps / the host at runtime. The thrift relocation **reuses** the existing `shade.doris.hive` prefix, so the vendored patch `HiveMetaStoreClient` in `fe-connector-hms` needs no source change. - **`fe-connector-hms`** now depends on the slim shade (compile, transitive to hive/hudi/iceberg) instead of the fat shade; the patch client stays in place. - **`fe-connector-iceberg`** drops its direct fat-shade dependency in the **same commit** (the hms-flavor `HiveCatalog` now arrives transitively via the slim shade), so iceberg is never left with two `HiveCatalog` copies; adds `iceberg-bundled-guava` at compile scope for the vendored `DeleteFileIndex`. - **`fe/pom.xml`** keeps the `hive-catalog-shade` version pin (still used by `be-java-extensions`) with an explanatory comment; stale comments in the iceberg/hudi poms are corrected. ### Size impact The shade artifact drops **122 MB → 15 MB** (to ~12% of its size). Each of the hive / hudi / iceberg plugins bundles its own copy, so each deployed plugin shrinks by **~107 MB** (~322 MB across the three). ### Verification - `fe-connector-hms` / `-hive` / `-hudi` / `-iceberg` build + unit tests green (197 test classes, 0 failures, checkstyle 0). - `dependency:tree` across all 19 fe-connector modules shows no `hive-catalog-shade`. - The three assembled plugin zips contain the slim shade once, no fat shade, no original-package libthrift, and exactly one copy each of the metastore-api `Table` / `HiveConf` / iceberg `HiveCatalog` / relocated thrift; the metastore and `iceberg.hive` bytecode is md5-identical to the fat shade. - **Remaining gate**: heterogeneous-HMS e2e (hive read/write, iceberg-on-HMS INSERT/DELETE/MERGE, hudi-on-HMS read; TCCL / filter-hook / kerberos paths) — to be exercised by CI / a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AkwFY4c7kZkLKZeazy3UMW --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, not latest
`t$audit_log@options('scan.tag-name'='top_cp0')` bound fine after de87be4 but then failed
at PLAN time with "Column old_name not found in table top_timeline$audit_log"
(FileQueryScanNode.setColumnPositionMapping:315). That commit closed two of the three consumers
of a sys table's pin and missed the third, so the failure moved one stage later rather than away:
External Regression 1007447 failed the same suite line during ANALYSIS ("Unknown column
'old_name' in 'table list' in PROJECT clause"), 1007887 fails it at the scan node.
A system table is not an MvccTable and BindRelation returns from handleMetaTable BEFORE
StatementContext.loadSnapshots, so MvccUtil.getSnapshotFromContext answers empty BY CONSTRUCTION
for a $-suffixed relation. Three places on the scan path need the pin, and each needs its own
fallback off the source table:
- pinMvccSnapshot -> resolveSysTableSnapshotPin (already had it)
- buildColumnHandles (de87be4 added it)
- resolvePinnedFullSchema (this commit)
Behind getPinnedFullSchema/getPinnedBaseSchema, resolvePinnedFullSchema still went through the
context lookup, got empty, and fell back to super -- i.e. the LATEST schema -- while the slots
were bound and the column handles built at the PINNED one. setColumnPositionMapping then indexed
the pinned slots into the latest column list.
The loud half is the reported one: `old_name`, renamed to `MixedName` after the tag, has no index
and the query dies. The silent half is worse and is why this is a correctness fix rather than a
message fix -- a column that merely MOVED or changed type maps to the wrong file column and BE
reads wrong data with nothing raised. In the failing suite `victim` is BIGINT at latest and
STRING at the tag.
Fix: route a PluginDrivenSysExternalTable through its own getFullSchemaAt(tableSnapshot,
scanParams) -- the exact method LogicalFileScan.computePluginDrivenOutput binds from -- so
binding, column handles and position mapping share the sys table's memoized pin and cannot
disagree. The instance is the same one for all three: generateTupleDesc carries the
LogicalFileScan's table onto the tuple descriptor, and the scan node's selectors are forwarded
from the same fileScan, so the memo key matches field for field. A reference with no pin (or a
connector that declines the selector) resolves empty and getFullSchemaAt returns getFullSchema(),
which is byte-for-byte what super.getPinnedFullSchema()/getPinnedBaseSchema() returned before --
getBaseSchema(false) is visibleColumns(getFullSchema()) and neither is overridden in the plugin
hierarchy. Both call sites already null-check, so the no-schema path is unchanged.
Tests: PluginDrivenScanNodePinnedSchemaTest gains sysTablePositionMapsAgainstItsOwnPinNotLatest
(the pinned schema reaches getPinnedFullSchema, and the MVCC-context route -- which answers empty
by construction -- is never consulted) and sysTablePinnedBaseSchemaDropsHiddenColumns
(getBaseSchema(false) parity on the pinned schema, since it feeds numOfColumnsFromFile). Removing
the new arm turns both red; restoring it turns both green. fe-core datasource.scan/plugin pin
suites 18/18.
Not yet verified end to end: test_paimon_schema_time_travel_matrix needs a docker run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ually emits
test_iceberg_on_hms_gateway_ddl_parity failed in External Regression 1007887 on the plain-hive
guard at line 195: it expected the refusal to contain "not supported", but got
"Nested column path is only supported for Iceberg tables: s.b".
The product behavior is correct. AlterTableCommand admits nested column paths only for a target
that reports SUPPORTS_NESTED_COLUMN_SCHEMA_CHANGE -- i.e. the gateway routes by TABLE capability,
not by catalog type -- and raises that message otherwise (AlterTableCommand:179, pinned by
AlterTableCommandTest:187). The expectation was master's wording ("Nested add column operation is
not supported for this table type."), a string that appears nowhere in this branch's tree, so the
assertion was vacuous-by-accident here and would have passed on any message containing those two
words.
It surfaced only now because the line had never run. The suite died at its first executable
assertion in all four earlier builds (1006034, 1006199, 1006647, 1007447); 31ae926 reworked
the parity probe into an environment-independent verdict, the probe passed for the first time in
1007887 -- "gateway: [REFUSED_BY_METASTORE], dedicated: [REFUSED_BY_METASTORE]", parity held --
and execution reached line 195 for the first time in the suite's history.
Also teach classifyOutcome the same message. It recognized only "not supported" as
REFUSED_BY_DORIS, and that is precisely the wording a gateway which stopped recognizing its
iceberg tables would NOT produce today -- it would produce "only supported for Iceberg tables"
and be filed as OTHER. Still red, but naming the metastore's absence rather than the regression
the suite exists to catch. Both wordings now classify as REFUSED_BY_DORIS, so the verdict stays
accurate whichever side of the rename the code is on.
Verified: brace/paren/triple-quote balance checked; the two edits are the only behavioral change,
the comment-parity half is untouched. Not yet run end to end -- lines 195+ still need a docker run
to execute for the first time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
run-thirdparties-docker.sh assumes a Linux host in three places, and each
one aborts the run on macOS:
- IP_HOST detection only knows `ip` and `hostname -I`, and macOS has
neither, so IP_HOST came out empty. Darwin now gets 127.0.0.1, which is
also the only correct answer there: the "host" the `network_mode: host`
services join is the Docker Desktop VM rather than the Mac, and Docker
Desktop forwards the Mac's loopback into it. A LAN address would leave
the VM and arrive where nothing listens.
- ensure_hosts_alias rewrote /etc/hosts unconditionally, so every run
needed sudo -- which on macOS has no passwordless default and no tty to
prompt on. It now builds the file it wants, compares, and escalates only
when the alias is missing or points somewhere else.
- mysql:5.7.36 publishes a manifest list holding linux/amd64 alone, so an
Apple Silicon host resolves no manifest and the pull fails. That takes
hive down with it, because the default JuiceFS metadata URI makes mysql
an implicit hive dependency. Pinned to linux/amd64 so the host emulates
the one image that exists.
Separately, the iceberg entrypoint linked /opt/spark/{bin,sbin}/* into
/usr/local/bin with a bare `ln -s`. A container's writable layer survives
a stop, so on `docker start` those links are still present, `ln` fails,
and `set -e` takes the entrypoint down with it -- spark-iceberg could only
ever be recreated, never restarted. Now `ln -sf`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ing it An iceberg table created with `COMMENT '...'` persisted no comment at all. The COMMENT clause and the PROPERTIES map are two DISTINCT fields on the create request -- CreateTableInfoToConnectorRequestConverter fills request.getComment() from the clause and request.getProperties() from PROPERTIES(...) -- and iceberg's createTable forwarded only the latter, the same gap legacy IcebergMetadataOps.performCreateTable had. Every reader downstream (getTableComment, the COMMENT clause of SHOW CREATE TABLE, information_schema.tables.TABLE_COMMENT, SHOW TABLE STATUS) therefore reported a table with no comment, through a dedicated iceberg catalog and through an HMS gateway catalog alike. Fold the clause into the `comment` table property in createTable. Resolution mirrors the fix already shipped for paimon (PaimonSchemaBuilder.build): an explicit properties["comment"] WINS, preserving the legacy persisted-comment behavior including a deliberately empty one (keyed on containsKey, not on emptiness), else the COMMENT clause is used. A blank clause writes nothing: nereids defaults CreateTableInfo.comment to "" when the clause is omitted, so stamping it unconditionally would add a noise `"comment" = ""` to the PROPERTIES of every iceberg table Doris creates. Also fixes two defects this unmasked in test_iceberg_on_hms_gateway_ddl_parity, whose comment half seeds the comment with a plain COMMENT clause and so had been asserting against a table that never carried one: - information_schema is PER-CATALOG: `information_schema.tables` resolves inside whichever catalog is current and lists only that catalog's tables, so the TABLE_CATALOG predicate can only narrow rows already scoped to the current catalog, never reach across to another one. The suite read both catalogs from a single session position, so the non-current one silently returned zero rows and failed as an NPE. Each read is now taken from inside its own catalog. - Added size() assertions on those reads, so a future regression to zero rows reports what actually went wrong instead of an NPE. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y cluster and the dead storage/AWS-credential surface (apache#66210) ### What problem does this PR solve? Issue Number: apache#65185 Problem Summary: Part of the catalog-SPI migration (apache#65185). Review round 21, targeting `branch-catalog-spi`. This round is one self-contained work line: finish the architectural goal "**`fe-core` holds no property parsing**" for `org.apache.doris.datasource.property`. Everything here is deletion of code that is unreachable on this branch — nothing moves and nothing is migrated, because every successor is already live connector-side. `fe-core` loses ~860 lines and gains none. The investigation, the per-package verdicts, the rejected alternatives and the rolling handoff are included under `plan-doc/fecore-property-cleanup/`. The two packages under `datasource/property/` were investigated together but got **different** verdicts, and are deliberately not treated as one job: | package | verdict | reason | | --- | --- | --- | | `metastore/` (4 files, 333 lines) + the orphaned `ConnectionProperties` | deleted outright | unreachable at runtime; its successor has been live for some time — nothing needs to move | | `common/` (2 files, 237 lines) | **stays in `fe-core`**, only its provably dead half is cut | it serves *internal* storage (cold-storage `StoragePolicy`, cloud `StorageVault`), not external datasources; every relocation target is dependency- or classloader-illegal | **1. The metastore property cluster is deleted, not migrated** `datasource/property/metastore/` (4 files) plus the `ConnectionProperties` base it orphans. The successor is already running: `fe-connector-metastore-api` `MetaStoreProperties`, `fe-connector-metastore-spi` `MetaStoreProviders.bind` and `Connector.deriveStorageProperties`. The cluster was unreachable. Its registry held only `TRINO_CONNECTOR`, whose factory returns a bare `MetastoreProperties` without ever calling `initNormalizeAndCheckProps` — so it parsed nothing, and `createInternal`, `getDerivedStorageProperties`, `getExecutionAuthenticator` and the `StorageAuthenticatorBridge` all had zero callers. `CatalogProperty` was the sole importer and offered two doors: `checkMetaStoreAndStorageProperties` (no callers repo-wide) and `resolveDerivedStorageDefaults`, whose metastore branch runs only when `pluginDerivedStorageDefaultsSupplier` is null — and `PluginDrivenExternalCatalog` installs that supplier unconditionally. Nothing here was Gson-persisted, so no image or replay compatibility is at stake (note `MetastoreProperties.Type` is unrelated to the persisted `InitCatalogLog.Type` / `InitDatabaseLog.Type` constants of the same name). `resolveDerivedStorageDefaults` now **throws** when the supplier is unwired instead of deriving nothing. That preserves today's behavior, where the metastore parse threw on this path. Deriving an empty map would silently drop the iceberg `warehouse` → `fs.defaultFS` bridge and, because the setter deliberately does not reset caches, would cache the under-derived `StorageBindings` for good. `CatalogPropertyPluginStorageDerivationTest` gains a case pinning that, and its stale mutation comments are retargeted at mutations that still exist. **2. The callerless AWS provider-instance arm is dropped** `StorageAdapter.getAwsCredentialsProvider()` and its two helpers build a live AWS SDK v2 credentials object for FE-side SDK clients; `AwsCredentialsProviderFactory.createV2`, `createDefaultV2` and the single-arg `getV2ClassName` existed only to serve that arm. Both consumers — the catalog connectivity testers and `IcebergAwsClientCredentialsProperties` — were removed earlier in this migration along with the whole `datasource/connectivity` package, so nothing calls them here anymore. **This one is a deliberate trade, not merely dead code.** It is still live on `apache/doris` master, where both consumers exist; it reads as dead only on this branch. Removing it means future upstream edits to that region will surface as modify/delete conflicts on rebase, to be resolved by keeping the deletion. The investigation first flipped its own recommendation to *do not delete* for exactly that reason (see the `[doc](catalog) record that the dead AWS provider arm is live upstream` commit); the owner then accepted the trade, and only after that was the deletion carried out. Deliberately **kept**, because they are not part of that arm: `getAwsCredentialsProviderMode()` and the `s3CredentialsMode` field, and `AwsCredentialsProviderFactory.getV2ClassName(mode, boolean)` with its two env probes. Those emit the provider class-name string that reaches BE via `AWS_CREDENTIALS_PROVIDER_TYPE` and the hadoop `fs.s3a.aws.credentials.provider` map, and `AzureGuessRoutingParityTest` pins the mode accessor. Folding them into the look-alike `fe-filesystem-s3-base` twin was considered and **rejected**: the two implementations differ in two live behaviors (the hadoop string gains a `ProfileCredentialsProvider`; the accepted mode strings widen), and no test in the repo pins the emitted string — so the swap would have shipped a regression green. Three connector comments naming the deleted methods are reworded; the iceberg twin is now the only implementation of the provider-instance mapping. **3. The dead storage doors on the catalog base are removed** Five callerless members of `ExternalCatalog` — `getHadoopProperties`, `getConfiguration`, `buildConf`, `buildHadoopConfiguration`, `ifNotSetFallbackToSimpleAuth` — together with the `cachedConf` / `confLock` fields they cached through, and three of `CatalogProperty` — `getHadoopProperties`, `getBackendStorageProperties`, `getOrderedStorageAdapters` — with their two lazily populated fields. `getConfiguration` was already marked deprecated "until connector SPI extraction is complete", which it now is. The point is not the 135 lines. `initStorageAdapters` is now reachable only through `getStorageAdaptersMap` and `getEffectiveRawStorageProperties`, both called from `PluginDrivenExternalCatalog`, so "**`fe-core` storage has a single entrance**" holds by construction rather than by audit — which is also what backs the fail-loud branch in item 1. **4. Two test-infrastructure notes (no code)** `plan-doc/fe-core-ut-runtime-problem.md` records, with measurements rather than guesswork, why a full `fe-core` run takes 3h+: `reuseForks=false` (deliberate, to avoid singleton conflicts) means a fresh JVM per class at ~5.5s startup over 1225 classes, `forkCount` comes from `fe_ut_parallel` which defaults to 1 unless `FE_UT_PARALLEL` is set, and nereids accounts for 72% of test execution time. It also records what the FE UT pipeline actually runs — TeamCity `Doris_Doris_FeUt` invokes the repo's own `run-fe-ut.sh --coverage`, passes `-Dmaven.test.failure.ignore=true`, and forces the exit code back to 0 when a run has more than 6000 tests with at most 10 failures and 10 errors, so a green FeUt build is not the same as zero failures. Both notes are directions only, filed for a separate branch — raising `FE_UT_PARALLEL` is untested and risks fork contention over ports, BDB directories and temp paths. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> Local verification, stated at the coverage it actually has: - repo-wide grep for every deleted symbol: clean. - full reactor **clean** `test-compile` **including test sources** (no `-Dmaven.test.skip`), after each of the four deletions: BUILD SUCCESS. Deletion changes cannot be trusted to incremental compilation, so `fe-core/target/{classes,test-classes}` was removed before each run. - `fe-core` `checkstyle:check`: 0 violations. - targeted suites: 95 `fe-core` tests including the Gson replay suites; 110 `fe-connector-api` tests with `connector-metadata-methods.txt` unchanged; 52 storage-adapter and catalog-property tests (1 pre-existing `@Disabled` in `LocationPathTest`). - **the full `fe-core` suite was not run to completion.** It was still going at 3h29m after 1232 test classes and was stopped by the owner rather than finished, so the claim is "no failures among the 1232 classes executed", **not** "the suite passes". Its one failure, `ForwardToMasterTest.testAddBeDropBe` (`ClassCastException: JSONObject → JSONArray`), reproduces identically on a clean HEAD with these changes stashed, so it is pre-existing. - **no e2e was run** (no cluster available here); this PR adds no regression case. All four deletions remove unreachable code, and the storage-binding path they sit next to (iceberg hadoop `warehouse` → `fs.defaultFS`) is covered by unit tests only. CI is relied on for the rest. - no BE file is touched by this PR, so BE was not rebuilt. - Behavior changed: - [x] No. Every deleted member is callerless on this branch. The one semantic change — `resolveDerivedStorageDefaults` throwing when `pluginDerivedStorageDefaultsSupplier` is unwired — is not reachable in production wiring, since `PluginDrivenExternalCatalog` installs that supplier unconditionally, and it replaces a path that already threw. The one accepted forward cost is stated in item 2: modify/delete rebase conflicts against upstream master in the `StorageAdapter` region. - Does this need documentation? - [x] No. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gin families The connector family already carries an apiVersion() gate, but it can never reject anything: no ConnectorProvider overrides the default method, and the SPI interface is excluded from every plugin zip and loaded parent-first, so the default body executing at runtime is the kernel's own. The number never leaves the kernel. The other three families have no version check at all. Record the agreed design: the version travels in the plugin jar MANIFEST, derived from a single maven property per family that also feeds a filtered resource in that family's SPI module, so the kernel's expectation and the plugin's declaration cannot drift within a build. Major must match; minor and patch are ignored, which is safe because every SPI surface change is a major change by definition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dently The version model already gave each family its own maven property, but the spec buried that in a table and only mentioned the shared-artifact coupling in passing. State it directly: changing fe-connector-api bumps CONNECTOR alone, and the filesystem, authentication and lineage plugins need no rebuild. Add the matrix of which artifact change bumps which families, and record why the two remaining couplings cannot be designed away: fe-extension-spi is mandatory parent-first for all four families, and the hive, iceberg and paimon connectors genuinely import fe-filesystem-api types. A separate version number for those artifacts would describe the same coupling without removing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n design The design said what the version model is but not what anyone does when the SPI actually changes. Add a runbook: how to tell major from minor, the exact Doris-side steps for each, what in-tree plugins need (nothing - they inherit the property from the family parent pom, so the number appears in exactly one place), what a third-party plugin author has to write, and how to diagnose a plugin that failed to load. Note the two ways to read the kernel's expected version, and that declaring too low a version fails safe while declaring too high does not - the gate only guarantees that writing nothing cannot pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sion gate Eight tasks, each ending in an independently testable deliverable: the ApiVersion value type, the family-neutral gate, the loader wiring, then one task per plugin family, then the contract-surface baselines. The pom edits are written against what each file actually contains rather than conditionally - fe-connector-spi and fe-filesystem-spi have a build section without resources, both authentication poms have no build section at all, and fe-core already has a resources block that must be replaced wholesale so the antlr generated-sources entry survives. Task 8 includes a mutation step that adds a throwaway default method to ConnectorProvider and asserts the baseline goes red, because those baselines are the only guard that an SPI surface change gets a major bump. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eaves the kernel
`ConnectorProvider.apiVersion()` could never reject anything. The SPI interface is
listed parent-first, every plugin zip excludes the SPI jar, and no shipped connector
overrides the method -- so the `default` body executed on a plugin's behalf is the
kernel's own code, and the number it returns is the kernel talking to itself. Raising
`CURRENT_API_VERSION` and the default together admits every stale plugin; raising only
the constant rejects every plugin including freshly built ones. Neither is a check.
The other three plugin families had none at all.
Replace it with one number that is physically distributed to both sides of the
comparison. Each family declares its API version once, as a maven property in its
parent pom, and that single property flows to:
- a maven-filtered resource inside the family's SPI module -- what the kernel expects;
- the `<manifestEntries>` of every jar built from that pom -- what a plugin declares.
Same build, same property, so the two can never disagree by accident; across builds
they disagree exactly when they should. There is no second number, so no test has to
keep them in sync.
`ApiVersionGate` (fe-extension-loader, family-neutral) compares MAJOR only. Minor and
patch are ignored in both directions, which is sound only because "major" is defined as
ANY change to the SPI surface, additions included -- a compatible minor can never change
the set of API elements a plugin can reach. A jar that declares nothing is refused:
fail-closed is what makes the check meaningful for plugins built before this existed.
Directory loading is the only production path, and classpath ServiceLoader builtins
never pass through it, so they are exempt structurally rather than by a flag.
`DirectoryPluginRuntimeManager#loadAll` takes the gate as a mandatory fifth parameter.
Not an optional one: an unwired gate would be the same silently-true check this commit
removes. It runs after `loadClass` but before `asSubclass`/`newInstance`, so an
incompatible plugin's constructor never executes and the operator gets a message naming
both versions instead of a `ClassCastException`.
All four families are wired: CONNECTOR, FILESYSTEM, LINEAGE (start-up load, one bad
plugin still cannot stop FE) and AUTHENTICATION. Authentication loads lazily and reports
"no factory for this type" from a different place than the load, so the rejection reason
is carried into that exception at both call sites -- otherwise a refused plugin is
indistinguishable from one that was never installed.
Four surface baselines freeze what each family's plugins implement or call, including
`fe-extension-spi`'s `Plugin`/`PluginFactory`/`PluginContext` in all four, so a change
there turns all four red at once. Signatures are recorded with their return type: a
changed return type is a major change by the same definition, and the older
name-and-parameters-only baseline cannot see it. No test can prove someone actually
bumped the property -- a test sees the current state, never the delta -- so the failure
message says, in the same commit as the diff, that this is a major change.
Two build facts worth recording, both found by trying rather than reasoning:
- maven-build-cache hashes a module from its sources, dependencies and
`<build><plugins>`, NOT from `<properties>`, and the filtered resource's source text
is the literal placeholder. AUTHENTICATION and LINEAGE were going to ship without
`<manifestEntries>` because they have no in-tree plugin zips -- which would have put
their version outside every hashed input. Bumping 1.0 to 2.0 then produced an
identical checksum and a cached jar still declaring 1.0. All four families now carry
the block.
- `fe-connector-es` and `-trino` never excluded `fe-filesystem-api` from their plugin
zips, shipping a duplicate that parent-first loading guarantees is never read. Now
excluded, so all eight connector zips agree.
The four recorded baselines cannot carry a license header -- every non-blank line is
read back as a signature -- so they join `connector-metadata-methods.txt` in
`.licenserc.yaml`.
Verified: full reactor `package` with the build cache disabled and test sources
compiled; checkstyle clean on all six touched modules; the gate's decision rule, the
loader's behaviour on real plugin jars, and each family's wiring covered by tests; and
the produced artifacts read back -- `doris-fe-connector-es.jar` carries
`Doris-Connector-Plugin-Api-Version: 1.0` alongside the inherited `Implementation-*`
entries, and `doris-fe-connector-spi.jar` carries `api.version=1.0` rather than an
unfiltered placeholder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman
requested review from
924060929,
CalvinKirs,
Gabriel39,
dataroaring,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde and
starocean999
as code owners
July 29, 2026 05:07
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29953 ms |
Contributor
TPC-DS: Total hot run time: 176514 ms |
Contributor
ClickBench: Total hot run time: 25.43 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Only for testing #66211