[Draft] Master catalog spi review 18 - #65930
Closed
morningman wants to merge 74 commits into
Closed
Conversation
morningman
requested review from
924060929,
CalvinKirs,
Gabriel39,
dataroaring,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde and
starocean999
as code owners
July 22, 2026 23:31
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29627 ms |
Contributor
TPC-DS: Total hot run time: 176599 ms |
Contributor
ClickBench: Total hot run time: 25.51 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
morningman
force-pushed
the
master-catalog-spi-review-18
branch
from
July 23, 2026 05:46
0dbe2ed to
799cb12
Compare
This multi-month refactor needs persistent state for progress, decisions, risks, and cross-session agent handoff. Establishes a file-based tracking system including dashboard, ADR decision log, deviation log, risk register, per-stage task files, per-connector tracking, and an agent collaboration playbook covering context budget / subagent usage / handoff norms. Closes 18 design decisions (D-001..D-018) and registers 14 risks (R-001..R-014). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…T27) (apache#63582) ## Summary Lands the P0 SPI baseline for the catalog-SPI migration (master plan §3.1 / RFC §2.1), with zero impact on the already-migrated JDBC + ES connectors. - **Batch 0** (commits 1-2): SPI types + fe-core bridges — `ConnectorMetaInvalidator`, `ConnectorTransaction`, `ConnectorMvccSnapshot`, `ExternalMetaCacheInvalidator`, `ConnectorMvccSnapshotAdapter`, `PluginDrivenTransactionManager` generalization. - **Batch 1** (commit 3): DDL + Partition SPI — `ConnectorCreateTableRequest` + 4 spec POJOs, 4 new defaults on `ConnectorTableOps`, 3 new fields on `ConnectorPartitionInfo`, fe-core converter, `PluginDrivenExternalCatalog.createTable` routing. - **Batch 2** (commit 4): Import-gate + unit tests — `tools/check-connector-imports.sh` wired through exec-maven-plugin; `FakeConnectorPlugin` covering every default fall-through; routing tests for the invalidator; converter tests for all 4 partition styles + 2 bucket flavors. ## Commits - `[feat](connector) add P0 batch 0 SPI baseline: MetaInvalidator / Transaction / MvccSnapshot` (T03-T08) - `[feat](connector) wire P0 batch 0 SPI into fe-core` (T09-T12) - `[feat](connector) add P0 batch 1 SPI: CreateTableRequest + listPartitions` (T13-T20) - `[feat](connector) add P0 batch 2 gate + unit tests` (T21-T23, T26-T27) ## Test plan - [x] `mvn -pl fe-connector/fe-connector-api,fe-connector/fe-connector-spi -am compile` — SPI modules compile - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` — fe-core compile - [x] `mvn -pl fe-core checkstyle:check` — 0 violations - [x] `mvn -pl fe-connector validate` — import gate runs and passes (baseline clean) - [x] `mvn -pl fe-core -am test -Dtest='FakeConnectorPluginTest,ExternalMetaCacheInvalidatorTest,CreateTableInfoToConnectorRequestConverterTest,ConnectorPluginManagerTest,ConnectorSessionImplTest'` — 39/39 green - [x] `mvn -pl fe-connector/fe-connector-jdbc,fe-connector/fe-connector-es -am compile` — downstream connectors compile unchanged - [ ] JDBC regression-test suite (T24) — to be exercised by this PR's CI pipeline - [ ] ES regression-test suite (T25) — to be exercised by this PR's CI pipeline ## Tracking Full plan, decisions, and risk log live under `plan-doc/` in the repo (introduced by 6315983, already on the base branch). Per-task status: `plan-doc/tasks/P0-spi-foundation.md`. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…pache#63641) P1 batch A — close out scan-node SPI consolidation while keeping migration-period fallbacks in place. Three surgical changes route `PluginDrivenExternalTable` first in the nereids translator hot paths so already-migrated SPI connectors (JDBC, ES) take the SPI route, while the existing `instanceof XExternalTable` chains remain as fallbacks for connectors still pending migration (P3–P7). - **T3** — `PhysicalPlanTranslator.visitPhysicalFileScan`: move the existing `PluginDrivenExternalTable` branch from position 8 to position 1; the 7 connector-specific branches (HMS / Iceberg / Paimon / Trino / MaxCompute / LakeSoul / RemoteDoris) stay in place as migration-period fallbacks - **T4** — `PhysicalPlanTranslator.visitPhysicalHudiScan`: add a `PluginDrivenExternalTable` branch routed to `PluginDrivenScanNode.create(...)`, threading `tableSnapshot` + `scanParams` through `FileQueryScanNode` setters; `incrementalRelation` flagged as a P3 Hudi SPI extension TODO. The new branch is unreachable today (`PhysicalHudiScan` is only built for `HMSExternalTable + DLAType.HUDI`), so this is groundwork for P3 with zero current-day runtime impact - **T5** — `LogicalFileScan`: in `computeOutput()`, add a `PluginDrivenExternalTable` branch calling new helper `computePluginDrivenOutput()` — same shape as `computeIcebergOutput`, using `getFullSchema()` + virtualColumns; in `supportPruneNestedColumn()`, add an explicit `PluginDrivenExternalTable → false` branch. Both behaviorally equivalent for JDBC/ES today since they have no hidden cols and no virtualColumns P1 batch B (T1 — delete 13 legacy `Jdbc*Client` + `JdbcFieldSchema`) is deferred to P8 because the 3 fe-core callers — `PostgresResourceValidator`, `StreamingJobUtils`, `CdcStreamTableValuedFunction` — are live CDC streaming code that requires SPI extension for `getPrimaryKeys` / `getColumnsFromJdbc` / `listTables`, which is out of P1 surgical scope. Background and tracking docs live in `plan-doc/` (Master Plan §3.2 P1, tasks/P1-scan-node-cleanup.md, decisions log). - [x] `mvn -pl fe-core -am compile -Dmaven.build.cache.enabled=false` → BUILD SUCCESS - [x] `mvn -pl fe-core checkstyle:check` → 0 violations - [x] JDBC + ES regression-test passing — baseline established in P0 / PR apache#63582 - [ ] PR CI green on this PR - [ ] Manual scan-node smoke for an SPI connector — JDBC `SELECT *` should fall into the new `PluginDrivenExternalTable` branch first 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…apache#64096) ### What problem does this PR solve? Related PR: apache#63582 (P0 — SPI baseline), apache#63641 (P1 — nereids plugin-driven routing) Problem Summary: This is **P2** of the catalog SPI migration and targets the `branch-catalog-spi` feature branch (continuing P0 apache#63582 and P1 apache#63641). It fully migrates `trino-connector` off the legacy in-tree `fe-core/datasource/trinoconnector/` implementation and onto the connector SPI module `fe-connector-trino`, making `trino-connector` the first connector to complete the SPI consumption playbook that later connectors will reuse as a template. All five batches land together so there is no intermediate state where a newly-created trino catalog cannot be serialized. **Batch A — complete the SPI surface (`fe-connector-trino` only, no fe-core changes)** - `TrinoConnectorProvider.validateProperties`: enforce the required `trino.connector.name` property at `CREATE CATALOG` time (ported from the legacy `checkProperties`). - `TrinoDorisConnector.preCreateValidation`: call `ensureInitialized()` so plugin loading + connector-factory resolution happen at catalog creation instead of being deferred to the first `SELECT`. - `TrinoConnectorDorisMetadata.applyFilter` / `applyProjection`: bridge Trino native filter/projection pushdown, reusing `TrinoPredicateConverter` to translate a Doris `ConnectorExpression` into a Trino `TupleDomain`. `remainingFilter` is conservatively returned as the original expression to match legacy behavior (conjuncts are not stripped; BE re-evaluates them). **Batch B — fe-core bridge for image compatibility** - `GsonUtils`: atomically replace the three legacy `registerSubtype` entries (`TrinoConnectorExternalCatalog` / `Database` / `Table`) with `registerCompatibleSubtype` redirects onto the `PluginDrivenExternal*` hierarchy. This must be atomic — `RuntimeTypeAdapterFactory` rejects duplicate labels, so keeping both bindings would throw at static init. Mirrors what ES/JDBC already did. - `PluginDrivenExternalCatalog.gsonPostProcess`: extract a `legacyLogTypeToCatalogType()` helper that maps `Type.TRINO_CONNECTOR` → `"trino-connector"`; the generic `name().toLowerCase()` would otherwise produce the wrong `"trino_connector"` (underscore) that `CatalogFactory` does not recognize. - `PluginDrivenExternalTable.getEngine()` / `getEngineTableTypeName()`: add `trino-connector` branches that preserve the legacy engine-name / table-type display across `SHOW TABLE STATUS` and `information_schema`. **Batch C — flip the switch** - Add `"trino-connector"` to `CatalogFactory.SPI_READY_TYPES` so catalog creation routes through the SPI path. **Batch D — remove legacy code** - Drop the `instanceof TrinoConnectorExternalTable` scan branch in `PhysicalPlanTranslator` (the `PluginDrivenExternalTable` SPI branch already handles it). - Drop `case "trino-connector"` in `CatalogFactory`. - Delete `fe-core/datasource/trinoconnector/` (10 files) and the now-dead legacy `TrinoConnectorPredicateTest`. - Route the `TRINO_CONNECTOR` db-build case in `ExternalCatalog` to `PluginDrivenExternalDatabase` (mirrors the migrated JDBC case). - **Retained for image compatibility**: the `InitCatalogLog.Type.TRINO_CONNECTOR` and `TableType.TRINO_CONNECTOR_EXTERNAL_TABLE` enums, the GsonUtils redirects, and the `MetastoreProperties` trino-connector entry. **Batch E — tests + tracking docs** - 29 JUnit 5 unit tests over the plugin-free converters: - `TrinoPredicateConverterTest` — `ConnectorExpression` pushdown trees → Trino `TupleDomain` (EQ / range / NE / IN / IS [NOT] NULL / AND / OR, Slice encoding), plus graceful degradation to `TupleDomain.all()` on null/unsupported input. - `TrinoTypeMappingTest` — Trino SPI type → Doris `ConnectorType` (scalars, decimal precision/scale, timestamp precision clamp, array/map/struct, unsupported-type failure). - `TrinoConnectorProviderTest` — `validateProperties` fast-fails when `trino.connector.name` is missing/empty. - No Trino plugin/cluster required; plugin-dependent paths remain covered by the existing `external_table_p0/p2` `trino_connector` regression suites. - Sync the migration tracking docs under `plan-doc/` (already carried on this feature branch since P0). **Net effect**: 28 files, +1025 / −2681 (~1656 LOC net removed). Old FE images holding legacy trino catalogs / databases / tables deserialize onto the `PluginDrivenExternal*` hierarchy through the GsonUtils string-name redirect, with engine-name display preserved. **Deferred (follow-ups, not in this PR)**: - `trino_connector_migration_compat` regression test (old-image deserialization) — requires a running cluster + Trino plugin + docker, unavailable in this dev environment; tracked as a CI/cluster follow-up. - The plugin-install documentation update lives in the `doris-website` repo and is handled separately. ### Release note None ### Check List (For Author) - Test - [x] Unit Test — 29 new tests in `fe-connector-trino` (predicate converter / type mapping / property validation). - [ ] Regression test — existing `trino_connector` suites cover plugin paths; the new old-image compat regression is deferred to a CI/cluster follow-up. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. Internal routing moves from the legacy fe-core path to the SPI path; image compatibility, engine-name display, and pushdown semantics all mirror the legacy behavior. All batches land together, so there is no serialization-gap window. - Does this need documentation? - [x] Yes. The trino-connector plugin-install doc update is a follow-up in the `doris-website` repo. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
…tch design (hybrid, T02-T08) (apache#64143) ## Proposed changes testing with apache#64146 P3 of the catalog-SPI migration (base: `branch-catalog-spi`). Migrates the **hudi** connector following the **hybrid** strategy (D-019): harden the dormant HMS-over-SPI hudi connector to correctness parity, build a test baseline, and write the per-table dispatch design — **all behind the closed gate** (`SPI_READY_TYPES` unchanged). >⚠️ **No user-visible behavior change.** The SPI hudi path stays dormant (gate closed); hudi queries continue to use the legacy `HMSExternalTable.dlaType=HUDI` path. This PR removes correctness blockers ahead of the live cutover (deferred to P7 / batch E). ### What's included **Correctness fixes (hardening dormant code, behind gate):** - **T02** — fix hudi JNI `column_types` double bug: emit full Hive type strings (was Doris bare type names, losing precision/scale/subtypes) and send `column_names`/`column_types`/`delta_logs` as typed lists end-to-end (was comma join/split, which shattered `decimal(10,2)` / `struct<...>`). Matches the BE `hudi_jni_reader.cpp` contract (names `,` / types `#` / delta `,`). - **T04** — fail loud on time-travel / incremental read in the SPI `visitPhysicalHudiScan` branch (was silently returning the latest snapshot / silently full-scanning). - **T05** — real EQ/IN partition pruning in `HudiConnectorMetadata.applyFilter` (was a placeholder that ignored predicates and unconditionally switched the partition source from Hudi-metadata to HMS); faithfully mirrors `HiveConnectorMetadata.applyFilter`. - **T07** — column-name casing fix in `avroSchemaToColumns` (top-level lowercase, mirroring legacy `HMSExternalTable`). **Test baseline (all three connector modules started P3 with 0 tests):** - `fe-connector-hudi` (33): type-mapping / schema-parity (COW/MOR golden) / table-type / partition-pruning / scan-range. - `fe-connector-hms` (12): shared Hive-type-string parser tests. - `fe-connector-hive` (14): file-format / partition-pruning (mirrors T05). - COW/MOR schema is **type-agnostic** (golden parity vs legacy `initHudiSchema`); table type only affects scan planning. **Decisions / design (code-grounded, design-only):** - **T03** — defer `schema_id`/`history_schema_info` field-id evolution to batch E (DV-006; not a model-agnostic SPI fix). - **T06** — keep MVCC/snapshot SPI defaults (opt-out) + document (DV-007). - **T08** — `tableFormatType` dispatch design memo + **D-020**: single `hms` catalog per-table routing via a new backward-compatible `ConnectorMetadata.getScanPlanProvider(handle)` (per-table provider seam); refines D-005. The keystone gap is split into M1 (identity consumption, fe-core reads `tableFormatType` as an opaque string) and M2 (scan routing). ### Deferred to batch E / P7 (not in this PR) Gate flip (`SPI_READY_TYPES += hms/hudi`), fe-core `tableFormatType` consumption (M1+M2 implementation), live cutover, delete legacy `datasource/hudi/`, full incremental/time-travel/MVCC, Iceberg-on-hms via SPI (needs P6 `IcebergScanPlanProvider`), cluster/runtime validation. ### Verification Per task tracking, each code batch landed with: per-module compile + checkstyle 0 (incl. test sources) + connector import-gate pass + new unit tests green. The two most recent commits are docs-only (`plan-doc/`); the code is unchanged since the last green batch. Gate stays closed → the dormant SPI path is unreachable at runtime → zero live-path risk. CI re-verifies. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…core + make fe-core odps-free (T07-T09) (apache#64300) Follow-up to apache#64253 (the MaxCompute catalog-SPI cutover). After the cutover a `max_compute` catalog deserializes to `PluginDrivenExternalCatalog` and no legacy `MaxComputeExternal*` object is ever instantiated, so the legacy MaxCompute subsystem in fe-core is dead code. This removes it and makes fe-core's dependency tree fully odps-free. **1. Remove legacy subsystem** (`7a4db351100`) - Delete 20 fe-core files: `datasource/maxcompute/*` (incl. `MCTransaction`, `MaxComputeScanNode`/`Split`), the MaxCompute sink/insert/txn plumbing, and 2 legacy-only tests. - Clean ~21 reverse-reference sites (imports + dead `instanceof`/visitor/rule branches), keeping every `PluginDriven`/connector sibling branch and the image/replay keep-set (GsonUtils compat strings; `TableType`/`TransactionType`/`TableFormatType`/`InitCatalogLog.Type` `MAX_COMPUTE` enums; block-id thrift). - Rewire 3 tests; e.g. `FrontendServiceImplTest`'s block-id RPC test now mocks the generic `Transaction` SPI, since `getMaxComputeBlockIdRange` reads the PluginDriven connector transaction. **2. Make fe-core odps-free** (`409300a75b8`) - Drop the two odps deps from `fe-core/pom.xml`. - Move `MCUtils` from fe-common into `be-java-extensions/max-compute-connector` (its only consumer after the removal); keep `MCProperties` (odps-free constants) in fe-common. - Drop `odps-sdk-core` from fe-common — it was also leaking netty/protobuf transitively to fe-common's own `DorisHttpException`/`GsonUtilsBase`, so declare `netty-all` + `protobuf-java` directly (proper dependency hygiene). **3. Doc-sync** (`f8c305765e8`) — plan-doc PROGRESS/HANDOFF/deviations/design tracking notes. - `mvn -pl :fe-core -am test-compile` (main+test) passes; checkstyle 0 violations; connector import-gate passes. - `grep -rn com.aliyun.odps fe/fe-core/src` → empty. - `mvn -pl :fe-core dependency:tree | grep odps` → empty (no odps, direct or transitive). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he#64446) (apache#64446) Migrates the legacy in-tree Paimon catalog (fe-core `datasource/paimon`, `metacache/paimon`, `property/metastore/*Paimon*` and their reverse references) onto the catalog SPI as a self-contained `fe-connector-paimon` plugin, following the maxcompute full-adopter + cutover template. `paimon` is added to `SPI_READY_TYPES`, so a Paimon catalog now deserializes to `PluginDrivenExternalCatalog` and all five functional areas — normal-table read, system-table read, DDL, MTMV/MVCC/time-travel (procedures are a doc-only no-op: fe-core has none) — flow through the generic `PluginDriven*` bridge with no Paimon-specific branch in fe-core. Paimon is the first lakehouse full-adopter to exercise the MVCC / sys-table / MTMV SPI surfaces, which become the reusable template for the upcoming iceberg/hudi migrations, so the cutover is held to byte-parity with the legacy path. The legacy `datasource/paimon/*` classes are left in place (now dead on the cutover path) and removed in a follow-up, mirroring the maxcompute cutover -> P4 removal. DefaultConnectorContext) Source-agnostic seams so any connector can express these capabilities; every new method is a default no-op/identity, so hive/iceberg/jdbc/maxcompute/ trino/es are byte-for-byte unaffected. - MVCC/time-travel: `resolveTimeTravel(session, handle, ConnectorTimeTravelSpec)` + `applySnapshot(...)` (threads a pinned snapshot into the handle before planScan); new immutable `ConnectorTimeTravelSpec` (SNAPSHOT_ID / TIMESTAMP / TAG / BRANCH / INCREMENTAL) that fe-core extracts from SQL; `ConnectorMvccSnapshot` carries `schemaId` for schema-evolution-aware time travel. - System tables: `listSupportedSysTables` / `getSysTableHandle`; fe-core composes the `{base}${sys}` reference name. - Scan: proportional split-weight (`getSelfSplitWeight`/`getTargetSplitSize`), COUNT(*) pushdown (`getPushDownRowCount`, 7-arg `planScan`), native-read accounting (`isNativeReadRange`), `getDeleteFiles` (VERBOSE merge-on-read), `ignorePartitionPruneShortCircuit` (predicate-driven connectors keep genuine-null partitions). - Metadata: `ConnectorPartitionInfo.fileCount` + `SUPPORTS_PARTITION_STATS` (rich SHOW PARTITIONS); `ConnectorColumn.withTimeZone` (WITH_TIMEZONE DESC marker, independent of the timestamp_tz mapping flag); connector cache control (`invalidateTable`/`invalidateAll`, `schemaCacheTtlSecondOverride`). - `ConnectorContext`: six engine-side hooks the connector must not implement itself (`loadHiveConfResources`, `vendStorageCredentials`, `getBackendStorageProperties`, `normalizeStorageUri` x2, `getStorageProperties`), bound in `DefaultConnectorContext` over fe-core's single-source-of-truth credential/URI/HiveConf utilities (no re-ported logic that could drift). The metastore-side counterpart of the fe-filesystem StorageProperties SPI: a Paimon catalog resolves its backend (hive/dlf/rest/jdbc/filesystem) by ServiceLoader dispatch on `paimon.catalog.type` instead of a fe-core switch. - `fe-connector-metastore-api`: hadoop/SDK-free `MetaStoreProperties` contracts (HMS/DLF/REST/JDBC/FileSystem) carrying only Map/scalar facts. - `fe-connector-metastore-spi`: `MetaStoreProvider<P>` + first-hit `MetaStoreProviders` dispatcher + five impls, registered via META-INF/services. `HmsMetaStorePropertiesImpl` reproduces the legacy `buildHmsHiveConf` key set and parity-critical ordering (storage overlay before the kerberos block; username/sasl last). Adding a backend = one provider class + one services line (no central enum/switch). - ServiceLoader is loaded with the SPI interface's own (plugin) classloader, NOT the thread-context CL: at CREATE CATALOG the static init first fires on an FE worker thread whose TCCL is the FE app loader, so a 1-arg load would cache an empty provider list process-wide. - fe-core bridges add `initExecutionAuthenticator(List<StorageProperties>)` so HDFS-Kerberos doAs is wired on the plugin path (legacy `initializeCatalog` is dead). Full read + DDL + partitions + statistics + system tables + MVCC/time-travel across all five metastore flavors, planning native (ORC/Parquet) vs JNI reads. - `PaimonCatalogFactory` (pure flavor switch: Options + Hadoop Configuration + HiveConf, classloader-pinned for child-first loading), `PaimonCatalogOps` (injection seam over the remote Catalog for offline-testable metadata), `PaimonConnectorMetadata` (~9 -> 28 methods: schema latest/at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor). - `PaimonScanPlanProvider` (~6 -> 30 methods): predicate/projection pushdown, native-vs-JNI routing, large-file sub-splitting with deletion vectors, COUNT(*) collapse, schema-evolution field-id dictionary, `$ro` unwrap to base FileStoreTable. - New: `PaimonSchemaBuilder` (CREATE TABLE), `PaimonIncrementalScanParams` (@incr), `PaimonTableResolver` (sys/branch-aware handle -> Table), `PaimonLatestSnapshotCache` + `PaimonSchemaAtMemo` (per-catalog caches on the long-lived connector). - `PaimonTypeMapping` gains the write direction (Doris -> Paimon) for CREATE TABLE. - NEW `PluginDrivenMvccExternalTable` (generic MvccTable + MTMV host: the runtime class for paimon tables), `PluginDrivenMvccSnapshot`, `PluginDrivenSysExternalTable` + `systable/PluginDrivenSysTable`. Behavior is selected by `ConnectorCapability` (SUPPORTS_MVCC_SNAPSHOT, SUPPORTS_PARTITION_STATS) — no paimon branch in fe-core. - `PluginDrivenScanNode`: MVCC pin threaded at every handle-consumption site, sys-table time-travel guard, COUNT pushdown, connector-agnostic EXPLAIN re-emission. - Cutover: `CatalogFactory` adds `paimon` to `SPI_READY_TYPES` and drops the built-in case; `GsonUtils` removes the 7 built-in `Paimon*` subtype registrations and adds replay compat so persisted `PaimonExternalCatalog` (all 5 flavors) / `...Database` -> `PluginDriven*` and `PaimonExternalTable` -> `PluginDrivenMvccExternalTable` on deserialization (existing clusters upgrade without losing catalogs). - Nereids/DDL: `PhysicalPlanTranslator` drops the `PaimonScanNode` branch; `UserAuthentication`, `CreateTableInfo`, `ShowPartitionsCommand` generalized; `Env.getDdlStmt` renders LOCATION/PROPERTIES for SHOW CREATE (paimon engine, no credential leak). - fe-filesystem: NEW `HdfsFileSystemProperties` / `HdfsConfigFileLoader` (typed HDFS; defaults-laden BE map vs defaults-free FE Hadoop-config map so it cannot clobber a co-bound object store's fs.s3a.* during a multi-backend merge); S3 MinIO support + legacy tuning defaults; OSS/OBS/COS ANONYMOUS provider-type; new `FileSystemFactory.bindAllStorageProperties` / `FileSystemPluginManager.bindAll`. - NEW `fe-kerberos` leaf module: dependency-free `AuthType` / `KerberosAuthSpec` facts shared across modules without pulling in Hadoop. - Packaging: NEW `fe-connector-paimon-hive-shade` relocates `org.apache.thrift` for HMS 2.3.7; the paimon plugin-zip excludes fe-thrift/libthrift (provided parent-first) to avoid a split `TBase` across loaders; self-contained bundle (hadoop-aws, AWS-SDK, hadoop-hdfs-client). `paimon.version` (1.3.1) pinned as the single property across FE connector / BE paimon-scanner / preload-extensions for FE<->BE Table/Split serde. - BE `PaimonJniScanner.getPredicates()` null-predicate backstop (no-filter scan; pairs with the FE producer now always emitting an empty predicate list). Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Storage/credentials: canonical `s3.*`/`oss.*` keys -> `fs.s3a.*`/`fs.oss.*` (STORAGE-CREDS); canonical `AWS_*` BE creds (STATIC-CREDS-BE); per-table vended DLF/STS token (REST-VENDED); `oss/cos/obs/s3a` -> `s3://` for data + deletion-vector paths (URI-NORMALIZE); external `hive.conf.resources` hive-site.xml (HMS-CONFRES); real doAs on filesystem/jdbc + wrapped HMS read RPCs (KERBEROS-DOAS). - Reads/types: paimon columns forced nullable (READ-NOTNULL); `VARCHAR(65533)` not widened to STRING (VARCHAR-BOUNDARY); dotted `enable.mapping.varbinary` / `.timestamp_tz` keys (MAPPING-FLAG-KEYS); per-type partition-value rendering (NATIVE-PARTVAL); genuine `\N` not coerced to NULL (PARTITION-NULL-SENTINEL); field-id history dict for rename-safe native reads (SCHEMA-EVOLUTION); paimon-native split encode under `enable_paimon_cpp_reader` (CPP-READER). - Planning/perf: restore the `force_jni_scanner` escape hatch (FORCE-JNI-SCANNER); intra-file native sub-splitting (NATIVE-SUBSPLIT); precomputed COUNT(*) row count (COUNT-PUSHDOWN); table row count for CBO / SHOW TABLE STATUS (TABLE-STATS). - Time travel / DDL: `FOR TIME AS OF` under CST/PST session zones (TZ-ALIAS); resolved + validated `driver_url` for jdbc catalogs (JDBC-DRIVER-URL); reject a local-only name conflict on CREATE TABLE (CREATE-TABLE-LOCAL-CONFLICT). plan-doc P5 design / task-list / HANDOFF / deviation tracking notes. - ~70 new FE unit suites: paimon connector (offline recording fakes: `RecordingPaimonCatalogOps` / `FakePaimonTable`), fe-core PluginDriven* MVCC / sys-table / scan-node / split-weight, metastore-spi dispatch + per-backend properties, fe-filesystem HDFS/MinIO/anonymous, fe-kerberos. - checkstyle 0 violations; connector import-gate passes; FE main+test compile clean. - End-to-end paimon behavior is docker-gated (`enablePaimonTest=true`); the five functional areas are covered by the `external_table_p0/paimon` regression suites, green on the latest CI run for this branch.
… make fe-core paimon-SDK-free (T29) (apache#64653) Follow-up to apache#64446 (the Paimon catalog-SPI cutover). After the cutover a `paimon` catalog deserializes to `PluginDrivenExternalCatalog` and its tables to `PluginDrivenMvccExternalTable`, so no legacy `Paimon*` catalog / table / scan-node object is ever instantiated — the entire legacy Paimon subsystem in fe-core is dead code. This PR removes it and makes fe-core's dependency tree fully paimon-SDK-free (zero `org.apache.paimon` imports, the 5 paimon maven deps dropped). Mirrors the P4 maxcompute cutover → legacy-removal (apache#64256). **1. Remove legacy subsystem + sever reverse-refs** (`7632a074e4b`) - Delete the dead fe-core paimon files: `datasource/paimon/*` (all 5 flavor catalogs, `PaimonExternalTable`, `PaimonMetadataOps`, `PaimonUtil`, `source/PaimonScanNode` / `PaimonSplit` / `PaimonPredicateConverter` / `PaimonSource` / `PaimonValueConverter`, `profile/*`), `datasource/metacache/paimon/*`, `systable/PaimonSysTable`, and the legacy-only tests. - Sever the live reverse-references to the deleted classes, keeping every `PluginDriven` / connector sibling branch and the image/replay keep-set: drop the `PAIMON` db-creation switch arm in `ExternalCatalog`; the paimon engine routing in `ExternalMetaCacheMgr` / `ExternalMetaCacheRouteResolver`; the dead `PAIMON_EXTERNAL_TABLE` branch in `Env.getDdlStmt` (keep the LIVE `PLUGIN_EXTERNAL_TABLE` SHOW CREATE LOCATION/PROPERTIES rendering); the `PaimonSysExternalTable` branch in `UserAuthentication`; the legacy clauses + `handleShowPaimonTablePartitions()` in `ShowPartitionsCommand` (keep `hasPartitionStatsCapability()` and the `TableType.PAIMON_EXTERNAL_TABLE` enum). Landed as one compiling unit (mirrors the P4 apache#64300 precedent: `PaimonUtils` still called the removed `ExternalMetaCacheMgr.paimon()`). - Inline the `getPaimonCatalogType()` string literals (`hms`/`filesystem`/`dlf`/`rest`/ `jdbc`) to decouple the still-consumed thin metastore-property classes from the deleted `PaimonExternalCatalog`. **2. Make fe-core paimon-SDK-free** (`32de7d16702`) - Strip the dead paimon-SDK catalog-building cluster from `AbstractPaimonProperties` + the 5 flavors (`initializeCatalog` / `buildCatalogOptions` / `appendCatalogOptions` / `getMetastoreType` / `normalizeS3Config` / … — 0 live callers; the plugin path builds catalogs connector-side). Keep every LIVE SDK-free duty: `warehouse` `@ConnectorProperty`, the Kerberos `executionAuthenticator` doAs wiring (read by `PluginDrivenExternalCatalog`), property normalization/validation, `getPaimonCatalogType`, `Type.PAIMON`. - Vended credentials: delete `PaimonVendedCredentialsProvider` + its test + the `VendedCredentialsFactory` `case PAIMON`. Its SDK methods were dead — reachable only via the iceberg-only `getStoragePropertiesMapWithVendedCredentials`; the real paimon vended path is the connector's `PaimonScanPlanProvider.extractVendedToken` (moved off fe-core at cutover). Relocate its one LIVE duty (the REST "skip static storage map" gate) to a new SDK-free `MetastoreProperties.isVendedCredentialsEnabled()` (base=false, `PaimonRestMetaStoreProperties`=true); `CatalogProperty` routes iceberg through its provider (byte-identical) and everything else through the metastore-props method. - pom: drop `paimon-core` / `-common` / `-format` / `-s3` / `-jindo` from `fe-core/pom.xml`. `fe/pom.xml` `paimon.version` is kept (fe-connector-paimon + BE paimon-scanner still consume it). **3. Doc-sync** (`58160cbcb56`, `895e214b2bd`) — plan-doc HANDOFF / PROGRESS / connectors / P5-T29 legacy-removal design tracking notes. - `mvn -pl :fe-core -am test-compile` (main + test) passes; checkstyle 0 violations; connector import-gate (`tools/check-connector-imports.sh`) passes. - `grep -rn org.apache.paimon fe/fe-core/src` → empty (main and test). - `mvn -pl :fe-core dependency:tree -Dincludes=org.apache.paimon` → empty (no paimon dep, direct or transitive). - End-to-end paimon regression (`external_table_p0/paimon`, docker `enablePaimonTest=true`) passed on this branch.
…kerberos (apache#64655) **P3b: consolidate the drifted Kerberos/Hadoop authentication implementations into the new top-level neutral leaf module `fe-kerberos`** as the single source of truth. Done as 3 commits: 1. **trino → JDK** (`4a740e1`) — replace the only external dependency in the auth path, trino's `KerberosTicketUtils`, with a JDK-only (`javax.security.auth.kerberos`) byte-for-byte equivalent, so the kerberos path is trino-free. 2. **relocate** (`8898e15`) — move the 13 `fe-common` `security.authentication.*` classes to `org.apache.doris.kerberos.*` in `fe-kerberos`; retarget all consumer imports (fe-core + 3 be-java-extensions scanners); merge the duplicate `AuthType`. 3. **unify interface** (`5e3e896`) — merge the two competing `HadoopAuthenticator` interfaces (fe-common's `PrivilegedExceptionAction` variant vs fe-filesystem-spi's `IOCallable` variant) into the single fe-kerberos one, and delete fe-filesystem-hdfs's own `KerberosHadoopAuthenticator`/`SimpleHadoopAuthenticator` copies (which had drifted from the canonical impls). `DFSFileSystem` now routes through the shared authenticators. `fe-kerberos` remains a top-level neutral leaf (no dependency cycle). HDFS filesystem access now uses the same authenticators as the HMS path (restoring parity). Two intentional behavior changes in fe-filesystem-hdfs: simple / no-`hadoop.username` now runs as remote user `hadoop` (was: FE process user, direct); kerberos uses the shared `LoginContext` + 80%-lifetime refresh. fe-filesystem-hdfs 79/0/0 (+fe-kerberos/spi), checkstyle 0, connector import-gate clean, whole-repo grep for the removed symbols = 0. >⚠️ docker kerberos e2e (HDFS kerberized + HMS) NOT yet run — the real gate; UGI login can't be exercised in unit tests.
…move legacy fe-core subsystem (apache#64688) Migrates the legacy in-tree Iceberg catalog (fe-core `datasource/iceberg`, its `property/metastore/*Iceberg*` clusters, and every reverse `instanceof IcebergExternal*` / `case ICEBERG` coupling scattered across nereids / planner / alter) onto the catalog SPI as a self-contained `fe-connector-iceberg` plugin, following the paimon full-adopter + cutover template. `iceberg` is added to `SPI_READY_TYPES`, so an Iceberg catalog now deserializes to `PluginDrivenExternalCatalog` and every functional area — read, write, row-level DML, `ALTER TABLE EXECUTE` procedures, system tables, DDL and MVCC/time-travel — flows through the generic `PluginDriven*` bridge with no Iceberg-specific branch in fe-core. (Iceberg tables surfaced *through* a `hive`/HMS catalog stay on the hive path until P7; this PR covers the dedicated `type=iceberg` catalog.) Iceberg is the widest lakehouse adopter migrated so far — 7 catalog flavors, merge-on-read row-level DML, 9 stored procedures, v3 row-lineage / deletion vectors — so the cutover is held to behavior-parity with the legacy path. Unlike the paimon migration (a cutover PR + a separate removal PR), this PR does the build-out, the live cutover **and** the full legacy-subsystem removal on one branch: fe-core `datasource/iceberg` and its property clusters are deleted, not just left dead. Part of the catalog-SPI migration tracked in apache#65185 (phase **P6**). Full read + write + row-level DML + procedures + system tables + DDL + MVCC/time-travel, planning native (Parquet/ORC) vs JNI reads. Key seams: - `IcebergCatalogFactory` (pure flavor switch, classloader-pinned for child-first loading), `IcebergCatalogOps` (metadata / DDL ops), `IcebergConnectorMetadata` (schema latest / at-snapshot, sys tables, MVCC/time-travel, DDL, partitions, statistics, table descriptor), `IcebergConnectorProvider` (ServiceLoader entry). - `IcebergScanPlanProvider` + `IcebergScanRange`: predicate/projection pushdown (`IcebergPredicateConverter` → Iceberg `Expression`), FileScanTask byte-offset sub-splitting, COUNT(\*) collapse, merge-on-read delete attachment (position delete / Puffin DV / equality delete), `IcebergColumnHandle` field-id column pruning, schema-evolution-safe reads (`IcebergSchemaUtils` field-id dictionary), synthetic rowid + v3 row-lineage metadata columns. - Write path: `IcebergWritePlanProvider` / `IcebergWriteContext` / `IcebergWriterHelper` / `IcebergConnectorTransaction` — INSERT / INSERT OVERWRITE (dynamic + static partition), branch-targeted writes, WRITE ORDERED BY, per-spec distribution, optimistic-concurrency conflict detection, one SDK transaction per statement, BE commit-fragment → DataFile/DeleteFile conversion. - Row-level DML: DELETE / UPDATE / MERGE INTO over position-delete (v2) and deletion-vector (v3) merge-on-read, baseSnapshot-anchored read. - `action/`: the 9 `ALTER TABLE EXECUTE` procedures (rollback_to_snapshot, rollback_to_timestamp, set_current_snapshot, cherrypick_snapshot, fast_forward, expire_snapshots, publish_changes, rewrite_manifests, rewrite_data_files) dispatched by `IcebergExecuteActionFactory`; `rewrite/` hosts the distributed bin-pack `rewrite_data_files` engine. - `IcebergSchemaBuilder` (CREATE TABLE), `IcebergTypeMapping` (both directions), `IcebergColumnChange` / `IcebergComplexTypeDiff` (schema + partition evolution), `IcebergLatestSnapshotCache` / `IcebergManifestCache` (per-catalog caches on the long-lived connector), `dlf/` (Aliyun DLF client). (`fe-connector-metastore-iceberg`) An Iceberg catalog resolves its backend by `MetaStoreProvider` ServiceLoader dispatch on `iceberg.catalog.type` (registered via `META-INF/services`), instead of a fe-core switch — **7 flavors**: REST, Hive Metastore, AWS Glue, Hadoop / filesystem, JDBC, Aliyun DLF, AWS S3Tables. Adding a backend = one provider class + one services line. The connector self-builds the HMS-metastore Kerberos authenticator, so metastore auth is owned entirely by the plugin. - `CatalogFactory` adds `iceberg` to `SPI_READY_TYPES` and drops the built-in case. - `GsonUtils` replay-compat: persisted `IcebergExternalCatalog` → `PluginDrivenExternalCatalog`, `IcebergExternalDatabase` → `PluginDrivenExternalDatabase`, `IcebergExternalTable` → `PluginDrivenMvccExternalTable` on deserialization, so existing clusters upgrade without losing catalogs (metadata backward-compat invariant). - Nereids/DDL/alter generalized off the concrete Iceberg types; SHOW CREATE renders LOCATION / PARTITION BY / ORDER BY / PROPERTIES via the generic path (iceberg engine name preserved, no credential leak). Per the 2026-07 architecture decision (fe-core holds **no** property parsing for migrated connectors; storage parsing → `fe-filesystem`, metastore parsing → `fe-connector`, both plugin-side; the plugin assembles → BE thrift → hands back to fe-core), iceberg's property/credential/auth resolution moves entirely into the connector: - Storage credentials bound to `fe-filesystem` on both scan and write paths (one source), retiring the second fe-core `StorageProperties.createAll` parse. - Vended temporary credentials, `warehouse` → `fs.defaultFS` derivation (hadoop flavor only, via the neutral `Connector.deriveStorageProperties` SPI), and Kerberos doAs are all connector-owned; the fe-core pre-execution authenticator is a no-op for plugin catalogs. - fe-core's shared static property/metastore helpers are kept alive only for the connectors still on the legacy path (Hive / Hudi / LakeSoul / HMS-iceberg). Net **−21.7k** lines from fe-core. Removes the 4 dead entity classes (`IcebergExternalTable` / `IcebergExternalDatabase` / `IcebergSysExternalTable` / `IcebergExternalCatalog`), the fe-core Iceberg `MetastoreProperties` cluster + its credential helpers, the 5 fe-core Iceberg connectivity probes and the coordinator branches, and un-registers the Iceberg property factories (fail-loud on any stray call). The now-dead paimon `MetastoreProperties` cluster left behind by P5 is swept in the same pass. `GsonUtils` retains only the three string compat labels for old-image upgrade. Parity fixes surfaced by the SPI cutover (each restores legacy semantics): - Classloader: TCCL pinned to the plugin classloader at all three loci where fe-core crosses into bundled Iceberg (scan call thread, write/DDL/commit engine thread via `TcclPinningConnectorContext`, and Iceberg's own manifest-writer worker pool), preventing a duplicate-copy `ClassCastException`. - Nested schema dictionary field names lowercased so a mixed-case nested struct (e.g. Iceberg `DROP_AND_ADD`) can't crash BE with `std::out_of_range`. - Session time zone: lazy `ZoneId` resolution tolerant of Doris CST/PST-style aliases in `FOR TIME AS OF` and datetime partition predicates. - Storage / URI normalization (`oss/cos/obs/s3a` → `s3`), SHOW CREATE secret redaction, and `ALTER TABLE EXECUTE ... WHERE` rejection kept at legacy wording. - ~68 FE unit test classes: iceberg connector (offline recording fakes for the remote Catalog / Table), metastore-iceberg per-flavor dispatch + properties, and the fe-core `PluginDriven*` MVCC / sys-table / scan-node / GSON-replay surfaces; many carry mutation checks. - checkstyle 0 violations; connector import-gate passes (no `fe-connector-iceberg → fe-core` import); FE main + test compile clean. - End-to-end iceberg behavior is docker-gated (`enableIcebergTest=true`); read, write, row-level DML, procedures, system tables, DDL and time-travel are covered by the `external_table_p0/iceberg` (+ `external_table_p2/iceberg`) regression suites. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…() TVF handles DATE/null partition_values() over a paimon table failed the whole query on a DATE partition and mis-rendered null partitions. PaimonConnectorMetadata.collectPartitions put the un-rendered partition.spec() (DATE=epoch-day, null=__DEFAULT_PARTITION__) into the ConnectorPartitionInfo value map, which the active TVF feeder (PluginDrivenExternalTable.getNameToPartitionValues) reads by remote name. The consumer parses DATE via convertStringToDateV2 -> throws on the epoch-day "19723" -> error result -> the whole TVF query fails; a null value stayed the paimon sentinel and rendered as a literal string instead of SQL NULL. Build the value map with the same rendered/normalized values already computed into orderedValues (DATE via DateTimeUtils.formatDate, genuine-null via HIVE_DEFAULT_PARTITION), keyed by the raw remote column name. Only the values change; keys stay remote names, so the lookup is unaffected. This aligns paimon with hive/iceberg, whose value maps already hold decoded canonical strings. Connector-local, no fe-core change. Rejected switching the shared fe-core consumer to getOrderedPartitionValues(): it would regress iceberg null partitions (rendered as literal "null") and break MaxCompute (empty ordered values). Flip the two paimon unit tests that pinned the old raw-value contract to assert the rendered values, and add PaimonConnectorMetadataPartitionTest.partitionValueMapCarriesRenderedValuesForTvf (DATE rendered + genuine-null DATE -> HIVE_DEFAULT_PARTITION). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
ConnectorDeleteFile only modeled (path, fileFormat, recordCount, properties) and was never constructed or read anywhere. ConnectorScanRange.getDeleteFiles() returned an empty list by default with zero overrides and zero callers across the whole fe tree; Iceberg models merge-on-read deletes via its own typed IcebergScanRange.DeleteFile / thrift, bypassing this abstraction entirely. Remove the ConnectorDeleteFile class and the getDeleteFiles():List<ConnectorDeleteFile> default method. The unrelated live method ConnectorScanPlanProvider.getDeleteFiles(TTableFormatFileDesc):List<String> (EXPLAIN delete-file path readback, with iceberg/paimon overrides) is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…torRange pushdown abstraction ConnectorDomain and ConnectorRange (javadoc: "fast partition pruning") were never constructed or read anywhere. Their only appearance was as the declared type of ConnectorFilterConstraint.columnDomains, which the sole production producer (PluginDrivenScanNode.buildFilterConstraint) always populated with an empty map; all three applyFilter consumers (hive/hudi/trino) read only getExpression() and none call getColumnDomains(). Delete ConnectorDomain and ConnectorRange, drop the columnDomains field, the two-arg constructor and getColumnDomains() from ConnectorFilterConstraint (keeping the expression / getExpression() path), and update the single caller PluginDrivenScanNode.buildFilterConstraint to the one-arg constructor. Verified: connector-api + hive/hudi/trino (main+test) + fe-core compile clean; HiveConnectorMetadataPartitionPruningTest + HudiPartitionPruningTest green (26). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…nnectorMvccSnapshot ConnectorMvccSnapshot was the only member of its value-object family (ConnectorMvccPartition, ConnectorMvccPartitionView, ConnectorTableFreshness, ConnectorTimeTravelSpec, ConnectorTableStatistics) missing value semantics. Add equals/hashCode/toString over all six fields, mirroring the sibling style. Purely additive: nothing currently uses the snapshot as a Map/Set key or compares it by value, so there is zero runtime behavior change; both fe-core wrappers (ConnectorMvccSnapshotAdapter, PluginDrivenMvccSnapshot) keep their own identity equality. Add ConnectorMvccSnapshotTest coverage asserting every field participates in equality/hashCode and appears in toString. Tests green (5); checkstyle clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…ES" comments
The hms->SPI flip is complete: CatalogFactory.SPI_READY_TYPES contains "hms",
dispatch to the connector is purely `catalog instanceof PluginDrivenExternalCatalog`
(there is no separate write/DDL/P7.x runtime gate), and the legacy classes
(HMSExternalCatalog, HMSExternalTable, HiveInsertExecutor, PhysicalHiveTableSink,
MetastoreEventsProcessor, HiveScanNode) are removed. Every read / write / DDL / scan /
procedure path on a flipped hms catalog is therefore live now, which makes the
pervasive "dormant/inert until hms enters SPI_READY_TYPES / until the flip / until the
P7.4/P7.5 cutover" comments factually wrong (they claim the path is not yet reached).
Remove or reword these stale gate clauses across HiveConnector, HiveConnectorMetadata,
HiveWritePlanProvider, HiveFileListingCache, HiveReadTransactionManager, CachingHmsClient,
ConnectorExecuteAction, and the hive connector test suites, matching the codebase's own
post-cutover wording ("Live since the hms flip", cf. HiveScanPlanProvider /
HiveConnectorTransaction). Comment-only; no behavior change.
Preserved: genuine runtime "dormant path" notes (a gateway that never delegated must not
build a sibling). Deliberately left untouched: the MVCC/MTMV per-partition freshness
substep comments and the metastore event-sync comments, whose subsystem state is out of
this cleanup's verified scope.
Verified: fe-connector-hive + fe-connector-hms compile (main + test); checkstyle clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…eanup) done Record completion of the B3 batch (findings apache#21/apache#23/apache#25/apache#28) in HANDOFF + TASKLIST: delete dead ConnectorDeleteFile, delete dead ConnectorDomain/ConnectorRange, add ConnectorMvccSnapshot value semantics, and clean up the stale "dormant until SPI_READY_TYPES" comments. Also corrects the tracking record for apache#28, which was mistakenly filed STALE_FIXED (the earlier recon searched the wrong module); the comments were in fact still present and are now cleaned. Notes the deliberate scope carve-outs (MVCC/MTMV freshness + event-sync subsystems) and the suggested follow-up sweep for the iceberg/maxcompute own-cutover comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
… reads Paimon branch/time-travel/incremental/system-table reads over an HMS Paimon catalog fail on BE with: NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf paimon-hive-connector-3.1 marks hive-exec as `provided` and only bundles relocated hive classes under org.apache.paimon.shade.*, so the real org.apache.hadoop.hive.conf.HiveConf (needed by org.apache.paimon.hive.SerializableHiveConf when a HMS-backed Paimon table is deserialized on BE, and by HiveCatalog.createHiveConf) was never in paimon-scanner's own fat jar. It used to be provided accidentally by java-udf's hive-catalog-shade, which sits on BE's shared JVM classpath (bin/start_be.sh preloads both preload-extensions and java-udf, and that classloader is the parent of every per-scanner JniScannerClassLoader). apache#65733 replaced hive-catalog-shade with the slim hive-udf-shade, dropping HiveConf from the shared classpath and breaking paimon-scanner. Make paimon-scanner self-contained by bundling the Hive classes it actually needs: hive-common (HiveConf plus its construction closure) and hive-shims-common (shims/Utils, used by the lazy HiveConf#getUser). Both are pinned to ${hive.version} (3.1.x, matching the Hive the connector was built against and the previous hive-catalog-shade 3.1.1) and have their transitive trees pruned; hadoop is already on BE's shared classpath. Net jar growth is ~540KB (hive-common + hive-shims-common only), no jetty/orc/ant bloat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n HMS reads Reading an HMS-backed Paimon table (branch/time-travel/incremental/system tables) over an HMS Paimon catalog fails on BE with: NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException This is the metastore-API counterpart of the HiveConf gap fixed in the previous commit. When BE deserializes an HMS-backed Paimon table, the serialized graph's org.apache.paimon.hive.HiveCatalogLoader links org.apache.paimon.hive.HiveCatalog, which references a broad slice of org.apache.hadoop.hive.metastore.api.* (Table, Database, Partition, StorageDescriptor, SerDeInfo, FieldSchema, NoSuchObjectException, UnknownTableException, ...). paimon-hive-connector-3.1 marks hive-exec `provided` and only bundles relocated hive under org.apache.paimon.shade.*, so this real `api` package was never in paimon-scanner's own fat jar. It used to be provided accidentally by java-udf's hive-catalog-shade on BE's shared classpath, but apache#65733 replaced that with the slim hive-udf-shade. In Hive 3.x the whole thrift-generated metastore `api` package lives in hive-standalone-metastore (the `hive-metastore` artifact is a thin shim and does NOT contain it). Bundle it, pinned to ${hive.version} (3.1.x) like the two hive jars added previously, with its transitive tree pruned: the api classes only extend org.apache.thrift.* (already on BE's shared classpath via java-common's libthrift 0.16, exactly as under the old hive-catalog-shade, which itself bundled no plain thrift) and the java stdlib; datanucleus/orc/hadoop/jetty/etc. are only needed by the metastore server/client, which BE never runs (it reads the already-resolved table). Net add is the ~11MB metastore api package only, no jetty/orc/datanucleus bloat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DHwqsiGWNosZCeS4ecK97e
…for time-travel reads The generic scan node builds column handles BEFORE it pins the MVCC snapshot, so it can only key the handle map by the LATEST schema. A time-travel query (FOR VERSION AS OF <old>) binds its slots to the PINNED (old) schema; when a column was renamed after the pinned snapshot, its old-name slot misses the latest-keyed map and is silently dropped. In a "mixed projection" (a surviving column plus the renamed one) the dropped column still reaches BE as a scan slot but is absent from paimon's field-id dictionary -> BE StructNode std::out_of_range -> SIGABRT. Iceberg is immune because it rebuilds its dict from the full pinned schema; paimon has no such defense. Root cause: getColumnHandles has no snapshot dimension, unlike getTableSchema which already carries a (session, handle, snapshot) overload. Fix (framework-level, mirrors the getTableSchema 2-arg/3-arg split): - connector-api: add a default getColumnHandles(session, handle, snapshot) overload (delegates to the 2-arg latest path) and a supportsColumnHandleSnapshotPin() capability (default false). - fe-core PluginDrivenScanNode.buildColumnHandles: resolve this scan's snapshot with the same version-aware selector as pinMvccSnapshot and, ONLY when a DISTINCT historical schema is pinned (getPinnedSchema() != null), build the handles at the pinned schema via the 3-arg overload. Normal and same-schema reads keep the 2-arg latest path byte-for-byte. Add a fail-loud guard: for a connector that declares supportsColumnHandleSnapshotPin, a bound column that exists in the pinned schema but has no handle throws instead of being silently dropped into a BE crash. The guard is capability-gated so iceberg (which recovers via its own dict rebuild) keeps the unchanged silent-skip path. - paimon: implement the 3-arg overload keyed by the pinned names via the same memoized schemaAt read the 3-arg getTableSchema uses, and declare the capability. No projection/dict change is needed: the paimon scan side already resolves the pinned schema, so feeding it the pinned column names is enough. Adds an e2e regression (test_paimon_time_travel_rename) reproducing the crash on the preinstalled sc_parquet table (snapshot 1 predates the renames): the mixed projection and SELECT * time-travel queries crash before the fix and return the pinned-snapshot rows after it, on both the native and JNI reader paths. Part of apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…or time-travel reads Sibling of the getColumnHandles fix. getTableStatistics has no snapshot dimension and the table-level row count is served from a cross-statement cache keyed by table only, computed at the LATEST snapshot. A FOR VERSION/TIME AS OF (or @branch/@tag) query's scan reads the pinned snapshot, but the optimizer estimated its cardinality from the latest count -> skewed join reorder / build-side selection. This is estimate-only: results stay correct, but the plan can be poor when the pinned snapshot's size differs from latest. The row-count cache loader runs on a background thread with no query context, so the snapshot cannot be recovered inside it; it must be captured in the query thread at the lookup site. Fix: - connector-api: add a default getTableStatistics(session, handle, snapshot) overload (delegates to the 2-arg latest path). - fe-core StatementContext.getVersionedSnapshot: resolve a GENUINE time-travel snapshot for a table (one pinned under a non-empty version key), or empty for a plain/latest reference or an ambiguous multi-version pin. Row-count skew occurs for ANY time-travel to a differently sized snapshot, so this keys off the version selector rather than schema evolution. - fe-core PluginDrivenExternalTable.getRowCount: when this statement pins a versioned snapshot for the table, compute the row count directly AT that snapshot in the query thread (bypassing the latest-keyed cross-statement cache; a historical count is not worth caching cross-statement) and fall back to the latest cached estimate if the connector cannot count at it. Plain reads and no-context calls keep the cached path byte-for-byte. - iceberg: read the pinned snapshot's summary via table.snapshot(snapshotId) instead of currentSnapshot(), reusing the same total-records/position-delete/ equality-delete formula. - paimon: apply the snapshot to the handle with the same applySnapshot the scan path uses, copy its scan options onto the resolved table, and sum the split row counts; any failure degrades to empty (caller falls back to latest). Adds an e2e regression (test_paimon_time_travel_rowcount) asserting the EXPLAIN cardinality of a FOR VERSION AS OF scan equals the pinned snapshot's row count (3) rather than the latest (18). Part of apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…teContext javadoc to the static partition spec getWriteContext()'s javadoc (method + class level) promised a "free-form write context (static partition spec, write path, and other connector-defined keys)", but the sole producer (PluginDrivenTableSink.bindDataSink) only ever populates it with PluginDrivenInsertCommandContext.getStaticPartitionSpec() -- the "write path" and "other keys" never exist. All three write providers (hive/iceberg/ maxcompute) consume it purely as the static partition spec (iceberg ships it verbatim as TDataSink.static_partition_values). Doc-only: narrow both javadoc loci to state it is the static partition spec (an empty map when the write is not statically partitioned). The getWriteContext method name is kept; the rename is a separate, larger change. Part of apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…c (dead iceberg_bucket / wrong algorithm names) ConnectorBucketSpec models a TABLE-LEVEL hash/random distribution (DISTRIBUTE BY) and is consumed only by the hive connector. Its javadoc listed three algorithm values, all misleading: - "iceberg_bucket" is dead documentation: it appears nowhere else in the tree. Iceberg's bucket(num, column) is a per-column partition-spec transform inside PARTITIONED BY (IcebergSchemaBuilder.buildPartitionSpec), never routed through this spec; iceberg rejects a whole-table DISTRIBUTE BY outright (IcebergConnectorMetadata.rejectDistribution). - "hive_hash" is not a value this spec ever carries either. The real algorithm strings, produced by CreateTableInfoToConnectorRequestConverter, are "doris_default" (hash) and "doris_random" (random). Doc-only: rewrite the javadoc to state the table-level hash/random purpose, point iceberg's per-column bucket transform at its actual PARTITIONED BY path, and list the real algorithm values. No behavior change (the optional iceberg fail-loud the survey suggested already exists as rejectDistribution). Part of apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…rrow opt-in capability interfaces
ConnectorTransaction (and fe-core Transaction) carried source-specific methods as
default-throwing methods every connector/transaction inherited: a maxcompute
write-block allocator and an iceberg-compaction rewrite pair. This is a
god-interface smell — jdbc/hive/paimon transactions carry dead throwing methods
they must never call, and fe-core's generic Transaction embeds a maxcompute
concept. Aligns with Trino, which gates such capabilities by type/registration
BEFORE dispatch rather than by an inherited default-throw.
Sink each capability into a narrow interface the consumer checks via instanceof,
turning "unsupported" from a runtime throw into a type mismatch:
Rewrite (iceberg), connector-api only:
- New RewriteCapableTransaction { registerRewriteSourceFiles,
getRewriteAddedDataFilesCount }; removed both from ConnectorTransaction.
- IcebergConnectorTransaction implements it (already overrode the methods).
- ConnectorRewriteDriver instanceof-checks and casts once, failing loud with a
clear type-mismatch message instead of a mid-rewrite UnsupportedOperationException.
Write-block (maxcompute), both sides:
- New connector-api WriteBlockAllocatingConnectorTransaction { allocateWriteBlockRange };
removed allocateWriteBlockRange + supportsWriteBlockAllocation from ConnectorTransaction.
MaxComputeConnectorTransaction implements it.
- New fe-core WriteBlockAllocatingTransaction extends Transaction; removed the same
pair from the generic Transaction contract (fe-core shrinks). PluginDrivenTransaction's
write-block wrapper becomes a subclass created only when the connector transaction is
write-block-capable, so instanceof is a real gate.
- FrontendServiceImpl.getMaxComputeBlockIdRange gates on
instanceof WriteBlockAllocatingTransaction instead of a supportsWriteBlockAllocation()
runtime flag.
Behavior is unchanged (txn-id granularity, commit/rollback, and all generic methods
untouched); this is a type-level refactor of a latent interface smell, not a bug fix.
Updated the transaction/defaults/RPC unit tests to assert the instanceof gate; compile
+ checkstyle green; PluginDrivenTransactionManagerTest, ConnectorTransactionDefaultsTest,
MaxComputeConnectorTransactionTest, IcebergConnectorTransactionTest all pass.
Part of apache#65185
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…ne, only B5 (sign-off) + apache#27 remain Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…rade apache#10/apache#12/apache#16 next session, skip apache#11/apache#13/apache#19 Captures the 2026-07-22 decision to upgrade three optional enhancements (paimon nested-field comment, iceberg writeDefault, freshness-marker rename + iceberg SqlCache suppression root cause) with per-item implementation pointers and examples, and to leave apache#11/apache#13/apache#19 as intentional non-goals. Flags that apache#16's SqlCache fix touches cache correctness and needs a design sign-off first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…te and read paths A COMMENT on a field nested inside a STRUCT column was dropped on both the paimon write and read paths, and — for every connector — again in the shared fe-core converter, so DESCRIBE / SHOW CREATE TABLE never reported it. Root cause: three narrow constructors each dropped the nested field metadata. - write PaimonTypeMapping.toPaimonRowType: 3-arg new DataField(id, name, type) - read PaimonTypeMapping.toStructType: 2-arg ConnectorType.structOf(names, types) - fe-core ConnectorColumnConverter.convertStructType: 2-arg new StructField(name, type) (shared by all connectors' read paths — the real reason no connector surfaced nested comments in DESCRIBE). Fix: thread the comment (and, on the read paths, the nullability) through the matching 4-arg constructors. - toPaimonRowType -> new DataField(id, name, type.copy(isChildNullable(i)), getChildComment(i)); null comment is byte-identical to the 3-arg form. - toStructType -> structOf(names, types, f.type().isNullable(), f.description()). - convertStructType -> new StructField(name, type, getChildComment(i), isChildNullable(i)). Backward compatible: connectors that leave these unset get null comment / nullable=true, identical to the prior 2-arg StructField, so only paimon (which now carries the data) changes behavior. Tests: - PaimonTypeMappingToPaimonTest.nestedStructFieldCommentPreserved (write) - PaimonTypeMappingReadTest.nestedStructFieldCommentAndNullabilityCarried (read) - ConnectorColumnConverterTest.convertStructTypeCarriesFieldNullabilityAndComment - e2e test_paimon_nested_struct_comment.groovy: CREATE TABLE with a commented nested STRUCT field on a paimon(hms) catalog, then assert SHOW CREATE / DESC render the comment (read paths) and t$schemas.fields contains it (write path). paimon 16 + fe-core 28 unit tests green; both modules build + checkstyle pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…itted and DESCRIBE apply it parseSchema hardcoded every ConnectorColumn's defaultValue to null, so an iceberg (v3) column write default set via ALTER TABLE ADD COLUMN c INT DEFAULT 42 silently vanished after a schema refresh: DESCRIBE showed no default and an INSERT that omitted the column wrote NULL. Types.NestedField.writeDefault() was never read. Fix: add IcebergSchemaUtils.writeDefaultToDorisString(type, writeDefault, enableTimestampTz), which reuses the read-side serializeInitialDefault human-string form (Transforms.identity(type).toHumanString, timestamp normalized to DATETIMEV2 spacing, timestamptz offset dropped when tz-mapping is off) so a write default displays exactly like a read default. It returns null for a missing default, for complex types (STRUCT/LIST/MAP), and for binary-like types (UUID/BINARY/FIXED), whose value can't be carried as a flat Doris default literal. parseSchema passes field.writeDefault() through it as the ConnectorColumn default. This only populates the FE Column metadata (DESCRIBE + INSERT-omitted fill). It is orthogonal to the read default (initialDefault), which flows to BE through the schema dictionary (apache#65502) and is untouched. iceberg ALTER/MODIFY read the DDL target column's default, not the loaded schema's, so there is no ALTER side-effect. Tests: - IcebergSchemaUtilsTest.writeDefaultCarriedAsDorisString: INT/STRING/BOOLEAN/DATE/ TIMESTAMP/timestamptz string forms, plus UUID/BINARY/LIST/no-default -> null. - e2e test_iceberg_write_default.groovy: create a v3 table, ALTER ADD COLUMN c INT DEFAULT 42, refresh, assert DESC shows 42 and INSERT (id) omitting c yields c=42. IcebergSchemaUtilsTest 24 tests green; module builds + checkstyle pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…misleading Millis unit ConnectorMvccPartitionView.getNewestUpdateTimeMillis() (mirrored on PluginDrivenMvccSnapshot) is named as if it returns epoch milliseconds, but the value is a source-defined-scale monotonic change token: iceberg fills it with the PARTITIONS metadata last_updated_at, which iceberg stores in MICROSECONDS, and the generic model passes it through verbatim (master parity). Every consumer treats it purely as a version token (equality / ordering for the dictionary auto-refresh and MTMV staleness), never as a wall clock, so the name is the only lie. Rename it to getNewestUpdateMonotonicMarker (field newestUpdateMonotonicMarker) and correct the javadoc that wrongly claimed "epoch millis". Pure symbol rename with no behavior change; the already-correct "iceberg represents this in MICROSECONDS, do not treat it as millis" explanation is preserved. This makes the unit unmistakable ahead of the SqlCache eligibility fix, which will add a separate, genuinely wall-clock accessor for the quiet-window gate. fe-connector-api ConnectorMvccPartitionView + fe-core PluginDrivenMvccSnapshot / PluginDrivenMvccExternalTable + fe-connector-iceberg IcebergPartitionUtils and the two tests. Builds across all three modules; IcebergPartitionUtilsTest + PluginDrivenMvccExternalTableTest green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
… clock so iceberg tables can cache An iceberg table (and any query joining one) was never eligible for SqlCache. This is an availability bug, NOT a stale-read bug: staleness is guarded independently by a version-token equality check (SqlCacheContext stores getNewestUpdateVersionOrTime, NereidsSqlCacheManager compares it), which this change does not touch. Root cause: CacheAnalyzer.CacheTable.latestPartitionTime is overloaded as both the BE PCache version key AND the wall-clock input to the "quiet for N seconds" gate (now - latestPartitionTime >= cache_last_version_interval_second). For an external table it holds getNewestUpdateVersionOrTime(); hive/paimon return epoch millis so the gate works by luck, but iceberg returns last_updated_at in MICROSECONDS (~1.7e15). Math.max(now, micros) clamps now up to the micros value, so the delta is always 0 and never clears the window -> iceberg never caches, and its huge value also poisons any multi-table query it joins. Fix (connector supplies a genuine wall-clock millis, kept separate from the token): - ConnectorMvccPartitionView carries a new newestUpdateWallClockMillis alongside the monotonic marker; iceberg fills it from last_updated_at / 1000 (the connector owns its unit), while the marker stays micros for the version-token path (master parity). - PluginDrivenMvccSnapshot mirrors the field; MTMVRelatedTableIf gains a default getNewestUpdateTimeMillisForCache() returning getNewestUpdateVersionOrTime() (correct for olap/hive/paimon whose token is already millis), which PluginDrivenMvccExternalTable overrides so the range-view (iceberg) branch returns the wall-clock millis. - CacheAnalyzer gains CacheTable.latestPartitionUpdateMillis (olap: visibleVersionTime; external: the new accessor). latestPartitionTime stays the raw token (BE PCache key, full-precision staleness), and the quiet-window gate now subtracts max(latestPartitionUpdateMillis), decoupled from the token-sorted latestTable. Only iceberg's gate behavior changes (hive/paimon/olap keep the same value). The SPI field / accessor are backward compatible (overloaded ctors default to 0). Tests: - IcebergPartitionUtilsTest: the view's wall clock == micros marker / 1000. - PluginDrivenMvccExternalTableTest.testRangeViewCacheGateUsesWallClockMillisNotMarker: getNewestUpdateTimeMillisForCache returns the wall clock; the token stays the marker. - PluginTableCacheAnalyzerTest.testGateValueSourcedFromWallClockAccessor: the gate value comes from the wall-clock accessor while latestPartitionTime stays the raw token. - e2e test_iceberg_sqlcache.groovy: a quiet iceberg table becomes cacheable, and a write invalidates it (no stale results). 67 unit tests green across fe-connector-api / iceberg / hive / fe-core; builds + checkstyle pass. The probe rename that precedes this landed in 8043536a812. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…no oshi links All 13 trino_connector regression cases failed with NoSuchMethodError com.sun.jna.Native.load(String, Class) at com.sun.jna.platform.linux.Udev.<clinit>, raised from oshi during TrinoConnectorJniScanner.createSession. Root cause: JNI scanners load via JniScannerClassLoader, a parent-first URLClassLoader whose parent is the BE system classpath. bin/start_be.sh puts preload-extensions ahead of lib/hadoop_hdfs, whose bundled jna is < 5.0 (no load(String, Class)). A modern jna used to reach the shared preload classpath transitively through fe-common -> trino-main -> oshi, shadowing the hadoop copy. Commit ca840c9 made fe-common trino-free, evicting jna from preload; the trino-connector-scanner fat jar still bundles jna 5.13 but cannot win, because parent-first resolves com.sun.jna.Native from the hadoop jar first while jna-platform's Udev (absent there) still loads 5.14 from the scanner -> version skew -> NoSuchMethodError. Restore net.java.dev.jna:jna and jna-platform (5.13.0, what oshi 6.4.5 pulls and what fe-core pins inline) as runtime deps in preload-extensions, the shared parent of all JNI scanners, so the modern jna again lands first on DORIS_CLASSPATH and shadows the hadoop copy. Same evicted-transitive-dep pattern already fixed here for commons-lang and fastutil-core. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
test_paimon_nested_struct_comment asserted that DESC renders the comment on a field nested inside a STRUCT column (note_a). DESC's Type column is a pure type signature produced by Type.hideVersionForVersionColumn, whose struct branch emits name:type only and by design omits nested field comments for every table (internal olap and external alike) -- so the assertion can never pass. It rendered struct<a:int,b:text>, failing the case, even though the fix is correct. The nested comment is already proven by the two other assertions: SHOW CREATE TABLE (read path, via StructField.toSql which does append the comment) and the on-disk $schemas.fields (write path). Remove the DESC assertion and note why; making DESC surface nested struct field comments would be a separate, branch-wide product change out of scope for this paimon fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
…omment and reject complex->primitive The flat scalar `modifyColumn` seam had two gaps versus the nested path (`IcebergNestedColumnEvolution.modifyColumn`) and legacy `IcebergMetadataOps`: 1. It dropped the apache#65329 `commentSpecified` flag and unconditionally wrote `column.getComment()` (empty when COMMENT is omitted), so a MODIFY that omits COMMENT cleared the column's existing doc instead of preserving it. The nested path already honored the "omit-preserves-metadata" semantics. 2. It lacked the complex->primitive guard, so modifying a STRUCT/ARRAY/MAP column to a primitive fell into the primitive branch and let iceberg's `updateColumn` leak a raw `struct<8: city ...>` type-diff error instead of the clean "Modify column type from complex to primitive is not supported". Both are fixed by threading `commentSpecified` through the flat seam (computing `targetComment = commentSpecified ? getComment() : current.doc()`) and adding the complex->primitive guard, symmetric with the existing non-complex->complex guard and mirroring the nested path. Regression coverage (external_table_p0/iceberg/iceberg_schema_change_ddl): `after_no_comment` DESC after `MODIFY COLUMN col1 DOUBLE`, and `MODIFY COLUMN address STRING`. Added unit tests: `testModifyColumnOmittedCommentPreservesExistingDoc`, `testModifyComplexToPrimitiveFailsLoud`, and `testModifyScalarColumnThreadsCommentSpecifiedToSeam`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
morningman
force-pushed
the
master-catalog-spi-review-18
branch
from
July 23, 2026 10:08
799cb12 to
d4da48c
Compare
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29540 ms |
Contributor
TPC-DS: Total hot run time: 177323 ms |
Contributor
ClickBench: Total hot run time: 25.1 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.