[feature](fe) Introduce fe-connector SPI framework and migrate JDBC/ES catalogs to plugin-driven architecture#62183
Merged
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
3 similar comments
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
morningman
added a commit
to morningman/doris
that referenced
this pull request
Apr 12, 2026
### What problem does this PR solve? Issue Number: close apache#62183 Problem Summary: Fix multiple regression failures in the connector-based JDBC/ES external catalog migration. This commit addresses 8 root causes covering 20+ test failures from CI External Regression pipeline build 926236. Fixes include: - detectDoris() initialization ordering: moved from constructor to postInitialize() hook so data source is live before detection runs. Added dorisTypeToConnectorType() for Doris-to-Doris type mapping including HLL, BITMAP, QUANTILE_STATE types. - Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match "TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc. - Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0 for integer branch, and match old boundary thresholds. - ClickHouse DB listing: use SHOW DATABASES for full listing; fix databaseTermIsCatalog inversion for old drivers. - DATETIMEV2 precision: ConnectorColumnConverter reads precision (not scale) for datetime fractional seconds, matching connector encoding convention. - EXPLAIN cache: invalidate scanNodeProperties after convertPredicate() so pushed conjuncts are reflected in EXPLAIN output. - ExecutionAuthenticator: call initPreExecutionAuthenticator() in PluginDrivenExternalCatalog.initLocalObjectsImpl(). - PassthroughQueryTableHandle: add instanceof guards in JdbcConnectorMetadata to prevent ClassCastException for TVF. - isKey: hardcode true for all columns matching legacy behavior. ### Release note None ### Check List (For Author) - Test: Regression test (CI External Regression pipeline) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
morningman
added a commit
to morningman/doris
that referenced
this pull request
Apr 13, 2026
…faces
Issue Number: close #xxx
Problem Summary: Add the fe-connector module hierarchy (fe-connector-api and
fe-connector-spi) as the foundation for Catalog SPI modularization. This
follows the same pattern as fe-filesystem, enabling external data source
connectors to be developed and deployed as independent plugins.
fe-connector-api defines 30 zero-dependency interfaces including:
- Opaque handle types (ConnectorTableHandle, ConnectorColumnHandle, etc.)
- ConnectorMetadata composed from sub-interfaces (SchemaOps, TableOps,
PushdownOps, StatisticsOps, WriteOps)
- apply* pushdown negotiation (FilterApplicationResult, ProjectionApplicationResult)
- ConnectorScanPlanProvider for split generation
- ConnectorCapability enum for capability declaration
- Lightweight type system (ConnectorColumn, ConnectorType) independent of fe-core
fe-connector-spi defines the provider SPI:
- ConnectorProvider extends PluginFactory for ServiceLoader discovery
- ConnectorContext for engine-to-connector runtime services
None
- Test: No need to test - module skeleton with interfaces only, no behavioral changes
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorPluginManager and session bridge for connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: This step adds the connector plugin loading infrastructure
to fe-core, mirroring the FileSystemPluginManager pattern. It enables
connector plugins to be discovered via ServiceLoader (builtins) and loaded
from the plugin directory at runtime via DirectoryPluginRuntimeManager with
ChildFirst ClassLoader isolation.
Key additions:
- ConnectorPluginManager: manages ConnectorProvider lifecycle (builtin +
directory-based plugin loading)
- ConnectorFactory: static factory singleton initialized by Env at startup
- ConnectorSessionBuilder: bridges ConnectContext → ConnectorSession for
ClassLoader-safe session passing across SPI boundary
- ConnectorSessionImpl: immutable ConnectorSession implementation
- DefaultConnectorContext: minimal ConnectorContext providing catalogName,
catalogId, and FS access
- Config.connector_plugin_root: configurable plugin directory
- Env.initConnectorPluginManager(): startup initialization hook
None
- Test: No need to test - infrastructure only, no connector plugins exist yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove Iceberg SDK dependency from ExternalMetadataOps interface
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The ExternalMetadataOps interface (a core fe-core interface
for external catalog metadata operations) had a direct import of
org.apache.iceberg.view.View, coupling the interface to Iceberg SDK. The
ExternalMetadataOperations factory class also imported all connector-specific
types (Iceberg, Paimon, MaxCompute, Hive), creating unnecessary coupling.
Changes:
- ExternalMetadataOps.loadView() return type changed from View to Object.
The method is only overridden by IcebergMetadataOps, and all callers are
in iceberg-specific code that already imports View for casting.
- Deleted ExternalMetadataOperations factory class entirely. Each catalog
now directly constructs its own MetadataOps (the factory was a thin
indirection adding no value since each caller already knows its type).
- Removed unused View import from IcebergMetadataOps (return type is now
Object; iceberg view.View is only used in IcebergExternalMetaCache).
None
- Test: No need to test - pure refactoring, no behavioral change
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Abstract Hadoop Configuration to Map-based catalog properties
Issue Number: close #xxx
Problem Summary: ExternalCatalog.getConfiguration() returns org.apache.hadoop.conf.Configuration,
which leaks Hadoop types into the core catalog interface. This blocks future SPI extraction of
connector modules since fe-connector-api must be free of Hadoop dependencies.
This commit introduces getHadoopProperties() returning Map<String, String> and a static
buildHadoopConfiguration() utility, then migrates all callers (HudiExternalMetaCache,
IcebergUtils, HiveMetaStoreClientHelper) to the new pattern. The original getConfiguration()
is deprecated.
None
- Test: No need to test - pure refactor, all callers produce identical Configuration objects
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add PluginDrivenExternalCatalog adapter and SPI-based CatalogFactory
Issue Number: close #xxx
Problem Summary: The CatalogFactory uses hardcoded switch-case to create catalogs. This commit
introduces a SPI-first creation path: ConnectorFactory.createConnector() is tried before falling
back to the built-in switch-case. A new PluginDrivenExternalCatalog bridges SPI Connector instances
with the existing ExternalCatalog hierarchy, enabling third-party catalog plugins.
None
- Test: No need to test — no SPI connector plugins exist yet, all catalogs still use the fallback switch-case path. Behavior is unchanged.
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add fe-connector-es module with SPI metadata implementation
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Creates the first real connector plugin module (Elasticsearch)
implementing the connector SPI interfaces. This validates the entire SPI design
with a working metadata-only implementation. The ES scan path remains in fe-core
temporarily — only metadata operations (list databases, list tables, get schema)
are handled via the SPI.
The module includes:
- EsConnectorProvider: SPI entry point discovered via ServiceLoader
- EsConnector: Main connector, creates metadata instances
- EsConnectorMetadata: Lists ES indices as tables, parses mappings to schema
- EsConnectorRestClient: HTTP client adapted from fe-core (no fe-core deps)
- EsTypeMapping: ES type → ConnectorType mapping
- EsTableHandle: Opaque handle for ES indices
- EsConnectorProperties: Property constants and compatibility processing
- Plugin assembly ZIP descriptor for runtime deployment
None
- Test: No need to test — metadata-only SPI implementation, fe-core ES path unchanged
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add shard routing and metadata pipeline to fe-connector-es
Issue Number: close #xxx
Problem Summary: The initial fe-connector-es module (Phase 1) only supported basic
metadata operations (list databases, list tables, get schema). This commit adds
Phase 2 capabilities: shard routing discovery, node info resolution, mapping field
analysis (keyword sniff, doc_value, date compat), and full metadata orchestration.
These are prerequisites for the future ConnectorScanPlanProvider which will generate
TScanRangeLocations from the plugin side.
New files (8):
- EsMajorVersion: ES version detection (adapted from fe-core, zero fe-core deps)
- EsNodeInfo: ES node with HTTP address (host+port instead of TNetworkAddress)
- EsShardRouting: Shard routing entry (host+port instead of TNetworkAddress)
- EsShardPartitions: Shard partition map with _search_shards JSON parsing
- EsFieldContext: Field context holder (fetchFields, docValue, dateCompat maps)
- EsMappingUtils: Mapping parsing + field resolution (from EsUtil + MappingPhase)
- EsMetadataState: Full metadata state (adapted from SearchContext)
- EsMetadataFetcher: Metadata fetch orchestrator (adapted from EsMetaStateTracker)
Enhanced files (3):
- EsConnectorRestClient: added searchShards(), getHttpNodes(), get() methods
- EsConnectorMetadata: added fetchMetadataState() for full metadata pipeline
- EsConnectorProperties: added MAPPING_TYPE constant
None
- Test: No need to test — plugin module only, not activated in production
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-trino plugin module (Step 7 Phase 1)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Trino Connector metadata logic from fe-core into an
independent plugin module fe-connector-trino, continuing the Catalog SPI
modularization. This phase migrates the metadata-only layer (schema discovery,
type mapping, Trino plugin bootstrap) while leaving scan/predicate code in
fe-core for Step 16.
Key files:
- TrinoConnectorProvider: SPI entry point (type="trino-connector")
- TrinoDorisConnector: Lazy init, wraps Trino Connector + Session
- TrinoBootstrap: Singleton Trino plugin infrastructure init
- TrinoConnectorDorisMetadata: ConnectorMetadata impl (list/get/schema)
- TrinoTypeMapping: Trino SPI types -> ConnectorType (15+ mappings)
- TrinoPluginManager/TrinoServicesProvider: Adapted from fe-common
Design decisions:
- Copied fe-common classes (~290 lines each) into plugin rather than depending
on fe-common (which has heavy transitive deps like hadoop-common)
- TrinoBootstrap uses singleton pattern for plugin loading, per-catalog
connector creation
- Plugin dir resolution: property -> DORIS_HOME/plugins/connectors -> fallback
None
- Test: No need to test (metadata-only extraction, fe-core code unchanged)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-maxcompute plugin module (Step 8)
Problem Summary: Extract MaxCompute connector metadata code into an independent
plugin module as part of the Catalog SPI modularization effort.
Changes:
- Create fe-connector-maxcompute module with 11 Java files + build config
- MaxComputeConnectorProvider (SPI entry, type="max_compute")
- MaxComputeDorisConnector (Odps client lifecycle, lazy init)
- MaxComputeConnectorMetadata (listDatabases, listTables, getTableHandle, getTableSchema)
- MCTypeMapping (15+ ODPS type -> ConnectorType mappings including nested ARRAY/MAP/STRUCT)
- McStructureHelper (namespace schema abstraction, copied from fe-core)
- MCConnectorClientFactory, MCConnectorEndpoint, MCConnectorProperties (utilities)
- MaxComputeTableHandle (opaque handle wrapping ODPS Table + TableIdentifier)
- Enhance ConnectorType API with factory methods (of, arrayOf, mapOf, structOf)
- Make Connector.getScanPlanProvider() a default method (scan planning is optional for Phase 1)
Phase 1 scope: metadata-only. ScanNode, Transaction, DDL operations stay in fe-core.
None
- Test: No need to test - Phase 1 metadata extraction, SPI plugin not yet activated
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-jdbc plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extracts JDBC connector metadata logic into an independent
fe-connector-jdbc plugin module as part of the Catalog SPI modularization.
This is Step 10 of the connector SPI plan.
The module implements:
- JdbcConnectorProvider (SPI entry point, type="jdbc")
- JdbcDorisConnector (manages HikariCP connection pool lifecycle)
- JdbcConnectorMetadata (delegates to JdbcConnectorClient for metadata discovery)
- JdbcConnectorClient base class with factory method dispatching to 10 DB-specific subclasses
- All 10 DB-specific clients: MySQL, PostgreSQL, Oracle, ClickHouse, SQLServer,
SAP HANA, Trino, OceanBase, DB2, GBase
- JdbcFieldInfo (column metadata, adapted from JdbcFieldSchema)
- JdbcDbType enum, JdbcTableHandle, JdbcConnectorProperties
Each subclass maps JDBC types to ConnectorType (the SPI type system) instead
of fe-core Type, achieving zero fe-core dependency.
None
- Test: No need to test (Phase 1 metadata-only extraction, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Extract HMS shared module to fe-connector-hms
Issue Number: close #xxx
Problem Summary: The HMS (Hive MetaStore) client code in fe-core is deeply
coupled to internal types (Column, Config, ExecutionAuthenticator, etc.), making
it impossible to directly extract. This commit creates a new clean HMS client
library module (fe-connector-hms) using SPI types, which will be shared by
future fe-connector-hive and fe-connector-hudi plugins.
Key design decisions:
- New clean interfaces using ConnectorType/ConnectorColumn (not code extraction)
- Connection-pooled Thrift client with taint-and-destroy error handling
- Supports HMS/DLF/Glue metastore types via string-based class dispatch
- Auth via functional interface (AuthAction) replacing fe-core ExecutionAuthenticator
- HmsTypeMapping mirrors HiveMetaStoreClientHelper logic but returns ConnectorType
Module contents (9 Java files, ~1440 lines):
- HmsClient: Clean interface for HMS operations
- ThriftHmsClient: Pooled Thrift implementation with ClassLoader safety
- HmsClientConfig/HmsConfHelper: Configuration DTOs and HiveConf creation
- HmsTableInfo/HmsDatabaseInfo/HmsPartitionInfo: Immutable DTOs
- HmsTypeMapping: Hive type string to ConnectorType mapping
- HmsClientException: Runtime exception class
None
- Test: No need to test - library module with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-hive plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Hive connector metadata operations into an independent
fe-connector-hive plugin module as Phase 1 of the Catalog SPI modularization.
This creates the fe-connector-hive plugin that:
- Implements ConnectorProvider (type="hms") for SPI-based catalog creation
- Provides HiveConnectorMetadata with listDatabases, listTables, getTableHandle,
getTableSchema operations via the shared fe-connector-hms client
- Detects table formats (HIVE/HUDI/ICEBERG) from HMS table metadata
- Supports configurable type mapping options (binary-as-string, timestamp-tz)
- Manages HmsClient lifecycle with lazy initialization and proper shutdown
Files: 10 new files (816 lines), 2 modified files
Dependencies: fe-connector-spi + fe-connector-hms + log4j-api
None
- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-paimon plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Paimon connector metadata operations into an independent
fe-connector-paimon plugin module as Phase 1 of the Catalog SPI modularization.
This creates the fe-connector-paimon plugin that:
- Implements ConnectorProvider (type="paimon") for SPI-based catalog creation
- Creates Paimon Catalog instances directly via Paimon SDK (Options + CatalogContext
+ CatalogFactory) supporting all backends: filesystem, HMS, DLF, REST, JDBC
- Provides PaimonConnectorMetadata with listDatabases, listTables, getTableHandle,
getTableSchema operations using the Paimon Catalog API
- Includes PaimonTypeMapping that converts Paimon DataType to ConnectorType,
mirroring existing PaimonUtil.paimonTypeToDorisType logic
- Supports configurable type mapping options (binary-as-varbinary, timestamp-tz)
- Manages Paimon Catalog lifecycle with lazy init and proper shutdown
Files: 8 new files (805 lines), 1 modified file
Dependencies: fe-connector-spi + paimon-core + log4j-api
None
- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add fe-connector-hudi Phase 1 plugin module
Issue Number: close #xxx
Problem Summary: Extract Hudi connector metadata operations into an
independent SPI plugin module (fe-connector-hudi) as part of the
catalog connector SPI modularization effort.
The Hudi connector plugin provides Phase 1 read-only metadata:
- List databases and tables via HMS (shared fe-connector-hms client)
- Get table schema from HoodieTableMetaClient Avro schema
(authoritative for schema-evolved Hudi tables)
- Detect Hudi table type (COW vs MOR) from input format
- Avro-to-ConnectorType mapping (HudiTypeMapping)
Dependencies: fe-connector-spi + fe-connector-hms + hudi-common +
hudi-hadoop-mr.
Scan planning, incremental query, statistics, and event processing
remain in fe-core temporarily.
None
- Test: No need to test (Phase 1 SPI module, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-iceberg Phase 1 — metadata-only SPI plugin
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Step 15 of the Catalog Connector SPI modularization. Extracts the Iceberg
connector into an independent plugin module (fe-connector-iceberg) that
provides read-only metadata operations through the Connector SPI.
Phase 1 implements:
- IcebergConnectorProvider: SPI entry point, type="iceberg"
- IcebergConnector: Lifecycle management, creates Iceberg SDK Catalog via
CatalogUtil.buildIcebergCatalog() — supports all 7 backends (REST, HMS,
Glue, DLF, JDBC, Hadoop, S3Tables)
- IcebergConnectorMetadata: listDatabaseNames (SupportsNamespaces), listTableNames,
getTableHandle, getTableSchema via Iceberg SDK Catalog API
- IcebergTypeMapping: Iceberg types → ConnectorType (BOOLEAN, INT, LONG, FLOAT,
DOUBLE, STRING, UUID/BINARY→VARBINARY or STRING, DECIMAL, DATE, TIMESTAMP,
LIST, MAP, STRUCT) with enableMappingVarbinary and enableMappingTimestampTz flags
- IcebergTableHandle: Opaque handle carrying db+table coordinates
- IcebergConnectorProperties: Property key constants
Key design decisions:
- No fe-connector-hms dependency: Iceberg SDK HiveCatalog handles HMS internally
- Property-driven catalog creation: CatalogUtil.buildIcebergCatalog() handles
all backends, subclass dispatch is just setting catalog-impl
- Hadoop Configuration built from user properties (hadoop.*, fs.*, dfs.*, hive.*)
6 Java files, ~770 lines added.
None
- Test: No need to test — Phase 1 metadata-only SPI module, not yet wired into fe-core runtime
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add PluginDrivenExternalTable schema support and GSON registration
Issue Number: close #xxx
Problem Summary:
PluginDrivenExternalTable was a minimal stub with no schema retrieval capability.
Plugin-driven catalogs could list databases and tables but could not fetch table
schemas, making them non-functional for queries. Additionally, the PluginDriven*
types were not registered in GsonUtils, preventing GSON serialization/deserialization
of plugin-driven catalog metadata for FE persistence.
This commit implements Step 18 of the Catalog Connector SPI plan:
1. **ConnectorColumnConverter** (new): Converts between the connector SPI type system
(ConnectorColumn/ConnectorType) and Doris internal types (Column/Type). Handles
all scalar types plus complex types (ARRAY, MAP, STRUCT) recursively, with proper
precision/scale handling for CHAR, VARCHAR, DECIMAL, and DATETIMEV2.
2. **PluginDrivenExternalTable**: Overrides initSchema() to fetch table schema from
the connector SPI. Uses ConnectorMetadata.getTableHandle() + getTableSchema(),
then converts via ConnectorColumnConverter.
3. **PluginDrivenExternalCatalog**: Makes buildConnectorSession() package-private so
PluginDrivenExternalTable can build sessions during schema init.
4. **GsonUtils**: Registers PluginDrivenExternalCatalog, PluginDrivenExternalDatabase,
and PluginDrivenExternalTable in the respective RuntimeTypeAdapterFactory instances
for proper GSON serialization/deserialization.
None
- Test: No need to test - infrastructure code with no active connector plugins yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorExpression framework and filter pushdown types
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Step 19 of the Catalog SPI modularization. Replaces the
placeholder ConnectorExpression class with a proper expression tree hierarchy
for cross-SPI filter/projection pushdown negotiation. Adds the
ExprToConnectorExpressionConverter in fe-core to convert Doris Expr trees
into ConnectorExpression trees at the SPI boundary.
Changes:
- ConnectorExpression: converted from final class to interface with
getExprType() and getChildren() methods
- ConnectorExprType: discriminator enum (COLUMN_REF, LITERAL, COMPARISON,
AND, OR, NOT, IN, BETWEEN, IS_NULL, LIKE, FUNCTION_CALL)
- 11 concrete expression types: ConnectorColumnRef, ConnectorLiteral,
ConnectorComparison, ConnectorAnd, ConnectorOr, ConnectorNot,
ConnectorIn, ConnectorBetween, ConnectorIsNull, ConnectorLike,
ConnectorFunctionCall
- ConnectorRange: range bound with low/high inclusive/exclusive endpoints
- ConnectorDomain: per-column domain (union of ranges + null handling)
for fast partition pruning
- ConnectorFilterConstraint: redesigned to carry full expression tree +
per-column domain map (replaces old flat conjuncts list)
- ExprToConnectorExpressionConverter: converts Doris Expr tree to
ConnectorExpression tree, plus Type to ConnectorType reverse mapping
None
- Test: No need to test - pure API/SPI type definitions and converter
with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorScanPlanProvider and PluginDrivenScanNode for connector SPI scan planning
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Plugin-driven catalogs need a generic scan node that delegates
scan planning to the connector SPI, rather than requiring connector-specific
ScanNode subclasses in fe-core.
This commit adds Step 20 of the Catalog SPI modularization:
**fe-connector-api enhancements:**
- ConnectorScanRangeType enum: FILE_SCAN, JDBC_SCAN, ES_SCAN, REMOTE_OLAP_SCAN, CUSTOM
- ConnectorDeleteFile: delete file descriptor for Iceberg MOR tables
- ConnectorScanRange: added getRangeType() (mandatory), getFileFormat(),
getFileSize(), getModificationTime(), getPartitionValues(), getDeleteFiles()
- ConnectorScanPlanProvider: added estimateScanRangeCount() default method
**fe-core additions:**
- PluginDrivenSplit: wraps ConnectorScanRange in FileSplit for the
FileQueryScanNode pipeline
- PluginDrivenScanNode: extends FileQueryScanNode, delegates scan planning
to ConnectorScanPlanProvider, uses FORMAT_JNI for BE execution
- ExprToConnectorExpressionConverter: added convertConjuncts() utility
- PhysicalPlanTranslator: dispatch to PluginDrivenScanNode for
PluginDrivenExternalTable instances
None
- Test: No need to test — SPI infrastructure, no actual connectors activate this path yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorWriteOps & PluginDrivenTableSink (Step 21)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Implements Step 21 of the Catalog SPI modularization plan.
Adds the write operation SPI interfaces and the generic table sink for
plugin-driven external tables.
**SPI layer (fe-connector-api):**
- Enhanced ConnectorWriteOps with full write lifecycle: getWriteConfig,
beginInsert/finishInsert/abortInsert (with column list), beginDelete/
finishDelete/abortDelete, beginMerge/finishMerge/abortMerge
- Added ConnectorWriteType enum (FILE_WRITE, JDBC_WRITE, REMOTE_OLAP_WRITE,
CUSTOM)
- Added ConnectorWriteConfig value object with builder pattern, carrying
write type, file format, compression, location, partition columns, and
generic properties
- Added ConnectorDeleteHandle and ConnectorMergeHandle marker interfaces
**Engine layer (fe-core):**
- Created PluginDrivenTableSink extending BaseExternalTableDataSink
- Constructs TDataSink based on ConnectorWriteConfig.getWriteType():
FILE_WRITE -> THiveTableSink, JDBC_WRITE -> TJdbcTableSink
- Property-driven Thrift construction using well-known keys from
ConnectorWriteConfig.properties
None
- Test: No need to test - infrastructure interfaces with no runtime behavior change
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add generic Nereids plan nodes for plugin-driven INSERT pipeline
Issue Number: close #xxx
Problem Summary: PluginDrivenExternalCatalog-backed tables cannot participate
in the full Nereids INSERT pipeline because there are no generic plan node
classes for them. Each existing connector (Hive, Iceberg, JDBC, etc.) has its
own typed plan nodes, but the plugin-driven generic path has none.
This commit adds the complete Nereids INSERT pipeline for plugin-driven
catalogs:
1. **Plan nodes**: UnboundConnectorTableSink, LogicalConnectorTableSink,
PhysicalConnectorTableSink — generic nodes that work with any
PluginDrivenExternalCatalog without connector-specific plan classes.
2. **Optimizer rules**: BINDING_INSERT_CONNECTOR_TABLE (BindSink),
LogicalConnectorTableSinkToPhysicalConnectorTableSink (implementation),
ExpressionRewrite for LogicalConnectorTableSink.
3. **Insert executor**: PluginDrivenInsertExecutor delegates begin/commit/
abort to ConnectorWriteOps SPI. PluginDrivenInsertCommandContext provides
the context wrapper.
4. **Transaction manager**: PluginDrivenTransactionManager provides
lightweight transaction lifecycle bookkeeping; actual commit/rollback
is handled by ConnectorWriteOps in the insert executor.
5. **Wiring**: PlanType enum entries, RuleType entries, SinkVisitor methods,
RuleSet registration, UnboundTableSinkCreator dispatch,
PhysicalPlanTranslator visitor, InsertIntoTableCommand executor selection.
None
- Test: No need to test — infrastructure code, no connector plugins exercise
this path yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Activate CatalogFactory SPI path with build integration and connectivity testing
Issue Number: close #xxx
Problem Summary: The connector SPI plugins were built but not deployed to
the output directory, making them undiscoverable at runtime. Additionally,
PluginDrivenExternalCatalog lacked connectivity testing support.
This commit:
1. Adds connector module compilation to build.sh FE module list
2. Deploys connector plugin ZIPs to output/fe/plugins/connector/<name>/
(mirroring the filesystem plugin deployment pattern)
3. Creates ConnectorTestResult value object in fe-connector-api
4. Adds Connector.testConnection(session) default method
5. Overrides checkWhenCreating() in PluginDrivenExternalCatalog to delegate
connectivity testing to the connector SPI
6. Adds ConnectorFactory.getRegisteredTypes() for diagnostics
7. Enhances logging in CatalogFactory and Env for SPI path transparency
None
- Test: No need to test (build infrastructure + SPI wiring, no behavioral change)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 24: ES full migration to connector SPI
Issue Number: close #xxx
Problem Summary: Migrate the ES (Elasticsearch) connector scan path from
legacy fe-core EsScanNode to the plugin-driven connector SPI. This enables
the ES connector to run as an isolated plugin while producing the same
ES_HTTP_SCAN_NODE Thrift types expected by BE.
Key changes:
- Enhanced ConnectorScanPlanProvider with getScanRangeType(),
getScanNodeProperties(), and getScanNodeMapProperties() to support
non-file scan types (ES, JDBC) that need custom Thrift structures
- Created EsQueryDslBuilder: ports QueryBuilders from legacy Expr-based
filter conversion to ConnectorExpression-based ES Query DSL generation
- Created EsScanPlanProvider: generates per-shard scan ranges with ES
host routing, query DSL, auth info, and field context maps
- Created EsScanRange: ConnectorScanRange impl for ES shard routing
- Created PluginDrivenEsScanNode: extends ExternalScanNode to produce
TPlanNodeType.ES_HTTP_SCAN_NODE with TEsScanNode/TEsScanRange Thrift
- Wired EsConnector.getScanPlanProvider() to return real provider
- Added EsConnector.testConnection() for catalog connectivity testing
- Fixed BindRelation missing PLUGIN_EXTERNAL_TABLE case (critical:
without this fix, no plugin-driven table can be queried)
- PhysicalPlanTranslator dispatches to PluginDrivenEsScanNode when
getScanRangeType() == ES_SCAN
- Removed "es" case from CatalogFactory switch (SPI handles it)
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 25: JDBC full migration (scan only) to connector SPI
Issue Number: close #xxx
Problem Summary: Migrate JDBC connector scan path to the connector SPI
framework, enabling JDBC catalogs to run through the PluginDrivenScanNode
pipeline instead of the legacy JdbcScanNode code path.
Key changes:
- JdbcQueryBuilder: converts ConnectorExpression to SQL WHERE clauses with
DB-specific date formatting (Oracle to_date, Trino date literal, SQL Server
CONVERT) and LIMIT syntax (MySQL LIMIT, Oracle ROWNUM, SQL Server TOP N)
- JdbcScanRange: ConnectorScanRange with table_format_type="jdbc" and Builder
pattern for all BE-expected jdbc_params (url, user, password, driver, query_sql,
connection pool settings)
- JdbcScanPlanProvider: builds SQL query via JdbcQueryBuilder, returns 1 scan
range with all JDBC connection parameters
- JdbcColumnHandle: carries localName + remoteName for SELECT clause column
quoting via JdbcIdentifierQuoter
- JdbcConnectorMetadata.getColumnHandles(): returns column handle map for
PluginDrivenScanNode column selection
- JdbcDorisConnector: wires getScanPlanProvider() and testConnection()
- ConnectorScanRange.getTableFormatType(): default "plugin_driven", JDBC
overrides to "jdbc" so BE routes to JdbcJniReader
- ConnectorScanPlanProvider: 5-arg planScan() overload with limit parameter
for JDBC LIMIT pushdown
- PluginDrivenScanNode: uses scanRange.getTableFormatType() instead of
hardcoded constant; builds column handles via metadata.getColumnHandles();
passes limit to 5-arg planScan()
- CatalogFactory: removed "jdbc" case from fallback switch (all JDBC catalogs
now go through SPI path)
None
- Test: Manual test (compilation + checkstyle verified)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 26: MaxCompute full migration (scan only) to connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Migrates MaxCompute scan planning from fe-core to the
fe-connector-maxcompute plugin module. This removes the "max_compute" fallback
case from CatalogFactory, routing MaxCompute catalog creation exclusively
through the SPI connector path (PluginDrivenExternalCatalog).
Key changes:
- MaxComputeScanPlanProvider: creates ODPS TableBatchReadSession, generates
splits (byte_size and row_offset strategies), supports limit optimization
- MaxComputePredicateConverter: converts ConnectorExpression to ODPS Predicate
(CompoundPredicate, RawPredicate, UnaryPredicate) with datetime timezone
conversion
- MaxComputeScanRange: ConnectorScanRange carrying serialized session, split
params, and timeout config with tableFormatType="max_compute"
- MaxComputeColumnHandle: tracks partition vs data column distinction
- MCConnectorEndpoint: region-to-timezone mapping for datetime pushdown
- PluginDrivenScanNode: format-type dispatch in setScanParams() -
"max_compute" builds TMaxComputeFileDesc, others use generic jdbc_params map
- CatalogFactory: removed "max_compute" fallback case and
MaxComputeExternalCatalog import
- Added odps-sdk-table-api dependency for table read session API
None
- Test: No need to test - structural migration only, MaxCompute not available
in dev environment
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Migrate Trino connector scan planning to connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Trino connector scan planning was still handled by the legacy
TrinoConnectorScanNode in fe-core with direct Trino SPI type dependencies.
This commit migrates the full scan flow to the connector SPI plugin module
(fe-connector-trino), keeping all Trino SPI types inside the plugin boundary.
Key changes:
- TrinoScanPlanProvider: core scan planning (beginQuery → applyFilter/Limit/
Projection → getSplits → JSON-serialize → TrinoScanRange list → cleanupQuery)
- TrinoJsonSerializer: ObjectMapperProvider + HandleJsonModule + BlockJsonSerde
for serializing Trino SPI objects to JSON
- TrinoPredicateConverter: ConnectorExpression → Trino TupleDomain<ColumnHandle>
- TrinoScanRange: ConnectorScanRange carrying pre-serialized JSON properties
- TrinoColumnHandle: ConnectorColumnHandle for projection pushdown
- PluginDrivenScanNode: "trino_connector" dispatch filling TTrinoConnectorFileDesc
from properties map (no Trino types needed in fe-core)
- CatalogFactory: removed "trino-connector" legacy fallback case
- TrinoBootstrap: added getHandleResolver()/getTypeRegistry() accessors
- TrinoConnectorDorisMetadata: added getColumnHandles() method
- TrinoDorisConnector: replaced stub with real getScanPlanProvider() + testConnection()
None
- Test: No need to test - structural migration, scan logic ported from existing
TrinoConnectorScanNode with no behavioral changes
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Hive scan planning to connector plugin (Step 31)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Adds scan planning capability to the Hive connector plugin
module (fe-connector-hive), enabling file-based scan range generation
through the ConnectorScanPlanProvider SPI. This is the scan-only phase of
the Hive full migration to the connector plugin architecture.
New files in fe-connector-hive:
- HiveScanPlanProvider: Core scan planning - partition resolution via HMS,
file listing via Hadoop FS, file splitting by configurable target size
- HiveColumnHandle: ConnectorColumnHandle wrapping column name/type/isPartKey
- HiveFileFormat: Enum mapping InputFormat/SerDe classes to format strings
- HiveScanRange: ConnectorScanRange with Builder for file split descriptors
- HiveTextProperties: Extracts text format props from HMS SerDe parameters
Enhanced existing files:
- HiveTableHandle: Rewritten with Builder pattern, now carries inputFormat,
serializationLib, location, partitionKeyNames, sdParameters, tableParameters,
and prunedPartitions for scan planning
- HiveConnectorMetadata: Added getColumnHandles(), applyFilter() with
partition pruning (equality + IN predicates on partition columns)
- HiveConnector: Added getScanPlanProvider() returning HiveScanPlanProvider
- HmsTableInfo: Added sdParameters field for SerDe parameters
- ThriftHmsClient: Populates sdParameters from SerDeInfo
- PluginDrivenScanNode: Major enhancement with property-driven overrides for
getFileFormatType, getPathPartitionKeys, getLocationProperties, getFileAttributes,
plus hive/transactional_hive dispatch in setScanParams
Scope: Non-ACID Hive tables (Parquet, ORC, Text). ACID tables, file listing
cache, and table sampling are deferred to future steps.
None
- Test: No need to test (scan planning code path not yet wired to production
query flow; requires HMS + Hadoop cluster for integration testing)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Hudi scan planning to connector plugin (Step 32)
Issue Number: close #xxx
Problem Summary: Adds scan planning capability to the fe-connector-hudi plugin
module, following the same pattern established in Step 31 (Hive scan). This
enables the Hudi connector plugin to generate scan ranges that BE can consume,
supporting both COW (native Parquet/ORC reader) and MOR (JNI reader with delta
log merging) table types.
New files:
- HudiScanPlanProvider: Core scan planning — builds MetaClient, resolves
partitions via HoodieTableMetadata API, generates COW splits (base files
only) and MOR splits (base + delta logs with dynamic native downgrade)
- HudiScanRange: ConnectorScanRange implementation with Builder pattern,
carrying both native reader fields and JNI metadata (instant_time, serde,
delta_logs, column_names/types)
- HudiColumnHandle: Column handle with name, typeName, isPartitionKey
Modified files:
- HudiTableHandle: Rewritten with Builder pattern, added scan-related fields
(inputFormat, serdeLib, partitionKeyNames, tableParameters, prunedPaths)
- HudiConnectorMetadata: Enhanced getTableHandle with scan fields from HMS,
added getColumnHandles() and applyFilter() for partition pruning
- HudiConnector: Added getScanPlanProvider()
- PluginDrivenScanNode: Added "hudi" case in setScanParams dispatch with
setHudiParams() method creating THudiFileDesc for both native/JNI paths
None
- Test: No need to test — plugin scan planning mirrors existing HudiScanNode
logic; will be validated when end-to-end regression tests are enabled
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Paimon scan planning to connector plugin (Step 33)
Problem Summary: Migrate Paimon scan planning logic from fe-core PaimonScanNode
to the fe-connector-paimon plugin module, following the same pattern established
by Hive (Step 31) and Hudi (Step 32) scan migrations.
This step adds:
- PaimonColumnHandle: Column handle with name and field index in RowType
- PaimonPredicateConverter: Converts ConnectorExpression to Paimon Predicate
(PredicateBuilder-based, supports AND/OR/comparison/IN/IS NULL/LIKE prefix)
- PaimonScanRange: Builder-pattern scan range supporting JNI, native, and
COUNT pushdown paths with dual serialization
- PaimonScanPlanProvider: Core scan planning using Paimon SDK (ReadBuilder →
TableScan → Split), converting to ConnectorScanRange with JNI/native dispatch
- Modified PaimonTableHandle to carry transient Table reference
- Modified PaimonConnectorMetadata to store Table in handle and add getColumnHandles()
- Modified PaimonConnector to add getScanPlanProvider()
- Modified PluginDrivenScanNode to add paimon dispatch (setPaimonParams),
getSerializedTable() override, and scan-level paimon params (predicate + options)
None
- Test: No need to test (compile-only migration, runtime wiring not yet active)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove old internal ES table code and clean up references
Issue Number: close #xxx
Problem Summary: The old internal Elasticsearch table implementation (EsScanNode,
EsExternalCatalog, EsExternalTable, etc.) has been superseded by the ES Catalog
connector plugin. This commit removes the legacy ES code from fe-core and cleans
up all references throughout the codebase.
Changes:
- Delete 21 old ES datasource files from datasource/es/
- Delete 3 nereids plan classes (LogicalEsScan, PhysicalEsScan, LogicalEsScanToPhysicalEsScan)
- Delete 6 old ES test files from external/elasticsearch/
- Clean up ES references in 13+ files (Env.java, ExternalCatalog.java, ESCatalogAction.java,
PhysicalPlanTranslator.java, BindRelation.java, RuleSet.java, RelationVisitor.java,
StatsCalculator.java, CostModel.java, ChildOutputPropertyDeriver.java,
TopnFilterPushDownVisitor.java, GsonUtils.java, TableIf.java, etc.)
- Preserve EsTable.java and EsResource.java for persistence compatibility
- Add ES stub in SHOW CREATE TABLE for deprecation notice
None
- Test: Manual test - fe-core compiles successfully, ES connector tests pass
- Behavior changed: Yes - old internal ES table queries will no longer work;
users should use ES Catalog instead
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Add metadata methods to JDBC connector for migration parity
Issue Number: close #xxx
Problem Summary: During JDBC connector modular migration from fe-core to
fe-connector-jdbc, several metadata methods were missing in the new connector
client and SPI interfaces. This commit adds the missing methods to achieve
parity with the old JdbcClient.
None
- Test: No need to test - pure method additions with no behavior change,
existing integration paths still use old JdbcClient
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Implement JDBC write path through connector plugin pipeline
Issue Number: close #xxx
Problem Summary: The JDBC INSERT INTO write path was not wired through the
new connector plugin pipeline. This commit implements the complete JDBC write
path so that INSERT INTO on PluginDriven JDBC tables produces the correct
TJdbcTableSink thrift structure, matching the old JdbcTableSink behavior.
Changes:
- JdbcConnectorMetadata now implements ConnectorWriteOps with getWriteConfig(),
beginInsert(), finishInsert() — builds INSERT SQL via JdbcIdentifierQuoter
and populates all JDBC connection/pool properties
- JdbcIdentifierQuoter gains buildInsertSql() for parameterized INSERT SQL
- PluginDrivenTableSink.bindJdbcWriteSink() now populates ALL TJdbcTable
fields: catalog_id, driver_checksum, table_name, resource_name, and all
connection pool settings; also sets TOdbcTableType on TJdbcTableSink
- PluginDrivenInsertExecutor resolves write type from connector metadata and
returns TransactionType.JDBC for JDBC writes (was hardcoded to HMS)
- PhysicalPlanTranslator passes actual column list to getWriteConfig()
(was passing empty list)
- ConnectorSessionBuilder now propagates enable_odbc_transcation session var
- Explain output enhanced for JDBC write sinks (shows table type, SQL, txn)
None
- Test: No need to test - infrastructure wiring, no end-to-end JDBC catalog
available in unit test environment; will be tested with regression tests
after full migration
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Port JDBC function pushdown and ClickHouse FINAL to connector module
Issue Number: close #xxx
Problem Summary:
The JDBC connector module (fe-connector-jdbc) lacked function pushdown support
and ClickHouse FINAL query support. These features existed only in the old
fe-core JdbcScanNode code path. This commit ports them to the connector SPI
pipeline so PluginDrivenScanNode-based JDBC scans have feature parity.
Key changes:
- New JdbcFunctionPushdownConfig: per-DB function whitelist/blacklist/replacement
rules with time arithmetic rewriting and JSON-configurable overrides
- Extended JdbcQueryBuilder: ConnectorFunctionCall/Like/Between to SQL conversion,
conjunct pushdown guards (Oracle NULL, cast, function-containing expressions),
ClickHouse SETTINGS final=1 support
- Updated JdbcScanPlanProvider to create pushdown config and pass session props
- Plumbed 4 session variables through ConnectorSessionBuilder:
enable_ext_func_pred_pushdown, jdbc_clickhouse_query_final,
enable_jdbc_oracle_null_predicate_push_down, enable_jdbc_cast_predicate_push_down
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Route JDBC TVF and CallExecuteStmt through connector SPI
Issue Number: close #xxx
Problem Summary:
JdbcQueryTableValueFunction and CallExecuteStmtFunc only supported the old
JdbcExternalCatalog code path. Newly created JDBC catalogs use
PluginDrivenExternalCatalog via the SPI connector, so these features were
broken for new catalogs.
Key changes:
- New PassthroughQueryTableHandle in connector API for raw query TVF scans
- JdbcQueryTableValueFunction: dual path — uses ConnectorMetadata for schema
discovery and PluginDrivenScanNode for scan planning when catalog is
PluginDrivenExternalCatalog
- CallExecuteStmtFunc: routes to connector metadata.executeStmt() for
PluginDrivenExternalCatalog, falls back to old JdbcExternalCatalog path
- QueryTableValueFunction factory: detects PluginDrivenExternalCatalog
- JdbcScanPlanProvider: handles PassthroughQueryTableHandle for TVF queries
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Migrate JDBC identifier mapping to connector module
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
The JDBC identifier mapping logic (lower_case_meta_names, meta_names_mapping,
lower_case_table_names) was tightly coupled to fe-core via JdbcIdentifierMapping
which depends on Jackson ObjectMapper. This commit migrates the functionality
to the connector-jdbc module as part of the SPI modularization effort.
Key changes:
- Add ConnectorIdentifierOps SPI interface in connector-api with 3 name mapping
methods (fromRemoteDatabaseName, fromRemoteTableName, fromRemoteColumnName)
- ConnectorMetadata now extends ConnectorIdentifierOps
- Create JdbcIdentifierMapper in connector-jdbc: pure Java reimplementation
with regex-based JSON parsing (no Jackson dependency), full validation with
case-conflict detection
- Wire identifier mapping through getColumnHandles() and getWriteConfig() so
column names are properly mapped in both read and write paths
- Add lower_case_table_names from GlobalVariable to ConnectorSessionBuilder
- PluginDrivenExternalCatalog overrides fromRemoteDatabaseName/TableName to
delegate to connector metadata
- PluginDrivenExternalTable.initSchema() applies fromRemoteColumnName mapping
None
- Test: Manual test / Compilation verified
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove old JDBC connector code after migration to plugin-driven architecture
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: After migrating JDBC connector to fe-connector-jdbc module with
plugin-driven architecture (PluginDrivenExternalCatalog/Database/Table), the old
JDBC code in fe-core/datasource/jdbc/ is dead code. This commit:
1. Adds Gson compatible subtype adapters so persisted metadata with old class names
(JdbcExternalCatalog/Database/Table) deserializes correctly as PluginDriven* classes
2. Deletes 21 old JDBC files: JdbcExternalCatalog, JdbcExternalDatabase,
JdbcExternalTable, JdbcNameUtil, JdbcSchemaCacheValue, JdbcTableSink, JdbcScanNode,
JdbcSplit, JdbcFunctionPushDownRule, IdentifierMapping, JdbcIdentifierMapping,
UnboundJdbcTableSink, LogicalJdbcTableSink, PhysicalJdbcTableSink,
JdbcInsertExecutor, JdbcInsertCommandContext,
LogicalJdbcScanToPhysicalJdbcScan, LogicalJdbcTableSinkToPhysicalJdbcTableSink,
LogicalJdbcScan, PhysicalJdbcScan
3. Cleans up all references in 23 files across nereids visitors, derivers, rules,
plan translator, expression rewrite, and external function rules
4. Inlines function pushdown constants from deleted JdbcFunctionPushDownRule into
ExternalFunctionRules
JdbcClient hierarchy (16 files) is kept for CDC binlog functionality.
None
- Test: No need to test - pure code deletion of dead code after migration
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix connector plugin packaging and ES scan node bugs
Issue Number: close #xxx
Problem Summary:
Three issues causing CI regression test failures (build 926124):
1. **JDBC/Hive/MaxCompute plugin JARs not discovered at runtime**
The assembly descriptors for these connectors used `includeBaseDirectory=true`,
creating a nested subdirectory inside the zip (e.g., `fe-connector-jdbc/lib/*.jar`).
When build.sh unzips into `plugins/connector/jdbc/`, the actual path becomes
`plugins/connector/jdbc/fe-connector-jdbc/lib/*.jar`, but DirectoryPluginRuntimeManager
expects JARs at `plugins/connector/jdbc/*.jar` and `plugins/connector/jdbc/lib/*.jar`.
Fix: Set `includeBaseDirectory=false` and place plugin JAR at root + deps in lib/,
matching the working ES/iceberg/paimon assembly layout.
2. **ES scan node URL parsing fails for http:// prefixed hosts**
PluginDrivenEsScanNode.convertToThrift() split host strings by ":" naively,
so `http://172.16.0.98:29200` was parsed as host="http", port="//172.16.0.98"
causing NumberFormatException. Fix: Strip scheme prefix before parsing host:port.
3. **ES JSONB type not recognized by ConnectorColumnConverter**
ES connector returns JSONB type but ScalarType.createType() only handles "JSON".
ConnectorColumnConverter now maps JSONB to JSON explicitly.
None
- Test: Manual verification of zip layout; CI rerun pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Resolve JDBC driver_url via ConnectorContext environment
Problem Summary: JDBC connector plugin fails to load driver classes because
plain driver filenames (e.g., "mysql-connector-j-8.4.0.jar") are not resolved
to absolute file:// URLs using Config.jdbc_drivers_dir.
The fix adds a generic getEnvironment() method to ConnectorContext SPI,
which fe-core populates with system configs (jdbc_drivers_dir, doris_home).
The JDBC connector uses this environment to resolve driver URLs, keeping
fe-core free of JDBC-specific logic.
None
- Test: Manual test - verified driver URL resolution logic
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Implement toThrift() for PluginDrivenExternalTable via connector SPI
Problem Summary: PluginDrivenExternalTable.toThrift() inherited the base
ExternalTable implementation which returns null, causing NPE during query
fragment serialization when BE needs TTableDescriptor.
The fix adds:
1. ConnectorTableOps.getTableDescriptorProperties() SPI method that lets
connectors declare the BE table descriptor type and properties
2. JdbcConnectorMetadata returns JDBC_TABLE type with all connection/pool
properties needed by TJdbcTable
3. EsConnectorMetadata returns ES_TABLE type
4. PluginDrivenExternalTable.toThrift() reads these properties and
constructs the appropriate typed TTableDescriptor (TJdbcTable/TEsTable)
None
- Test: Manual test — JDBC catalog query no longer hits null TTableDescriptor
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Connector directly builds TTableDescriptor via fe-thrift
Issue Number: close #xxx
Problem Summary: Previously PluginDrivenExternalTable.toThrift() used a Map-based
getTableDescriptorProperties() intermediary to build TTableDescriptor, which required
fe-core to contain connector-specific logic for constructing TJdbcTable/TEsTable.
This refactoring lets each connector module directly depend on fe-thrift (provided scope)
and build its own TTableDescriptor in buildTableDescriptor(), making PluginDrivenExternalTable
a pure delegator with zero connector-specific code.
Key changes:
- ConnectorTableOps: replaced getTableDescriptorProperties() with buildTableDescriptor()
returning TTableDescriptor directly
- JdbcConnectorMetadata: builds TJdbcTable + TTableDescriptor directly
- EsConnectorMetadata: builds TEsTable + TTableDescriptor directly
- PluginDrivenExternalTable.toThrift(): simplified to pure delegation
- Added fe-thrift as provided dependency to fe-connector-api, fe-connector-jdbc,
fe-connector-es (Maven provided scope is NOT transitive)
- All 4 plugin assembly ZIPs exclude fe-thrift and libthrift
None
- Test: Manual test (compilation verified, runtime pending BIT/BOOLEAN display fix)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix SPI column attributes, type mapping, EXPLAIN format, and DB/schema mapping
Issue Number: close #xxx
Problem Summary: Fix 13 CI test failures in the External Regression pipeline
caused by the connector migration. This batch addresses 4 root cause categories:
1. **isKey lost (5 tests)**: ConnectorColumn had no isKey field; convertColumn()
hardcoded false. Added isKey to ConnectorColumn, propagated through
ConnectorColumnConverter, populated from JDBC getPrimaryKeys().
2. **Type mapping (3 tests)**: HLL/BITMAP/QUANTILE_STATE types were not handled
in JdbcMySQLConnectorClient.mapSignedType(), falling to UNSUPPORTED. Added
the missing cases for Doris-to-Doris JDBC catalog support.
3. **DB/schema mapping (3 tests)**: ClickHouse connector ignored databaseterm
URL parameter; OceanBase only delegated type mapping (not metadata methods);
PostgreSQL missing bit/varbit/hstore type mappings.
4. **EXPLAIN format (2 tests)**: PluginDrivenScanNode did not output QUERY: line
in EXPLAIN. Added getNodeExplainString() override that shows pushed-down SQL
from JdbcScanPlanProvider.getScanNodeProperties().
None
- Test: Regression test (pending CI run)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Wire write path, statistics, row count, and test expectations through connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Fix 5 regression test failures in the connector SPI migration (Batch 2):
1. InsertUtils write sink: Add UnboundConnectorTableSink handling in
getTargetTableQualified() for write transaction support.
2. Statistics collection: Override createAnalysisTask() in
PluginDrivenExternalTable to return ExternalAnalysisTask.
3. Row count estimation: Add fetchRowCount() to PluginDrivenExternalTable
delegating to ConnectorMetadata.getTableStatistics(). Add
JdbcConnectorMetadata.getTableStatistics() and MySQL-specific
getRowCount() querying INFORMATION_SCHEMA.
4. CallExecuteStmt test: Update expected exception message from
"Only support JDBC catalog" to "executeStmt not supported" since
HMS catalogs now route through PluginDrivenExternalCatalog.
5. Oracle CHAR padding: Update test_oracle_jdbc_catalog.out to match
current OracleTypeHandler behavior which trims CHAR trailing spaces.
This is a pre-existing behavior change from master commit 9770ecf8a59
(JdbcTypeHandler refactor) where the .out file was never updated.
None
- Test: Regression test expectations updated
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ES URL parsing, predicate pushdown, date compat, and TVF classloader
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Four bugs introduced during the connector SPI migration:
1. **ES double-port URL** (test_es_query_no_http_url): EsNodeInfo seed constructor
produced "host:port:80" for inputs like "host:9200" without a scheme prefix.
Fixed by correctly parsing host and port when split(":") yields 2 parts.
2. **ES LIKE not pushed down** (test_es_query): EsQueryDslBuilder.likeToDsl()
checked column2typeMap BEFORE resolving keyword sub-fields from fieldsContext.
Text fields with .keyword sub-fields were rejected. Fixed by checking
fieldsContext.containsKey() first, matching the old QueryBuilders behavior.
3. **ES date equality returns empty** (test_es_query_nereids): compatDefaultDate()
used ZoneOffset.UTC, but the old Joda-Time code used the system default timezone.
When JVM runs in CST (UTC+8), "2022-08-08 08:00:00" should map to midnight UTC,
not 8am UTC. Fixed by using ZoneId.systemDefault().
4. **TVF cross-catalog ClassCastException** (test_query_tvf_cross_catalog,
test_query_tvf_auth): PassthroughQueryTableHandle (in org.apache.doris.connector.api)
was loaded by both app classloader and ChildFirstClassLoader, causing instanceof
to fail. Fixed by adding "org.apache.doris.connector.api." to the parent-first
package list in ChildFirstClassLoader.
None
- Test: Regression test (CI pipeline)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](be) Fix CTAS ArrayStoreException for MySQL YEAR column
Issue Number: close #xxx
Problem Summary: MySQL YEAR columns mapped to Doris SMALLINT caused
ArrayStoreException: java.sql.Date in JdbcJniScanner.getNext().
Root cause: MySQLTypeHandler.getColumnValue() used untyped
rs.getObject(columnIndex) for TINYINT/SMALLINT, which returns
java.sql.Date for YEAR columns when MySQL Connector/J default
yearIsDateType=true is in effect. The SMALLINT converter then could
not handle java.sql.Date (falls through to return input as-is),
and storing java.sql.Date into Short[] caused ArrayStoreException.
Fix: Use typed rs.getObject(columnIndex, Integer.class) for TINYINT
and SMALLINT to force the JDBC driver to convert to Integer regardless
of the underlying MySQL column type.
None
- Test: Regression test (test_mysql_all_types_ctas)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](build) Change mvn package to install to fix MDEP-187 reactor dependency issue
Problem Summary: The fe-core module copy-dependencies goal fails with
MDEP-187 error when new reactor modules (fe-connector-api,
fe-connector-spi) have not been installed to the local Maven
repository. This is a known Maven limitation where copy-dependencies
cannot resolve reactor artifacts during the same mvn package run.
Fix: Change build.sh from mvn package to mvn install so all reactor
artifacts are installed to the local repo before copy-dependencies runs.
None
- Test: Manual test (build.sh --fe)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](build) Add Apache license headers to SPI service files
Problem Summary: ConnectorProvider service files in META-INF/services/
were missing Apache license headers, causing the license checker to fail.
None
- Test: No need to test (license headers only)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](es) Fix ES date predicate pushdown timezone conversion
Problem Summary: ES date equality queries returned empty results because
compatDefaultDate() could not parse ISO-formatted datetime strings produced
by extractLiteralValue(). The extractLiteralValue() method formats
LocalDateTime as "2022-08-08T08:00:00" (ISO with T separator), but
compatDefaultDate() only tried to parse "yyyy-MM-dd HH:mm:ss" (space
separator). The parse failure caused the value to be sent without timezone
conversion, so ES interpreted it as UTC instead of the local timezone.
Fix: Add ISO_LOCAL_DATE_TIME parsing as the first attempt in
compatDefaultDate(), before falling back to space-separated format and
date-only format.
None
- Test: Regression test (test_es_query_nereids sql63)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix JDBC connector type mapping, auth, and EXPLAIN regressions
Issue Number: close #62183
Problem Summary: Fix multiple regression failures in the connector-based
JDBC/ES external catalog migration. This commit addresses 8 root causes
covering 20+ test failures from CI External Regression pipeline build 926236.
Fixes include:
- detectDoris() initialization ordering: moved from constructor to
postInitialize() hook so data source is live before detection runs.
Added dorisTypeToConnectorType() for Doris-to-Doris type mapping
including HLL, BITMAP, QUANTILE_STATE types.
- Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match
"TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc.
- Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0
for integer branch, and match old boundary thresholds.
- ClickHouse DB listing: use SHOW DATABASES for full listing;
fix databaseTermIsCatalog inversion for old drivers.
- DATETIMEV2 precision: ConnectorColumnConverter reads precision
(not scale) for datetime fractional seconds, matching connector
encoding convention.
- EXPLAIN cache: invalidate scanNodeProperties after convertPredicate()
so pushed conjuncts are reflected in EXPLAIN output.
- ExecutionAuthenticator: call initPreExecutionAuthenticator() in
PluginDrivenExternalCatalog.initLocalObjectsImpl().
- PassthroughQueryTableHandle: add instanceof guards in
JdbcConnectorMetadata to prevent ClassCastException for TVF.
- isKey: hardcode true for all columns matching legacy behavior.
None
- Test: Regression test (CI External Regression pipeline)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ES LIKE pushdown for Nereids planner
Problem Summary: Nereids planner translates Like/Regexp expressions to
FunctionCallExpr (via ExpressionTranslator.visitScalarFunction) rather
than LikePredicate. ExprToConnectorExpressionConverter only checked for
instanceof LikePredicate, so the LIKE predicate was converted to a
ConnectorFunctionCall instead of ConnectorLike. This caused ES query DSL
builder to not recognize it as a LIKE and fall back to match_all,
returning all documents instead of only those matching the wildcard.
Fix: Detect FunctionCallExpr with function name "like" or "regexp" in
ExprToConnectorExpressionConverter and convert to ConnectorLike.
None
- Test: Regression test (test_es_query sql_5_29/sql_6_29)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Add JDBC URL normalization for connector migration
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The connector migration moved JDBC catalog logic from
fe-core to fe-connector-jdbc, but the URL normalization step was lost.
The old path called JdbcResource.handleJdbcUrl() which added critical
JDBC driver parameters. Without these, OceanBase tests fail with
timezone shifts (8h offset), lost fractional seconds, BOOLEAN display
as true/false instead of 1/0, and DOUBLE precision loss.
The fix adds JdbcUrlNormalizer in fe-connector-jdbc that replicates the
same parameter injection logic:
- MySQL/OceanBase: yearIsDateType=false, tinyInt1isBit=false,
useUnicode=true, characterEncoding=utf-8, rewriteBatchedStatements=true
- OceanBase additionally: useCursorFetch=true
- PostgreSQL: reWriteBatchedInserts=true
- SQL Server: useBulkCopyForBatchInsert=true
The normalization is applied once in JdbcDorisConnector constructor,
so all downstream consumers (client, metadata, scan plan provider)
use the normalized URL.
None
- Test: Regression test (test_oceanbase_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Stop filtering ClickHouse system database from catalog listing
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The new JdbcClickHouseConnectorClient incorrectly filtered
the ClickHouse "system" database in getFilterInternalDatabases(). The old
JdbcClickHouseClient never overrode this method, so it used the base class
default which only filters information_schema, performance_schema, and mysql.
The system database contains useful tables (query_log, processes, etc.) and
should remain accessible to users.
None
- Test: Regression test (test_clickhouse_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ClickHouse JDBC catalog regression issues in plugin-driven architecture
Issue Number: close #xxx
Problem Summary: Multiple regression issues found after migr…
morningman
added a commit
to morningman/doris
that referenced
this pull request
Apr 13, 2026
…faces
Issue Number: close #xxx
Problem Summary: Add the fe-connector module hierarchy (fe-connector-api and
fe-connector-spi) as the foundation for Catalog SPI modularization. This
follows the same pattern as fe-filesystem, enabling external data source
connectors to be developed and deployed as independent plugins.
fe-connector-api defines 30 zero-dependency interfaces including:
- Opaque handle types (ConnectorTableHandle, ConnectorColumnHandle, etc.)
- ConnectorMetadata composed from sub-interfaces (SchemaOps, TableOps,
PushdownOps, StatisticsOps, WriteOps)
- apply* pushdown negotiation (FilterApplicationResult, ProjectionApplicationResult)
- ConnectorScanPlanProvider for split generation
- ConnectorCapability enum for capability declaration
- Lightweight type system (ConnectorColumn, ConnectorType) independent of fe-core
fe-connector-spi defines the provider SPI:
- ConnectorProvider extends PluginFactory for ServiceLoader discovery
- ConnectorContext for engine-to-connector runtime services
None
- Test: No need to test - module skeleton with interfaces only, no behavioral changes
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorPluginManager and session bridge for connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: This step adds the connector plugin loading infrastructure
to fe-core, mirroring the FileSystemPluginManager pattern. It enables
connector plugins to be discovered via ServiceLoader (builtins) and loaded
from the plugin directory at runtime via DirectoryPluginRuntimeManager with
ChildFirst ClassLoader isolation.
Key additions:
- ConnectorPluginManager: manages ConnectorProvider lifecycle (builtin +
directory-based plugin loading)
- ConnectorFactory: static factory singleton initialized by Env at startup
- ConnectorSessionBuilder: bridges ConnectContext → ConnectorSession for
ClassLoader-safe session passing across SPI boundary
- ConnectorSessionImpl: immutable ConnectorSession implementation
- DefaultConnectorContext: minimal ConnectorContext providing catalogName,
catalogId, and FS access
- Config.connector_plugin_root: configurable plugin directory
- Env.initConnectorPluginManager(): startup initialization hook
None
- Test: No need to test - infrastructure only, no connector plugins exist yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove Iceberg SDK dependency from ExternalMetadataOps interface
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The ExternalMetadataOps interface (a core fe-core interface
for external catalog metadata operations) had a direct import of
org.apache.iceberg.view.View, coupling the interface to Iceberg SDK. The
ExternalMetadataOperations factory class also imported all connector-specific
types (Iceberg, Paimon, MaxCompute, Hive), creating unnecessary coupling.
Changes:
- ExternalMetadataOps.loadView() return type changed from View to Object.
The method is only overridden by IcebergMetadataOps, and all callers are
in iceberg-specific code that already imports View for casting.
- Deleted ExternalMetadataOperations factory class entirely. Each catalog
now directly constructs its own MetadataOps (the factory was a thin
indirection adding no value since each caller already knows its type).
- Removed unused View import from IcebergMetadataOps (return type is now
Object; iceberg view.View is only used in IcebergExternalMetaCache).
None
- Test: No need to test - pure refactoring, no behavioral change
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Abstract Hadoop Configuration to Map-based catalog properties
Issue Number: close #xxx
Problem Summary: ExternalCatalog.getConfiguration() returns org.apache.hadoop.conf.Configuration,
which leaks Hadoop types into the core catalog interface. This blocks future SPI extraction of
connector modules since fe-connector-api must be free of Hadoop dependencies.
This commit introduces getHadoopProperties() returning Map<String, String> and a static
buildHadoopConfiguration() utility, then migrates all callers (HudiExternalMetaCache,
IcebergUtils, HiveMetaStoreClientHelper) to the new pattern. The original getConfiguration()
is deprecated.
None
- Test: No need to test - pure refactor, all callers produce identical Configuration objects
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add PluginDrivenExternalCatalog adapter and SPI-based CatalogFactory
Issue Number: close #xxx
Problem Summary: The CatalogFactory uses hardcoded switch-case to create catalogs. This commit
introduces a SPI-first creation path: ConnectorFactory.createConnector() is tried before falling
back to the built-in switch-case. A new PluginDrivenExternalCatalog bridges SPI Connector instances
with the existing ExternalCatalog hierarchy, enabling third-party catalog plugins.
None
- Test: No need to test — no SPI connector plugins exist yet, all catalogs still use the fallback switch-case path. Behavior is unchanged.
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add fe-connector-es module with SPI metadata implementation
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Creates the first real connector plugin module (Elasticsearch)
implementing the connector SPI interfaces. This validates the entire SPI design
with a working metadata-only implementation. The ES scan path remains in fe-core
temporarily — only metadata operations (list databases, list tables, get schema)
are handled via the SPI.
The module includes:
- EsConnectorProvider: SPI entry point discovered via ServiceLoader
- EsConnector: Main connector, creates metadata instances
- EsConnectorMetadata: Lists ES indices as tables, parses mappings to schema
- EsConnectorRestClient: HTTP client adapted from fe-core (no fe-core deps)
- EsTypeMapping: ES type → ConnectorType mapping
- EsTableHandle: Opaque handle for ES indices
- EsConnectorProperties: Property constants and compatibility processing
- Plugin assembly ZIP descriptor for runtime deployment
None
- Test: No need to test — metadata-only SPI implementation, fe-core ES path unchanged
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add shard routing and metadata pipeline to fe-connector-es
Issue Number: close #xxx
Problem Summary: The initial fe-connector-es module (Phase 1) only supported basic
metadata operations (list databases, list tables, get schema). This commit adds
Phase 2 capabilities: shard routing discovery, node info resolution, mapping field
analysis (keyword sniff, doc_value, date compat), and full metadata orchestration.
These are prerequisites for the future ConnectorScanPlanProvider which will generate
TScanRangeLocations from the plugin side.
New files (8):
- EsMajorVersion: ES version detection (adapted from fe-core, zero fe-core deps)
- EsNodeInfo: ES node with HTTP address (host+port instead of TNetworkAddress)
- EsShardRouting: Shard routing entry (host+port instead of TNetworkAddress)
- EsShardPartitions: Shard partition map with _search_shards JSON parsing
- EsFieldContext: Field context holder (fetchFields, docValue, dateCompat maps)
- EsMappingUtils: Mapping parsing + field resolution (from EsUtil + MappingPhase)
- EsMetadataState: Full metadata state (adapted from SearchContext)
- EsMetadataFetcher: Metadata fetch orchestrator (adapted from EsMetaStateTracker)
Enhanced files (3):
- EsConnectorRestClient: added searchShards(), getHttpNodes(), get() methods
- EsConnectorMetadata: added fetchMetadataState() for full metadata pipeline
- EsConnectorProperties: added MAPPING_TYPE constant
None
- Test: No need to test — plugin module only, not activated in production
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-trino plugin module (Step 7 Phase 1)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Trino Connector metadata logic from fe-core into an
independent plugin module fe-connector-trino, continuing the Catalog SPI
modularization. This phase migrates the metadata-only layer (schema discovery,
type mapping, Trino plugin bootstrap) while leaving scan/predicate code in
fe-core for Step 16.
Key files:
- TrinoConnectorProvider: SPI entry point (type="trino-connector")
- TrinoDorisConnector: Lazy init, wraps Trino Connector + Session
- TrinoBootstrap: Singleton Trino plugin infrastructure init
- TrinoConnectorDorisMetadata: ConnectorMetadata impl (list/get/schema)
- TrinoTypeMapping: Trino SPI types -> ConnectorType (15+ mappings)
- TrinoPluginManager/TrinoServicesProvider: Adapted from fe-common
Design decisions:
- Copied fe-common classes (~290 lines each) into plugin rather than depending
on fe-common (which has heavy transitive deps like hadoop-common)
- TrinoBootstrap uses singleton pattern for plugin loading, per-catalog
connector creation
- Plugin dir resolution: property -> DORIS_HOME/plugins/connectors -> fallback
None
- Test: No need to test (metadata-only extraction, fe-core code unchanged)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-maxcompute plugin module (Step 8)
Problem Summary: Extract MaxCompute connector metadata code into an independent
plugin module as part of the Catalog SPI modularization effort.
Changes:
- Create fe-connector-maxcompute module with 11 Java files + build config
- MaxComputeConnectorProvider (SPI entry, type="max_compute")
- MaxComputeDorisConnector (Odps client lifecycle, lazy init)
- MaxComputeConnectorMetadata (listDatabases, listTables, getTableHandle, getTableSchema)
- MCTypeMapping (15+ ODPS type -> ConnectorType mappings including nested ARRAY/MAP/STRUCT)
- McStructureHelper (namespace schema abstraction, copied from fe-core)
- MCConnectorClientFactory, MCConnectorEndpoint, MCConnectorProperties (utilities)
- MaxComputeTableHandle (opaque handle wrapping ODPS Table + TableIdentifier)
- Enhance ConnectorType API with factory methods (of, arrayOf, mapOf, structOf)
- Make Connector.getScanPlanProvider() a default method (scan planning is optional for Phase 1)
Phase 1 scope: metadata-only. ScanNode, Transaction, DDL operations stay in fe-core.
None
- Test: No need to test - Phase 1 metadata extraction, SPI plugin not yet activated
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-jdbc plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extracts JDBC connector metadata logic into an independent
fe-connector-jdbc plugin module as part of the Catalog SPI modularization.
This is Step 10 of the connector SPI plan.
The module implements:
- JdbcConnectorProvider (SPI entry point, type="jdbc")
- JdbcDorisConnector (manages HikariCP connection pool lifecycle)
- JdbcConnectorMetadata (delegates to JdbcConnectorClient for metadata discovery)
- JdbcConnectorClient base class with factory method dispatching to 10 DB-specific subclasses
- All 10 DB-specific clients: MySQL, PostgreSQL, Oracle, ClickHouse, SQLServer,
SAP HANA, Trino, OceanBase, DB2, GBase
- JdbcFieldInfo (column metadata, adapted from JdbcFieldSchema)
- JdbcDbType enum, JdbcTableHandle, JdbcConnectorProperties
Each subclass maps JDBC types to ConnectorType (the SPI type system) instead
of fe-core Type, achieving zero fe-core dependency.
None
- Test: No need to test (Phase 1 metadata-only extraction, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Extract HMS shared module to fe-connector-hms
Issue Number: close #xxx
Problem Summary: The HMS (Hive MetaStore) client code in fe-core is deeply
coupled to internal types (Column, Config, ExecutionAuthenticator, etc.), making
it impossible to directly extract. This commit creates a new clean HMS client
library module (fe-connector-hms) using SPI types, which will be shared by
future fe-connector-hive and fe-connector-hudi plugins.
Key design decisions:
- New clean interfaces using ConnectorType/ConnectorColumn (not code extraction)
- Connection-pooled Thrift client with taint-and-destroy error handling
- Supports HMS/DLF/Glue metastore types via string-based class dispatch
- Auth via functional interface (AuthAction) replacing fe-core ExecutionAuthenticator
- HmsTypeMapping mirrors HiveMetaStoreClientHelper logic but returns ConnectorType
Module contents (9 Java files, ~1440 lines):
- HmsClient: Clean interface for HMS operations
- ThriftHmsClient: Pooled Thrift implementation with ClassLoader safety
- HmsClientConfig/HmsConfHelper: Configuration DTOs and HiveConf creation
- HmsTableInfo/HmsDatabaseInfo/HmsPartitionInfo: Immutable DTOs
- HmsTypeMapping: Hive type string to ConnectorType mapping
- HmsClientException: Runtime exception class
None
- Test: No need to test - library module with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-hive plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Hive connector metadata operations into an independent
fe-connector-hive plugin module as Phase 1 of the Catalog SPI modularization.
This creates the fe-connector-hive plugin that:
- Implements ConnectorProvider (type="hms") for SPI-based catalog creation
- Provides HiveConnectorMetadata with listDatabases, listTables, getTableHandle,
getTableSchema operations via the shared fe-connector-hms client
- Detects table formats (HIVE/HUDI/ICEBERG) from HMS table metadata
- Supports configurable type mapping options (binary-as-string, timestamp-tz)
- Manages HmsClient lifecycle with lazy initialization and proper shutdown
Files: 10 new files (816 lines), 2 modified files
Dependencies: fe-connector-spi + fe-connector-hms + log4j-api
None
- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-paimon plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Paimon connector metadata operations into an independent
fe-connector-paimon plugin module as Phase 1 of the Catalog SPI modularization.
This creates the fe-connector-paimon plugin that:
- Implements ConnectorProvider (type="paimon") for SPI-based catalog creation
- Creates Paimon Catalog instances directly via Paimon SDK (Options + CatalogContext
+ CatalogFactory) supporting all backends: filesystem, HMS, DLF, REST, JDBC
- Provides PaimonConnectorMetadata with listDatabases, listTables, getTableHandle,
getTableSchema operations using the Paimon Catalog API
- Includes PaimonTypeMapping that converts Paimon DataType to ConnectorType,
mirroring existing PaimonUtil.paimonTypeToDorisType logic
- Supports configurable type mapping options (binary-as-varbinary, timestamp-tz)
- Manages Paimon Catalog lifecycle with lazy init and proper shutdown
Files: 8 new files (805 lines), 1 modified file
Dependencies: fe-connector-spi + paimon-core + log4j-api
None
- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add fe-connector-hudi Phase 1 plugin module
Issue Number: close #xxx
Problem Summary: Extract Hudi connector metadata operations into an
independent SPI plugin module (fe-connector-hudi) as part of the
catalog connector SPI modularization effort.
The Hudi connector plugin provides Phase 1 read-only metadata:
- List databases and tables via HMS (shared fe-connector-hms client)
- Get table schema from HoodieTableMetaClient Avro schema
(authoritative for schema-evolved Hudi tables)
- Detect Hudi table type (COW vs MOR) from input format
- Avro-to-ConnectorType mapping (HudiTypeMapping)
Dependencies: fe-connector-spi + fe-connector-hms + hudi-common +
hudi-hadoop-mr.
Scan planning, incremental query, statistics, and event processing
remain in fe-core temporarily.
None
- Test: No need to test (Phase 1 SPI module, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-iceberg Phase 1 — metadata-only SPI plugin
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Step 15 of the Catalog Connector SPI modularization. Extracts the Iceberg
connector into an independent plugin module (fe-connector-iceberg) that
provides read-only metadata operations through the Connector SPI.
Phase 1 implements:
- IcebergConnectorProvider: SPI entry point, type="iceberg"
- IcebergConnector: Lifecycle management, creates Iceberg SDK Catalog via
CatalogUtil.buildIcebergCatalog() — supports all 7 backends (REST, HMS,
Glue, DLF, JDBC, Hadoop, S3Tables)
- IcebergConnectorMetadata: listDatabaseNames (SupportsNamespaces), listTableNames,
getTableHandle, getTableSchema via Iceberg SDK Catalog API
- IcebergTypeMapping: Iceberg types → ConnectorType (BOOLEAN, INT, LONG, FLOAT,
DOUBLE, STRING, UUID/BINARY→VARBINARY or STRING, DECIMAL, DATE, TIMESTAMP,
LIST, MAP, STRUCT) with enableMappingVarbinary and enableMappingTimestampTz flags
- IcebergTableHandle: Opaque handle carrying db+table coordinates
- IcebergConnectorProperties: Property key constants
Key design decisions:
- No fe-connector-hms dependency: Iceberg SDK HiveCatalog handles HMS internally
- Property-driven catalog creation: CatalogUtil.buildIcebergCatalog() handles
all backends, subclass dispatch is just setting catalog-impl
- Hadoop Configuration built from user properties (hadoop.*, fs.*, dfs.*, hive.*)
6 Java files, ~770 lines added.
None
- Test: No need to test — Phase 1 metadata-only SPI module, not yet wired into fe-core runtime
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add PluginDrivenExternalTable schema support and GSON registration
Issue Number: close #xxx
Problem Summary:
PluginDrivenExternalTable was a minimal stub with no schema retrieval capability.
Plugin-driven catalogs could list databases and tables but could not fetch table
schemas, making them non-functional for queries. Additionally, the PluginDriven*
types were not registered in GsonUtils, preventing GSON serialization/deserialization
of plugin-driven catalog metadata for FE persistence.
This commit implements Step 18 of the Catalog Connector SPI plan:
1. **ConnectorColumnConverter** (new): Converts between the connector SPI type system
(ConnectorColumn/ConnectorType) and Doris internal types (Column/Type). Handles
all scalar types plus complex types (ARRAY, MAP, STRUCT) recursively, with proper
precision/scale handling for CHAR, VARCHAR, DECIMAL, and DATETIMEV2.
2. **PluginDrivenExternalTable**: Overrides initSchema() to fetch table schema from
the connector SPI. Uses ConnectorMetadata.getTableHandle() + getTableSchema(),
then converts via ConnectorColumnConverter.
3. **PluginDrivenExternalCatalog**: Makes buildConnectorSession() package-private so
PluginDrivenExternalTable can build sessions during schema init.
4. **GsonUtils**: Registers PluginDrivenExternalCatalog, PluginDrivenExternalDatabase,
and PluginDrivenExternalTable in the respective RuntimeTypeAdapterFactory instances
for proper GSON serialization/deserialization.
None
- Test: No need to test - infrastructure code with no active connector plugins yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorExpression framework and filter pushdown types
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Step 19 of the Catalog SPI modularization. Replaces the
placeholder ConnectorExpression class with a proper expression tree hierarchy
for cross-SPI filter/projection pushdown negotiation. Adds the
ExprToConnectorExpressionConverter in fe-core to convert Doris Expr trees
into ConnectorExpression trees at the SPI boundary.
Changes:
- ConnectorExpression: converted from final class to interface with
getExprType() and getChildren() methods
- ConnectorExprType: discriminator enum (COLUMN_REF, LITERAL, COMPARISON,
AND, OR, NOT, IN, BETWEEN, IS_NULL, LIKE, FUNCTION_CALL)
- 11 concrete expression types: ConnectorColumnRef, ConnectorLiteral,
ConnectorComparison, ConnectorAnd, ConnectorOr, ConnectorNot,
ConnectorIn, ConnectorBetween, ConnectorIsNull, ConnectorLike,
ConnectorFunctionCall
- ConnectorRange: range bound with low/high inclusive/exclusive endpoints
- ConnectorDomain: per-column domain (union of ranges + null handling)
for fast partition pruning
- ConnectorFilterConstraint: redesigned to carry full expression tree +
per-column domain map (replaces old flat conjuncts list)
- ExprToConnectorExpressionConverter: converts Doris Expr tree to
ConnectorExpression tree, plus Type to ConnectorType reverse mapping
None
- Test: No need to test - pure API/SPI type definitions and converter
with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorScanPlanProvider and PluginDrivenScanNode for connector SPI scan planning
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Plugin-driven catalogs need a generic scan node that delegates
scan planning to the connector SPI, rather than requiring connector-specific
ScanNode subclasses in fe-core.
This commit adds Step 20 of the Catalog SPI modularization:
**fe-connector-api enhancements:**
- ConnectorScanRangeType enum: FILE_SCAN, JDBC_SCAN, ES_SCAN, REMOTE_OLAP_SCAN, CUSTOM
- ConnectorDeleteFile: delete file descriptor for Iceberg MOR tables
- ConnectorScanRange: added getRangeType() (mandatory), getFileFormat(),
getFileSize(), getModificationTime(), getPartitionValues(), getDeleteFiles()
- ConnectorScanPlanProvider: added estimateScanRangeCount() default method
**fe-core additions:**
- PluginDrivenSplit: wraps ConnectorScanRange in FileSplit for the
FileQueryScanNode pipeline
- PluginDrivenScanNode: extends FileQueryScanNode, delegates scan planning
to ConnectorScanPlanProvider, uses FORMAT_JNI for BE execution
- ExprToConnectorExpressionConverter: added convertConjuncts() utility
- PhysicalPlanTranslator: dispatch to PluginDrivenScanNode for
PluginDrivenExternalTable instances
None
- Test: No need to test — SPI infrastructure, no actual connectors activate this path yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorWriteOps & PluginDrivenTableSink (Step 21)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Implements Step 21 of the Catalog SPI modularization plan.
Adds the write operation SPI interfaces and the generic table sink for
plugin-driven external tables.
**SPI layer (fe-connector-api):**
- Enhanced ConnectorWriteOps with full write lifecycle: getWriteConfig,
beginInsert/finishInsert/abortInsert (with column list), beginDelete/
finishDelete/abortDelete, beginMerge/finishMerge/abortMerge
- Added ConnectorWriteType enum (FILE_WRITE, JDBC_WRITE, REMOTE_OLAP_WRITE,
CUSTOM)
- Added ConnectorWriteConfig value object with builder pattern, carrying
write type, file format, compression, location, partition columns, and
generic properties
- Added ConnectorDeleteHandle and ConnectorMergeHandle marker interfaces
**Engine layer (fe-core):**
- Created PluginDrivenTableSink extending BaseExternalTableDataSink
- Constructs TDataSink based on ConnectorWriteConfig.getWriteType():
FILE_WRITE -> THiveTableSink, JDBC_WRITE -> TJdbcTableSink
- Property-driven Thrift construction using well-known keys from
ConnectorWriteConfig.properties
None
- Test: No need to test - infrastructure interfaces with no runtime behavior change
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add generic Nereids plan nodes for plugin-driven INSERT pipeline
Issue Number: close #xxx
Problem Summary: PluginDrivenExternalCatalog-backed tables cannot participate
in the full Nereids INSERT pipeline because there are no generic plan node
classes for them. Each existing connector (Hive, Iceberg, JDBC, etc.) has its
own typed plan nodes, but the plugin-driven generic path has none.
This commit adds the complete Nereids INSERT pipeline for plugin-driven
catalogs:
1. **Plan nodes**: UnboundConnectorTableSink, LogicalConnectorTableSink,
PhysicalConnectorTableSink — generic nodes that work with any
PluginDrivenExternalCatalog without connector-specific plan classes.
2. **Optimizer rules**: BINDING_INSERT_CONNECTOR_TABLE (BindSink),
LogicalConnectorTableSinkToPhysicalConnectorTableSink (implementation),
ExpressionRewrite for LogicalConnectorTableSink.
3. **Insert executor**: PluginDrivenInsertExecutor delegates begin/commit/
abort to ConnectorWriteOps SPI. PluginDrivenInsertCommandContext provides
the context wrapper.
4. **Transaction manager**: PluginDrivenTransactionManager provides
lightweight transaction lifecycle bookkeeping; actual commit/rollback
is handled by ConnectorWriteOps in the insert executor.
5. **Wiring**: PlanType enum entries, RuleType entries, SinkVisitor methods,
RuleSet registration, UnboundTableSinkCreator dispatch,
PhysicalPlanTranslator visitor, InsertIntoTableCommand executor selection.
None
- Test: No need to test — infrastructure code, no connector plugins exercise
this path yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Activate CatalogFactory SPI path with build integration and connectivity testing
Issue Number: close #xxx
Problem Summary: The connector SPI plugins were built but not deployed to
the output directory, making them undiscoverable at runtime. Additionally,
PluginDrivenExternalCatalog lacked connectivity testing support.
This commit:
1. Adds connector module compilation to build.sh FE module list
2. Deploys connector plugin ZIPs to output/fe/plugins/connector/<name>/
(mirroring the filesystem plugin deployment pattern)
3. Creates ConnectorTestResult value object in fe-connector-api
4. Adds Connector.testConnection(session) default method
5. Overrides checkWhenCreating() in PluginDrivenExternalCatalog to delegate
connectivity testing to the connector SPI
6. Adds ConnectorFactory.getRegisteredTypes() for diagnostics
7. Enhances logging in CatalogFactory and Env for SPI path transparency
None
- Test: No need to test (build infrastructure + SPI wiring, no behavioral change)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 24: ES full migration to connector SPI
Issue Number: close #xxx
Problem Summary: Migrate the ES (Elasticsearch) connector scan path from
legacy fe-core EsScanNode to the plugin-driven connector SPI. This enables
the ES connector to run as an isolated plugin while producing the same
ES_HTTP_SCAN_NODE Thrift types expected by BE.
Key changes:
- Enhanced ConnectorScanPlanProvider with getScanRangeType(),
getScanNodeProperties(), and getScanNodeMapProperties() to support
non-file scan types (ES, JDBC) that need custom Thrift structures
- Created EsQueryDslBuilder: ports QueryBuilders from legacy Expr-based
filter conversion to ConnectorExpression-based ES Query DSL generation
- Created EsScanPlanProvider: generates per-shard scan ranges with ES
host routing, query DSL, auth info, and field context maps
- Created EsScanRange: ConnectorScanRange impl for ES shard routing
- Created PluginDrivenEsScanNode: extends ExternalScanNode to produce
TPlanNodeType.ES_HTTP_SCAN_NODE with TEsScanNode/TEsScanRange Thrift
- Wired EsConnector.getScanPlanProvider() to return real provider
- Added EsConnector.testConnection() for catalog connectivity testing
- Fixed BindRelation missing PLUGIN_EXTERNAL_TABLE case (critical:
without this fix, no plugin-driven table can be queried)
- PhysicalPlanTranslator dispatches to PluginDrivenEsScanNode when
getScanRangeType() == ES_SCAN
- Removed "es" case from CatalogFactory switch (SPI handles it)
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 25: JDBC full migration (scan only) to connector SPI
Issue Number: close #xxx
Problem Summary: Migrate JDBC connector scan path to the connector SPI
framework, enabling JDBC catalogs to run through the PluginDrivenScanNode
pipeline instead of the legacy JdbcScanNode code path.
Key changes:
- JdbcQueryBuilder: converts ConnectorExpression to SQL WHERE clauses with
DB-specific date formatting (Oracle to_date, Trino date literal, SQL Server
CONVERT) and LIMIT syntax (MySQL LIMIT, Oracle ROWNUM, SQL Server TOP N)
- JdbcScanRange: ConnectorScanRange with table_format_type="jdbc" and Builder
pattern for all BE-expected jdbc_params (url, user, password, driver, query_sql,
connection pool settings)
- JdbcScanPlanProvider: builds SQL query via JdbcQueryBuilder, returns 1 scan
range with all JDBC connection parameters
- JdbcColumnHandle: carries localName + remoteName for SELECT clause column
quoting via JdbcIdentifierQuoter
- JdbcConnectorMetadata.getColumnHandles(): returns column handle map for
PluginDrivenScanNode column selection
- JdbcDorisConnector: wires getScanPlanProvider() and testConnection()
- ConnectorScanRange.getTableFormatType(): default "plugin_driven", JDBC
overrides to "jdbc" so BE routes to JdbcJniReader
- ConnectorScanPlanProvider: 5-arg planScan() overload with limit parameter
for JDBC LIMIT pushdown
- PluginDrivenScanNode: uses scanRange.getTableFormatType() instead of
hardcoded constant; builds column handles via metadata.getColumnHandles();
passes limit to 5-arg planScan()
- CatalogFactory: removed "jdbc" case from fallback switch (all JDBC catalogs
now go through SPI path)
None
- Test: Manual test (compilation + checkstyle verified)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 26: MaxCompute full migration (scan only) to connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Migrates MaxCompute scan planning from fe-core to the
fe-connector-maxcompute plugin module. This removes the "max_compute" fallback
case from CatalogFactory, routing MaxCompute catalog creation exclusively
through the SPI connector path (PluginDrivenExternalCatalog).
Key changes:
- MaxComputeScanPlanProvider: creates ODPS TableBatchReadSession, generates
splits (byte_size and row_offset strategies), supports limit optimization
- MaxComputePredicateConverter: converts ConnectorExpression to ODPS Predicate
(CompoundPredicate, RawPredicate, UnaryPredicate) with datetime timezone
conversion
- MaxComputeScanRange: ConnectorScanRange carrying serialized session, split
params, and timeout config with tableFormatType="max_compute"
- MaxComputeColumnHandle: tracks partition vs data column distinction
- MCConnectorEndpoint: region-to-timezone mapping for datetime pushdown
- PluginDrivenScanNode: format-type dispatch in setScanParams() -
"max_compute" builds TMaxComputeFileDesc, others use generic jdbc_params map
- CatalogFactory: removed "max_compute" fallback case and
MaxComputeExternalCatalog import
- Added odps-sdk-table-api dependency for table read session API
None
- Test: No need to test - structural migration only, MaxCompute not available
in dev environment
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Migrate Trino connector scan planning to connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Trino connector scan planning was still handled by the legacy
TrinoConnectorScanNode in fe-core with direct Trino SPI type dependencies.
This commit migrates the full scan flow to the connector SPI plugin module
(fe-connector-trino), keeping all Trino SPI types inside the plugin boundary.
Key changes:
- TrinoScanPlanProvider: core scan planning (beginQuery → applyFilter/Limit/
Projection → getSplits → JSON-serialize → TrinoScanRange list → cleanupQuery)
- TrinoJsonSerializer: ObjectMapperProvider + HandleJsonModule + BlockJsonSerde
for serializing Trino SPI objects to JSON
- TrinoPredicateConverter: ConnectorExpression → Trino TupleDomain<ColumnHandle>
- TrinoScanRange: ConnectorScanRange carrying pre-serialized JSON properties
- TrinoColumnHandle: ConnectorColumnHandle for projection pushdown
- PluginDrivenScanNode: "trino_connector" dispatch filling TTrinoConnectorFileDesc
from properties map (no Trino types needed in fe-core)
- CatalogFactory: removed "trino-connector" legacy fallback case
- TrinoBootstrap: added getHandleResolver()/getTypeRegistry() accessors
- TrinoConnectorDorisMetadata: added getColumnHandles() method
- TrinoDorisConnector: replaced stub with real getScanPlanProvider() + testConnection()
None
- Test: No need to test - structural migration, scan logic ported from existing
TrinoConnectorScanNode with no behavioral changes
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Hive scan planning to connector plugin (Step 31)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Adds scan planning capability to the Hive connector plugin
module (fe-connector-hive), enabling file-based scan range generation
through the ConnectorScanPlanProvider SPI. This is the scan-only phase of
the Hive full migration to the connector plugin architecture.
New files in fe-connector-hive:
- HiveScanPlanProvider: Core scan planning - partition resolution via HMS,
file listing via Hadoop FS, file splitting by configurable target size
- HiveColumnHandle: ConnectorColumnHandle wrapping column name/type/isPartKey
- HiveFileFormat: Enum mapping InputFormat/SerDe classes to format strings
- HiveScanRange: ConnectorScanRange with Builder for file split descriptors
- HiveTextProperties: Extracts text format props from HMS SerDe parameters
Enhanced existing files:
- HiveTableHandle: Rewritten with Builder pattern, now carries inputFormat,
serializationLib, location, partitionKeyNames, sdParameters, tableParameters,
and prunedPartitions for scan planning
- HiveConnectorMetadata: Added getColumnHandles(), applyFilter() with
partition pruning (equality + IN predicates on partition columns)
- HiveConnector: Added getScanPlanProvider() returning HiveScanPlanProvider
- HmsTableInfo: Added sdParameters field for SerDe parameters
- ThriftHmsClient: Populates sdParameters from SerDeInfo
- PluginDrivenScanNode: Major enhancement with property-driven overrides for
getFileFormatType, getPathPartitionKeys, getLocationProperties, getFileAttributes,
plus hive/transactional_hive dispatch in setScanParams
Scope: Non-ACID Hive tables (Parquet, ORC, Text). ACID tables, file listing
cache, and table sampling are deferred to future steps.
None
- Test: No need to test (scan planning code path not yet wired to production
query flow; requires HMS + Hadoop cluster for integration testing)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Hudi scan planning to connector plugin (Step 32)
Issue Number: close #xxx
Problem Summary: Adds scan planning capability to the fe-connector-hudi plugin
module, following the same pattern established in Step 31 (Hive scan). This
enables the Hudi connector plugin to generate scan ranges that BE can consume,
supporting both COW (native Parquet/ORC reader) and MOR (JNI reader with delta
log merging) table types.
New files:
- HudiScanPlanProvider: Core scan planning — builds MetaClient, resolves
partitions via HoodieTableMetadata API, generates COW splits (base files
only) and MOR splits (base + delta logs with dynamic native downgrade)
- HudiScanRange: ConnectorScanRange implementation with Builder pattern,
carrying both native reader fields and JNI metadata (instant_time, serde,
delta_logs, column_names/types)
- HudiColumnHandle: Column handle with name, typeName, isPartitionKey
Modified files:
- HudiTableHandle: Rewritten with Builder pattern, added scan-related fields
(inputFormat, serdeLib, partitionKeyNames, tableParameters, prunedPaths)
- HudiConnectorMetadata: Enhanced getTableHandle with scan fields from HMS,
added getColumnHandles() and applyFilter() for partition pruning
- HudiConnector: Added getScanPlanProvider()
- PluginDrivenScanNode: Added "hudi" case in setScanParams dispatch with
setHudiParams() method creating THudiFileDesc for both native/JNI paths
None
- Test: No need to test — plugin scan planning mirrors existing HudiScanNode
logic; will be validated when end-to-end regression tests are enabled
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Paimon scan planning to connector plugin (Step 33)
Problem Summary: Migrate Paimon scan planning logic from fe-core PaimonScanNode
to the fe-connector-paimon plugin module, following the same pattern established
by Hive (Step 31) and Hudi (Step 32) scan migrations.
This step adds:
- PaimonColumnHandle: Column handle with name and field index in RowType
- PaimonPredicateConverter: Converts ConnectorExpression to Paimon Predicate
(PredicateBuilder-based, supports AND/OR/comparison/IN/IS NULL/LIKE prefix)
- PaimonScanRange: Builder-pattern scan range supporting JNI, native, and
COUNT pushdown paths with dual serialization
- PaimonScanPlanProvider: Core scan planning using Paimon SDK (ReadBuilder →
TableScan → Split), converting to ConnectorScanRange with JNI/native dispatch
- Modified PaimonTableHandle to carry transient Table reference
- Modified PaimonConnectorMetadata to store Table in handle and add getColumnHandles()
- Modified PaimonConnector to add getScanPlanProvider()
- Modified PluginDrivenScanNode to add paimon dispatch (setPaimonParams),
getSerializedTable() override, and scan-level paimon params (predicate + options)
None
- Test: No need to test (compile-only migration, runtime wiring not yet active)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove old internal ES table code and clean up references
Issue Number: close #xxx
Problem Summary: The old internal Elasticsearch table implementation (EsScanNode,
EsExternalCatalog, EsExternalTable, etc.) has been superseded by the ES Catalog
connector plugin. This commit removes the legacy ES code from fe-core and cleans
up all references throughout the codebase.
Changes:
- Delete 21 old ES datasource files from datasource/es/
- Delete 3 nereids plan classes (LogicalEsScan, PhysicalEsScan, LogicalEsScanToPhysicalEsScan)
- Delete 6 old ES test files from external/elasticsearch/
- Clean up ES references in 13+ files (Env.java, ExternalCatalog.java, ESCatalogAction.java,
PhysicalPlanTranslator.java, BindRelation.java, RuleSet.java, RelationVisitor.java,
StatsCalculator.java, CostModel.java, ChildOutputPropertyDeriver.java,
TopnFilterPushDownVisitor.java, GsonUtils.java, TableIf.java, etc.)
- Preserve EsTable.java and EsResource.java for persistence compatibility
- Add ES stub in SHOW CREATE TABLE for deprecation notice
None
- Test: Manual test - fe-core compiles successfully, ES connector tests pass
- Behavior changed: Yes - old internal ES table queries will no longer work;
users should use ES Catalog instead
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Add metadata methods to JDBC connector for migration parity
Issue Number: close #xxx
Problem Summary: During JDBC connector modular migration from fe-core to
fe-connector-jdbc, several metadata methods were missing in the new connector
client and SPI interfaces. This commit adds the missing methods to achieve
parity with the old JdbcClient.
None
- Test: No need to test - pure method additions with no behavior change,
existing integration paths still use old JdbcClient
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Implement JDBC write path through connector plugin pipeline
Issue Number: close #xxx
Problem Summary: The JDBC INSERT INTO write path was not wired through the
new connector plugin pipeline. This commit implements the complete JDBC write
path so that INSERT INTO on PluginDriven JDBC tables produces the correct
TJdbcTableSink thrift structure, matching the old JdbcTableSink behavior.
Changes:
- JdbcConnectorMetadata now implements ConnectorWriteOps with getWriteConfig(),
beginInsert(), finishInsert() — builds INSERT SQL via JdbcIdentifierQuoter
and populates all JDBC connection/pool properties
- JdbcIdentifierQuoter gains buildInsertSql() for parameterized INSERT SQL
- PluginDrivenTableSink.bindJdbcWriteSink() now populates ALL TJdbcTable
fields: catalog_id, driver_checksum, table_name, resource_name, and all
connection pool settings; also sets TOdbcTableType on TJdbcTableSink
- PluginDrivenInsertExecutor resolves write type from connector metadata and
returns TransactionType.JDBC for JDBC writes (was hardcoded to HMS)
- PhysicalPlanTranslator passes actual column list to getWriteConfig()
(was passing empty list)
- ConnectorSessionBuilder now propagates enable_odbc_transcation session var
- Explain output enhanced for JDBC write sinks (shows table type, SQL, txn)
None
- Test: No need to test - infrastructure wiring, no end-to-end JDBC catalog
available in unit test environment; will be tested with regression tests
after full migration
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Port JDBC function pushdown and ClickHouse FINAL to connector module
Issue Number: close #xxx
Problem Summary:
The JDBC connector module (fe-connector-jdbc) lacked function pushdown support
and ClickHouse FINAL query support. These features existed only in the old
fe-core JdbcScanNode code path. This commit ports them to the connector SPI
pipeline so PluginDrivenScanNode-based JDBC scans have feature parity.
Key changes:
- New JdbcFunctionPushdownConfig: per-DB function whitelist/blacklist/replacement
rules with time arithmetic rewriting and JSON-configurable overrides
- Extended JdbcQueryBuilder: ConnectorFunctionCall/Like/Between to SQL conversion,
conjunct pushdown guards (Oracle NULL, cast, function-containing expressions),
ClickHouse SETTINGS final=1 support
- Updated JdbcScanPlanProvider to create pushdown config and pass session props
- Plumbed 4 session variables through ConnectorSessionBuilder:
enable_ext_func_pred_pushdown, jdbc_clickhouse_query_final,
enable_jdbc_oracle_null_predicate_push_down, enable_jdbc_cast_predicate_push_down
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Route JDBC TVF and CallExecuteStmt through connector SPI
Issue Number: close #xxx
Problem Summary:
JdbcQueryTableValueFunction and CallExecuteStmtFunc only supported the old
JdbcExternalCatalog code path. Newly created JDBC catalogs use
PluginDrivenExternalCatalog via the SPI connector, so these features were
broken for new catalogs.
Key changes:
- New PassthroughQueryTableHandle in connector API for raw query TVF scans
- JdbcQueryTableValueFunction: dual path — uses ConnectorMetadata for schema
discovery and PluginDrivenScanNode for scan planning when catalog is
PluginDrivenExternalCatalog
- CallExecuteStmtFunc: routes to connector metadata.executeStmt() for
PluginDrivenExternalCatalog, falls back to old JdbcExternalCatalog path
- QueryTableValueFunction factory: detects PluginDrivenExternalCatalog
- JdbcScanPlanProvider: handles PassthroughQueryTableHandle for TVF queries
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Migrate JDBC identifier mapping to connector module
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
The JDBC identifier mapping logic (lower_case_meta_names, meta_names_mapping,
lower_case_table_names) was tightly coupled to fe-core via JdbcIdentifierMapping
which depends on Jackson ObjectMapper. This commit migrates the functionality
to the connector-jdbc module as part of the SPI modularization effort.
Key changes:
- Add ConnectorIdentifierOps SPI interface in connector-api with 3 name mapping
methods (fromRemoteDatabaseName, fromRemoteTableName, fromRemoteColumnName)
- ConnectorMetadata now extends ConnectorIdentifierOps
- Create JdbcIdentifierMapper in connector-jdbc: pure Java reimplementation
with regex-based JSON parsing (no Jackson dependency), full validation with
case-conflict detection
- Wire identifier mapping through getColumnHandles() and getWriteConfig() so
column names are properly mapped in both read and write paths
- Add lower_case_table_names from GlobalVariable to ConnectorSessionBuilder
- PluginDrivenExternalCatalog overrides fromRemoteDatabaseName/TableName to
delegate to connector metadata
- PluginDrivenExternalTable.initSchema() applies fromRemoteColumnName mapping
None
- Test: Manual test / Compilation verified
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove old JDBC connector code after migration to plugin-driven architecture
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: After migrating JDBC connector to fe-connector-jdbc module with
plugin-driven architecture (PluginDrivenExternalCatalog/Database/Table), the old
JDBC code in fe-core/datasource/jdbc/ is dead code. This commit:
1. Adds Gson compatible subtype adapters so persisted metadata with old class names
(JdbcExternalCatalog/Database/Table) deserializes correctly as PluginDriven* classes
2. Deletes 21 old JDBC files: JdbcExternalCatalog, JdbcExternalDatabase,
JdbcExternalTable, JdbcNameUtil, JdbcSchemaCacheValue, JdbcTableSink, JdbcScanNode,
JdbcSplit, JdbcFunctionPushDownRule, IdentifierMapping, JdbcIdentifierMapping,
UnboundJdbcTableSink, LogicalJdbcTableSink, PhysicalJdbcTableSink,
JdbcInsertExecutor, JdbcInsertCommandContext,
LogicalJdbcScanToPhysicalJdbcScan, LogicalJdbcTableSinkToPhysicalJdbcTableSink,
LogicalJdbcScan, PhysicalJdbcScan
3. Cleans up all references in 23 files across nereids visitors, derivers, rules,
plan translator, expression rewrite, and external function rules
4. Inlines function pushdown constants from deleted JdbcFunctionPushDownRule into
ExternalFunctionRules
JdbcClient hierarchy (16 files) is kept for CDC binlog functionality.
None
- Test: No need to test - pure code deletion of dead code after migration
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix connector plugin packaging and ES scan node bugs
Issue Number: close #xxx
Problem Summary:
Three issues causing CI regression test failures (build 926124):
1. **JDBC/Hive/MaxCompute plugin JARs not discovered at runtime**
The assembly descriptors for these connectors used `includeBaseDirectory=true`,
creating a nested subdirectory inside the zip (e.g., `fe-connector-jdbc/lib/*.jar`).
When build.sh unzips into `plugins/connector/jdbc/`, the actual path becomes
`plugins/connector/jdbc/fe-connector-jdbc/lib/*.jar`, but DirectoryPluginRuntimeManager
expects JARs at `plugins/connector/jdbc/*.jar` and `plugins/connector/jdbc/lib/*.jar`.
Fix: Set `includeBaseDirectory=false` and place plugin JAR at root + deps in lib/,
matching the working ES/iceberg/paimon assembly layout.
2. **ES scan node URL parsing fails for http:// prefixed hosts**
PluginDrivenEsScanNode.convertToThrift() split host strings by ":" naively,
so `http://172.16.0.98:29200` was parsed as host="http", port="//172.16.0.98"
causing NumberFormatException. Fix: Strip scheme prefix before parsing host:port.
3. **ES JSONB type not recognized by ConnectorColumnConverter**
ES connector returns JSONB type but ScalarType.createType() only handles "JSON".
ConnectorColumnConverter now maps JSONB to JSON explicitly.
None
- Test: Manual verification of zip layout; CI rerun pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Resolve JDBC driver_url via ConnectorContext environment
Problem Summary: JDBC connector plugin fails to load driver classes because
plain driver filenames (e.g., "mysql-connector-j-8.4.0.jar") are not resolved
to absolute file:// URLs using Config.jdbc_drivers_dir.
The fix adds a generic getEnvironment() method to ConnectorContext SPI,
which fe-core populates with system configs (jdbc_drivers_dir, doris_home).
The JDBC connector uses this environment to resolve driver URLs, keeping
fe-core free of JDBC-specific logic.
None
- Test: Manual test - verified driver URL resolution logic
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Implement toThrift() for PluginDrivenExternalTable via connector SPI
Problem Summary: PluginDrivenExternalTable.toThrift() inherited the base
ExternalTable implementation which returns null, causing NPE during query
fragment serialization when BE needs TTableDescriptor.
The fix adds:
1. ConnectorTableOps.getTableDescriptorProperties() SPI method that lets
connectors declare the BE table descriptor type and properties
2. JdbcConnectorMetadata returns JDBC_TABLE type with all connection/pool
properties needed by TJdbcTable
3. EsConnectorMetadata returns ES_TABLE type
4. PluginDrivenExternalTable.toThrift() reads these properties and
constructs the appropriate typed TTableDescriptor (TJdbcTable/TEsTable)
None
- Test: Manual test — JDBC catalog query no longer hits null TTableDescriptor
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Connector directly builds TTableDescriptor via fe-thrift
Issue Number: close #xxx
Problem Summary: Previously PluginDrivenExternalTable.toThrift() used a Map-based
getTableDescriptorProperties() intermediary to build TTableDescriptor, which required
fe-core to contain connector-specific logic for constructing TJdbcTable/TEsTable.
This refactoring lets each connector module directly depend on fe-thrift (provided scope)
and build its own TTableDescriptor in buildTableDescriptor(), making PluginDrivenExternalTable
a pure delegator with zero connector-specific code.
Key changes:
- ConnectorTableOps: replaced getTableDescriptorProperties() with buildTableDescriptor()
returning TTableDescriptor directly
- JdbcConnectorMetadata: builds TJdbcTable + TTableDescriptor directly
- EsConnectorMetadata: builds TEsTable + TTableDescriptor directly
- PluginDrivenExternalTable.toThrift(): simplified to pure delegation
- Added fe-thrift as provided dependency to fe-connector-api, fe-connector-jdbc,
fe-connector-es (Maven provided scope is NOT transitive)
- All 4 plugin assembly ZIPs exclude fe-thrift and libthrift
None
- Test: Manual test (compilation verified, runtime pending BIT/BOOLEAN display fix)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix SPI column attributes, type mapping, EXPLAIN format, and DB/schema mapping
Issue Number: close #xxx
Problem Summary: Fix 13 CI test failures in the External Regression pipeline
caused by the connector migration. This batch addresses 4 root cause categories:
1. **isKey lost (5 tests)**: ConnectorColumn had no isKey field; convertColumn()
hardcoded false. Added isKey to ConnectorColumn, propagated through
ConnectorColumnConverter, populated from JDBC getPrimaryKeys().
2. **Type mapping (3 tests)**: HLL/BITMAP/QUANTILE_STATE types were not handled
in JdbcMySQLConnectorClient.mapSignedType(), falling to UNSUPPORTED. Added
the missing cases for Doris-to-Doris JDBC catalog support.
3. **DB/schema mapping (3 tests)**: ClickHouse connector ignored databaseterm
URL parameter; OceanBase only delegated type mapping (not metadata methods);
PostgreSQL missing bit/varbit/hstore type mappings.
4. **EXPLAIN format (2 tests)**: PluginDrivenScanNode did not output QUERY: line
in EXPLAIN. Added getNodeExplainString() override that shows pushed-down SQL
from JdbcScanPlanProvider.getScanNodeProperties().
None
- Test: Regression test (pending CI run)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Wire write path, statistics, row count, and test expectations through connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Fix 5 regression test failures in the connector SPI migration (Batch 2):
1. InsertUtils write sink: Add UnboundConnectorTableSink handling in
getTargetTableQualified() for write transaction support.
2. Statistics collection: Override createAnalysisTask() in
PluginDrivenExternalTable to return ExternalAnalysisTask.
3. Row count estimation: Add fetchRowCount() to PluginDrivenExternalTable
delegating to ConnectorMetadata.getTableStatistics(). Add
JdbcConnectorMetadata.getTableStatistics() and MySQL-specific
getRowCount() querying INFORMATION_SCHEMA.
4. CallExecuteStmt test: Update expected exception message from
"Only support JDBC catalog" to "executeStmt not supported" since
HMS catalogs now route through PluginDrivenExternalCatalog.
5. Oracle CHAR padding: Update test_oracle_jdbc_catalog.out to match
current OracleTypeHandler behavior which trims CHAR trailing spaces.
This is a pre-existing behavior change from master commit 9770ecf8a59
(JdbcTypeHandler refactor) where the .out file was never updated.
None
- Test: Regression test expectations updated
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ES URL parsing, predicate pushdown, date compat, and TVF classloader
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Four bugs introduced during the connector SPI migration:
1. **ES double-port URL** (test_es_query_no_http_url): EsNodeInfo seed constructor
produced "host:port:80" for inputs like "host:9200" without a scheme prefix.
Fixed by correctly parsing host and port when split(":") yields 2 parts.
2. **ES LIKE not pushed down** (test_es_query): EsQueryDslBuilder.likeToDsl()
checked column2typeMap BEFORE resolving keyword sub-fields from fieldsContext.
Text fields with .keyword sub-fields were rejected. Fixed by checking
fieldsContext.containsKey() first, matching the old QueryBuilders behavior.
3. **ES date equality returns empty** (test_es_query_nereids): compatDefaultDate()
used ZoneOffset.UTC, but the old Joda-Time code used the system default timezone.
When JVM runs in CST (UTC+8), "2022-08-08 08:00:00" should map to midnight UTC,
not 8am UTC. Fixed by using ZoneId.systemDefault().
4. **TVF cross-catalog ClassCastException** (test_query_tvf_cross_catalog,
test_query_tvf_auth): PassthroughQueryTableHandle (in org.apache.doris.connector.api)
was loaded by both app classloader and ChildFirstClassLoader, causing instanceof
to fail. Fixed by adding "org.apache.doris.connector.api." to the parent-first
package list in ChildFirstClassLoader.
None
- Test: Regression test (CI pipeline)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](be) Fix CTAS ArrayStoreException for MySQL YEAR column
Issue Number: close #xxx
Problem Summary: MySQL YEAR columns mapped to Doris SMALLINT caused
ArrayStoreException: java.sql.Date in JdbcJniScanner.getNext().
Root cause: MySQLTypeHandler.getColumnValue() used untyped
rs.getObject(columnIndex) for TINYINT/SMALLINT, which returns
java.sql.Date for YEAR columns when MySQL Connector/J default
yearIsDateType=true is in effect. The SMALLINT converter then could
not handle java.sql.Date (falls through to return input as-is),
and storing java.sql.Date into Short[] caused ArrayStoreException.
Fix: Use typed rs.getObject(columnIndex, Integer.class) for TINYINT
and SMALLINT to force the JDBC driver to convert to Integer regardless
of the underlying MySQL column type.
None
- Test: Regression test (test_mysql_all_types_ctas)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](build) Change mvn package to install to fix MDEP-187 reactor dependency issue
Problem Summary: The fe-core module copy-dependencies goal fails with
MDEP-187 error when new reactor modules (fe-connector-api,
fe-connector-spi) have not been installed to the local Maven
repository. This is a known Maven limitation where copy-dependencies
cannot resolve reactor artifacts during the same mvn package run.
Fix: Change build.sh from mvn package to mvn install so all reactor
artifacts are installed to the local repo before copy-dependencies runs.
None
- Test: Manual test (build.sh --fe)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](build) Add Apache license headers to SPI service files
Problem Summary: ConnectorProvider service files in META-INF/services/
were missing Apache license headers, causing the license checker to fail.
None
- Test: No need to test (license headers only)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](es) Fix ES date predicate pushdown timezone conversion
Problem Summary: ES date equality queries returned empty results because
compatDefaultDate() could not parse ISO-formatted datetime strings produced
by extractLiteralValue(). The extractLiteralValue() method formats
LocalDateTime as "2022-08-08T08:00:00" (ISO with T separator), but
compatDefaultDate() only tried to parse "yyyy-MM-dd HH:mm:ss" (space
separator). The parse failure caused the value to be sent without timezone
conversion, so ES interpreted it as UTC instead of the local timezone.
Fix: Add ISO_LOCAL_DATE_TIME parsing as the first attempt in
compatDefaultDate(), before falling back to space-separated format and
date-only format.
None
- Test: Regression test (test_es_query_nereids sql63)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix JDBC connector type mapping, auth, and EXPLAIN regressions
Issue Number: close #62183
Problem Summary: Fix multiple regression failures in the connector-based
JDBC/ES external catalog migration. This commit addresses 8 root causes
covering 20+ test failures from CI External Regression pipeline build 926236.
Fixes include:
- detectDoris() initialization ordering: moved from constructor to
postInitialize() hook so data source is live before detection runs.
Added dorisTypeToConnectorType() for Doris-to-Doris type mapping
including HLL, BITMAP, QUANTILE_STATE types.
- Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match
"TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc.
- Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0
for integer branch, and match old boundary thresholds.
- ClickHouse DB listing: use SHOW DATABASES for full listing;
fix databaseTermIsCatalog inversion for old drivers.
- DATETIMEV2 precision: ConnectorColumnConverter reads precision
(not scale) for datetime fractional seconds, matching connector
encoding convention.
- EXPLAIN cache: invalidate scanNodeProperties after convertPredicate()
so pushed conjuncts are reflected in EXPLAIN output.
- ExecutionAuthenticator: call initPreExecutionAuthenticator() in
PluginDrivenExternalCatalog.initLocalObjectsImpl().
- PassthroughQueryTableHandle: add instanceof guards in
JdbcConnectorMetadata to prevent ClassCastException for TVF.
- isKey: hardcode true for all columns matching legacy behavior.
None
- Test: Regression test (CI External Regression pipeline)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ES LIKE pushdown for Nereids planner
Problem Summary: Nereids planner translates Like/Regexp expressions to
FunctionCallExpr (via ExpressionTranslator.visitScalarFunction) rather
than LikePredicate. ExprToConnectorExpressionConverter only checked for
instanceof LikePredicate, so the LIKE predicate was converted to a
ConnectorFunctionCall instead of ConnectorLike. This caused ES query DSL
builder to not recognize it as a LIKE and fall back to match_all,
returning all documents instead of only those matching the wildcard.
Fix: Detect FunctionCallExpr with function name "like" or "regexp" in
ExprToConnectorExpressionConverter and convert to ConnectorLike.
None
- Test: Regression test (test_es_query sql_5_29/sql_6_29)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Add JDBC URL normalization for connector migration
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The connector migration moved JDBC catalog logic from
fe-core to fe-connector-jdbc, but the URL normalization step was lost.
The old path called JdbcResource.handleJdbcUrl() which added critical
JDBC driver parameters. Without these, OceanBase tests fail with
timezone shifts (8h offset), lost fractional seconds, BOOLEAN display
as true/false instead of 1/0, and DOUBLE precision loss.
The fix adds JdbcUrlNormalizer in fe-connector-jdbc that replicates the
same parameter injection logic:
- MySQL/OceanBase: yearIsDateType=false, tinyInt1isBit=false,
useUnicode=true, characterEncoding=utf-8, rewriteBatchedStatements=true
- OceanBase additionally: useCursorFetch=true
- PostgreSQL: reWriteBatchedInserts=true
- SQL Server: useBulkCopyForBatchInsert=true
The normalization is applied once in JdbcDorisConnector constructor,
so all downstream consumers (client, metadata, scan plan provider)
use the normalized URL.
None
- Test: Regression test (test_oceanbase_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Stop filtering ClickHouse system database from catalog listing
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The new JdbcClickHouseConnectorClient incorrectly filtered
the ClickHouse "system" database in getFilterInternalDatabases(). The old
JdbcClickHouseClient never overrode this method, so it used the base class
default which only filters information_schema, performance_schema, and mysql.
The system database contains useful tables (query_log, processes, etc.) and
should remain accessible to users.
None
- Test: Regression test (test_clickhouse_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ClickHouse JDBC catalog regression issues in plugin-driven architecture
Issue Number: close #xxx
Problem Summary: Multiple regression issues found after migr…
Contributor
Author
|
run buildall |
morningman
added a commit
to morningman/doris
that referenced
this pull request
Apr 13, 2026
…faces
Issue Number: close #xxx
Problem Summary: Add the fe-connector module hierarchy (fe-connector-api and
fe-connector-spi) as the foundation for Catalog SPI modularization. This
follows the same pattern as fe-filesystem, enabling external data source
connectors to be developed and deployed as independent plugins.
fe-connector-api defines 30 zero-dependency interfaces including:
- Opaque handle types (ConnectorTableHandle, ConnectorColumnHandle, etc.)
- ConnectorMetadata composed from sub-interfaces (SchemaOps, TableOps,
PushdownOps, StatisticsOps, WriteOps)
- apply* pushdown negotiation (FilterApplicationResult, ProjectionApplicationResult)
- ConnectorScanPlanProvider for split generation
- ConnectorCapability enum for capability declaration
- Lightweight type system (ConnectorColumn, ConnectorType) independent of fe-core
fe-connector-spi defines the provider SPI:
- ConnectorProvider extends PluginFactory for ServiceLoader discovery
- ConnectorContext for engine-to-connector runtime services
None
- Test: No need to test - module skeleton with interfaces only, no behavioral changes
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorPluginManager and session bridge for connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: This step adds the connector plugin loading infrastructure
to fe-core, mirroring the FileSystemPluginManager pattern. It enables
connector plugins to be discovered via ServiceLoader (builtins) and loaded
from the plugin directory at runtime via DirectoryPluginRuntimeManager with
ChildFirst ClassLoader isolation.
Key additions:
- ConnectorPluginManager: manages ConnectorProvider lifecycle (builtin +
directory-based plugin loading)
- ConnectorFactory: static factory singleton initialized by Env at startup
- ConnectorSessionBuilder: bridges ConnectContext → ConnectorSession for
ClassLoader-safe session passing across SPI boundary
- ConnectorSessionImpl: immutable ConnectorSession implementation
- DefaultConnectorContext: minimal ConnectorContext providing catalogName,
catalogId, and FS access
- Config.connector_plugin_root: configurable plugin directory
- Env.initConnectorPluginManager(): startup initialization hook
None
- Test: No need to test - infrastructure only, no connector plugins exist yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove Iceberg SDK dependency from ExternalMetadataOps interface
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The ExternalMetadataOps interface (a core fe-core interface
for external catalog metadata operations) had a direct import of
org.apache.iceberg.view.View, coupling the interface to Iceberg SDK. The
ExternalMetadataOperations factory class also imported all connector-specific
types (Iceberg, Paimon, MaxCompute, Hive), creating unnecessary coupling.
Changes:
- ExternalMetadataOps.loadView() return type changed from View to Object.
The method is only overridden by IcebergMetadataOps, and all callers are
in iceberg-specific code that already imports View for casting.
- Deleted ExternalMetadataOperations factory class entirely. Each catalog
now directly constructs its own MetadataOps (the factory was a thin
indirection adding no value since each caller already knows its type).
- Removed unused View import from IcebergMetadataOps (return type is now
Object; iceberg view.View is only used in IcebergExternalMetaCache).
None
- Test: No need to test - pure refactoring, no behavioral change
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Abstract Hadoop Configuration to Map-based catalog properties
Issue Number: close #xxx
Problem Summary: ExternalCatalog.getConfiguration() returns org.apache.hadoop.conf.Configuration,
which leaks Hadoop types into the core catalog interface. This blocks future SPI extraction of
connector modules since fe-connector-api must be free of Hadoop dependencies.
This commit introduces getHadoopProperties() returning Map<String, String> and a static
buildHadoopConfiguration() utility, then migrates all callers (HudiExternalMetaCache,
IcebergUtils, HiveMetaStoreClientHelper) to the new pattern. The original getConfiguration()
is deprecated.
None
- Test: No need to test - pure refactor, all callers produce identical Configuration objects
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add PluginDrivenExternalCatalog adapter and SPI-based CatalogFactory
Issue Number: close #xxx
Problem Summary: The CatalogFactory uses hardcoded switch-case to create catalogs. This commit
introduces a SPI-first creation path: ConnectorFactory.createConnector() is tried before falling
back to the built-in switch-case. A new PluginDrivenExternalCatalog bridges SPI Connector instances
with the existing ExternalCatalog hierarchy, enabling third-party catalog plugins.
None
- Test: No need to test — no SPI connector plugins exist yet, all catalogs still use the fallback switch-case path. Behavior is unchanged.
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add fe-connector-es module with SPI metadata implementation
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Creates the first real connector plugin module (Elasticsearch)
implementing the connector SPI interfaces. This validates the entire SPI design
with a working metadata-only implementation. The ES scan path remains in fe-core
temporarily — only metadata operations (list databases, list tables, get schema)
are handled via the SPI.
The module includes:
- EsConnectorProvider: SPI entry point discovered via ServiceLoader
- EsConnector: Main connector, creates metadata instances
- EsConnectorMetadata: Lists ES indices as tables, parses mappings to schema
- EsConnectorRestClient: HTTP client adapted from fe-core (no fe-core deps)
- EsTypeMapping: ES type → ConnectorType mapping
- EsTableHandle: Opaque handle for ES indices
- EsConnectorProperties: Property constants and compatibility processing
- Plugin assembly ZIP descriptor for runtime deployment
None
- Test: No need to test — metadata-only SPI implementation, fe-core ES path unchanged
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add shard routing and metadata pipeline to fe-connector-es
Issue Number: close #xxx
Problem Summary: The initial fe-connector-es module (Phase 1) only supported basic
metadata operations (list databases, list tables, get schema). This commit adds
Phase 2 capabilities: shard routing discovery, node info resolution, mapping field
analysis (keyword sniff, doc_value, date compat), and full metadata orchestration.
These are prerequisites for the future ConnectorScanPlanProvider which will generate
TScanRangeLocations from the plugin side.
New files (8):
- EsMajorVersion: ES version detection (adapted from fe-core, zero fe-core deps)
- EsNodeInfo: ES node with HTTP address (host+port instead of TNetworkAddress)
- EsShardRouting: Shard routing entry (host+port instead of TNetworkAddress)
- EsShardPartitions: Shard partition map with _search_shards JSON parsing
- EsFieldContext: Field context holder (fetchFields, docValue, dateCompat maps)
- EsMappingUtils: Mapping parsing + field resolution (from EsUtil + MappingPhase)
- EsMetadataState: Full metadata state (adapted from SearchContext)
- EsMetadataFetcher: Metadata fetch orchestrator (adapted from EsMetaStateTracker)
Enhanced files (3):
- EsConnectorRestClient: added searchShards(), getHttpNodes(), get() methods
- EsConnectorMetadata: added fetchMetadataState() for full metadata pipeline
- EsConnectorProperties: added MAPPING_TYPE constant
None
- Test: No need to test — plugin module only, not activated in production
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-trino plugin module (Step 7 Phase 1)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Trino Connector metadata logic from fe-core into an
independent plugin module fe-connector-trino, continuing the Catalog SPI
modularization. This phase migrates the metadata-only layer (schema discovery,
type mapping, Trino plugin bootstrap) while leaving scan/predicate code in
fe-core for Step 16.
Key files:
- TrinoConnectorProvider: SPI entry point (type="trino-connector")
- TrinoDorisConnector: Lazy init, wraps Trino Connector + Session
- TrinoBootstrap: Singleton Trino plugin infrastructure init
- TrinoConnectorDorisMetadata: ConnectorMetadata impl (list/get/schema)
- TrinoTypeMapping: Trino SPI types -> ConnectorType (15+ mappings)
- TrinoPluginManager/TrinoServicesProvider: Adapted from fe-common
Design decisions:
- Copied fe-common classes (~290 lines each) into plugin rather than depending
on fe-common (which has heavy transitive deps like hadoop-common)
- TrinoBootstrap uses singleton pattern for plugin loading, per-catalog
connector creation
- Plugin dir resolution: property -> DORIS_HOME/plugins/connectors -> fallback
None
- Test: No need to test (metadata-only extraction, fe-core code unchanged)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-maxcompute plugin module (Step 8)
Problem Summary: Extract MaxCompute connector metadata code into an independent
plugin module as part of the Catalog SPI modularization effort.
Changes:
- Create fe-connector-maxcompute module with 11 Java files + build config
- MaxComputeConnectorProvider (SPI entry, type="max_compute")
- MaxComputeDorisConnector (Odps client lifecycle, lazy init)
- MaxComputeConnectorMetadata (listDatabases, listTables, getTableHandle, getTableSchema)
- MCTypeMapping (15+ ODPS type -> ConnectorType mappings including nested ARRAY/MAP/STRUCT)
- McStructureHelper (namespace schema abstraction, copied from fe-core)
- MCConnectorClientFactory, MCConnectorEndpoint, MCConnectorProperties (utilities)
- MaxComputeTableHandle (opaque handle wrapping ODPS Table + TableIdentifier)
- Enhance ConnectorType API with factory methods (of, arrayOf, mapOf, structOf)
- Make Connector.getScanPlanProvider() a default method (scan planning is optional for Phase 1)
Phase 1 scope: metadata-only. ScanNode, Transaction, DDL operations stay in fe-core.
None
- Test: No need to test - Phase 1 metadata extraction, SPI plugin not yet activated
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-jdbc plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extracts JDBC connector metadata logic into an independent
fe-connector-jdbc plugin module as part of the Catalog SPI modularization.
This is Step 10 of the connector SPI plan.
The module implements:
- JdbcConnectorProvider (SPI entry point, type="jdbc")
- JdbcDorisConnector (manages HikariCP connection pool lifecycle)
- JdbcConnectorMetadata (delegates to JdbcConnectorClient for metadata discovery)
- JdbcConnectorClient base class with factory method dispatching to 10 DB-specific subclasses
- All 10 DB-specific clients: MySQL, PostgreSQL, Oracle, ClickHouse, SQLServer,
SAP HANA, Trino, OceanBase, DB2, GBase
- JdbcFieldInfo (column metadata, adapted from JdbcFieldSchema)
- JdbcDbType enum, JdbcTableHandle, JdbcConnectorProperties
Each subclass maps JDBC types to ConnectorType (the SPI type system) instead
of fe-core Type, achieving zero fe-core dependency.
None
- Test: No need to test (Phase 1 metadata-only extraction, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Extract HMS shared module to fe-connector-hms
Issue Number: close #xxx
Problem Summary: The HMS (Hive MetaStore) client code in fe-core is deeply
coupled to internal types (Column, Config, ExecutionAuthenticator, etc.), making
it impossible to directly extract. This commit creates a new clean HMS client
library module (fe-connector-hms) using SPI types, which will be shared by
future fe-connector-hive and fe-connector-hudi plugins.
Key design decisions:
- New clean interfaces using ConnectorType/ConnectorColumn (not code extraction)
- Connection-pooled Thrift client with taint-and-destroy error handling
- Supports HMS/DLF/Glue metastore types via string-based class dispatch
- Auth via functional interface (AuthAction) replacing fe-core ExecutionAuthenticator
- HmsTypeMapping mirrors HiveMetaStoreClientHelper logic but returns ConnectorType
Module contents (9 Java files, ~1440 lines):
- HmsClient: Clean interface for HMS operations
- ThriftHmsClient: Pooled Thrift implementation with ClassLoader safety
- HmsClientConfig/HmsConfHelper: Configuration DTOs and HiveConf creation
- HmsTableInfo/HmsDatabaseInfo/HmsPartitionInfo: Immutable DTOs
- HmsTypeMapping: Hive type string to ConnectorType mapping
- HmsClientException: Runtime exception class
None
- Test: No need to test - library module with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-hive plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Hive connector metadata operations into an independent
fe-connector-hive plugin module as Phase 1 of the Catalog SPI modularization.
This creates the fe-connector-hive plugin that:
- Implements ConnectorProvider (type="hms") for SPI-based catalog creation
- Provides HiveConnectorMetadata with listDatabases, listTables, getTableHandle,
getTableSchema operations via the shared fe-connector-hms client
- Detects table formats (HIVE/HUDI/ICEBERG) from HMS table metadata
- Supports configurable type mapping options (binary-as-string, timestamp-tz)
- Manages HmsClient lifecycle with lazy initialization and proper shutdown
Files: 10 new files (816 lines), 2 modified files
Dependencies: fe-connector-spi + fe-connector-hms + log4j-api
None
- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-paimon plugin module (Phase 1: metadata-only)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Extract Paimon connector metadata operations into an independent
fe-connector-paimon plugin module as Phase 1 of the Catalog SPI modularization.
This creates the fe-connector-paimon plugin that:
- Implements ConnectorProvider (type="paimon") for SPI-based catalog creation
- Creates Paimon Catalog instances directly via Paimon SDK (Options + CatalogContext
+ CatalogFactory) supporting all backends: filesystem, HMS, DLF, REST, JDBC
- Provides PaimonConnectorMetadata with listDatabases, listTables, getTableHandle,
getTableSchema operations using the Paimon Catalog API
- Includes PaimonTypeMapping that converts Paimon DataType to ConnectorType,
mirroring existing PaimonUtil.paimonTypeToDorisType logic
- Supports configurable type mapping options (binary-as-varbinary, timestamp-tz)
- Manages Paimon Catalog lifecycle with lazy init and proper shutdown
Files: 8 new files (805 lines), 1 modified file
Dependencies: fe-connector-spi + paimon-core + log4j-api
None
- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add fe-connector-hudi Phase 1 plugin module
Issue Number: close #xxx
Problem Summary: Extract Hudi connector metadata operations into an
independent SPI plugin module (fe-connector-hudi) as part of the
catalog connector SPI modularization effort.
The Hudi connector plugin provides Phase 1 read-only metadata:
- List databases and tables via HMS (shared fe-connector-hms client)
- Get table schema from HoodieTableMetaClient Avro schema
(authoritative for schema-evolved Hudi tables)
- Detect Hudi table type (COW vs MOR) from input format
- Avro-to-ConnectorType mapping (HudiTypeMapping)
Dependencies: fe-connector-spi + fe-connector-hms + hudi-common +
hudi-hadoop-mr.
Scan planning, incremental query, statistics, and event processing
remain in fe-core temporarily.
None
- Test: No need to test (Phase 1 SPI module, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Extract fe-connector-iceberg Phase 1 — metadata-only SPI plugin
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Step 15 of the Catalog Connector SPI modularization. Extracts the Iceberg
connector into an independent plugin module (fe-connector-iceberg) that
provides read-only metadata operations through the Connector SPI.
Phase 1 implements:
- IcebergConnectorProvider: SPI entry point, type="iceberg"
- IcebergConnector: Lifecycle management, creates Iceberg SDK Catalog via
CatalogUtil.buildIcebergCatalog() — supports all 7 backends (REST, HMS,
Glue, DLF, JDBC, Hadoop, S3Tables)
- IcebergConnectorMetadata: listDatabaseNames (SupportsNamespaces), listTableNames,
getTableHandle, getTableSchema via Iceberg SDK Catalog API
- IcebergTypeMapping: Iceberg types → ConnectorType (BOOLEAN, INT, LONG, FLOAT,
DOUBLE, STRING, UUID/BINARY→VARBINARY or STRING, DECIMAL, DATE, TIMESTAMP,
LIST, MAP, STRUCT) with enableMappingVarbinary and enableMappingTimestampTz flags
- IcebergTableHandle: Opaque handle carrying db+table coordinates
- IcebergConnectorProperties: Property key constants
Key design decisions:
- No fe-connector-hms dependency: Iceberg SDK HiveCatalog handles HMS internally
- Property-driven catalog creation: CatalogUtil.buildIcebergCatalog() handles
all backends, subclass dispatch is just setting catalog-impl
- Hadoop Configuration built from user properties (hadoop.*, fs.*, dfs.*, hive.*)
6 Java files, ~770 lines added.
None
- Test: No need to test — Phase 1 metadata-only SPI module, not yet wired into fe-core runtime
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add PluginDrivenExternalTable schema support and GSON registration
Issue Number: close #xxx
Problem Summary:
PluginDrivenExternalTable was a minimal stub with no schema retrieval capability.
Plugin-driven catalogs could list databases and tables but could not fetch table
schemas, making them non-functional for queries. Additionally, the PluginDriven*
types were not registered in GsonUtils, preventing GSON serialization/deserialization
of plugin-driven catalog metadata for FE persistence.
This commit implements Step 18 of the Catalog Connector SPI plan:
1. **ConnectorColumnConverter** (new): Converts between the connector SPI type system
(ConnectorColumn/ConnectorType) and Doris internal types (Column/Type). Handles
all scalar types plus complex types (ARRAY, MAP, STRUCT) recursively, with proper
precision/scale handling for CHAR, VARCHAR, DECIMAL, and DATETIMEV2.
2. **PluginDrivenExternalTable**: Overrides initSchema() to fetch table schema from
the connector SPI. Uses ConnectorMetadata.getTableHandle() + getTableSchema(),
then converts via ConnectorColumnConverter.
3. **PluginDrivenExternalCatalog**: Makes buildConnectorSession() package-private so
PluginDrivenExternalTable can build sessions during schema init.
4. **GsonUtils**: Registers PluginDrivenExternalCatalog, PluginDrivenExternalDatabase,
and PluginDrivenExternalTable in the respective RuntimeTypeAdapterFactory instances
for proper GSON serialization/deserialization.
None
- Test: No need to test - infrastructure code with no active connector plugins yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorExpression framework and filter pushdown types
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Step 19 of the Catalog SPI modularization. Replaces the
placeholder ConnectorExpression class with a proper expression tree hierarchy
for cross-SPI filter/projection pushdown negotiation. Adds the
ExprToConnectorExpressionConverter in fe-core to convert Doris Expr trees
into ConnectorExpression trees at the SPI boundary.
Changes:
- ConnectorExpression: converted from final class to interface with
getExprType() and getChildren() methods
- ConnectorExprType: discriminator enum (COLUMN_REF, LITERAL, COMPARISON,
AND, OR, NOT, IN, BETWEEN, IS_NULL, LIKE, FUNCTION_CALL)
- 11 concrete expression types: ConnectorColumnRef, ConnectorLiteral,
ConnectorComparison, ConnectorAnd, ConnectorOr, ConnectorNot,
ConnectorIn, ConnectorBetween, ConnectorIsNull, ConnectorLike,
ConnectorFunctionCall
- ConnectorRange: range bound with low/high inclusive/exclusive endpoints
- ConnectorDomain: per-column domain (union of ranges + null handling)
for fast partition pruning
- ConnectorFilterConstraint: redesigned to carry full expression tree +
per-column domain map (replaces old flat conjuncts list)
- ExprToConnectorExpressionConverter: converts Doris Expr tree to
ConnectorExpression tree, plus Type to ConnectorType reverse mapping
None
- Test: No need to test - pure API/SPI type definitions and converter
with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorScanPlanProvider and PluginDrivenScanNode for connector SPI scan planning
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Plugin-driven catalogs need a generic scan node that delegates
scan planning to the connector SPI, rather than requiring connector-specific
ScanNode subclasses in fe-core.
This commit adds Step 20 of the Catalog SPI modularization:
**fe-connector-api enhancements:**
- ConnectorScanRangeType enum: FILE_SCAN, JDBC_SCAN, ES_SCAN, REMOTE_OLAP_SCAN, CUSTOM
- ConnectorDeleteFile: delete file descriptor for Iceberg MOR tables
- ConnectorScanRange: added getRangeType() (mandatory), getFileFormat(),
getFileSize(), getModificationTime(), getPartitionValues(), getDeleteFiles()
- ConnectorScanPlanProvider: added estimateScanRangeCount() default method
**fe-core additions:**
- PluginDrivenSplit: wraps ConnectorScanRange in FileSplit for the
FileQueryScanNode pipeline
- PluginDrivenScanNode: extends FileQueryScanNode, delegates scan planning
to ConnectorScanPlanProvider, uses FORMAT_JNI for BE execution
- ExprToConnectorExpressionConverter: added convertConjuncts() utility
- PhysicalPlanTranslator: dispatch to PluginDrivenScanNode for
PluginDrivenExternalTable instances
None
- Test: No need to test — SPI infrastructure, no actual connectors activate this path yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add ConnectorWriteOps & PluginDrivenTableSink (Step 21)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Implements Step 21 of the Catalog SPI modularization plan.
Adds the write operation SPI interfaces and the generic table sink for
plugin-driven external tables.
**SPI layer (fe-connector-api):**
- Enhanced ConnectorWriteOps with full write lifecycle: getWriteConfig,
beginInsert/finishInsert/abortInsert (with column list), beginDelete/
finishDelete/abortDelete, beginMerge/finishMerge/abortMerge
- Added ConnectorWriteType enum (FILE_WRITE, JDBC_WRITE, REMOTE_OLAP_WRITE,
CUSTOM)
- Added ConnectorWriteConfig value object with builder pattern, carrying
write type, file format, compression, location, partition columns, and
generic properties
- Added ConnectorDeleteHandle and ConnectorMergeHandle marker interfaces
**Engine layer (fe-core):**
- Created PluginDrivenTableSink extending BaseExternalTableDataSink
- Constructs TDataSink based on ConnectorWriteConfig.getWriteType():
FILE_WRITE -> THiveTableSink, JDBC_WRITE -> TJdbcTableSink
- Property-driven Thrift construction using well-known keys from
ConnectorWriteConfig.properties
None
- Test: No need to test - infrastructure interfaces with no runtime behavior change
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add generic Nereids plan nodes for plugin-driven INSERT pipeline
Issue Number: close #xxx
Problem Summary: PluginDrivenExternalCatalog-backed tables cannot participate
in the full Nereids INSERT pipeline because there are no generic plan node
classes for them. Each existing connector (Hive, Iceberg, JDBC, etc.) has its
own typed plan nodes, but the plugin-driven generic path has none.
This commit adds the complete Nereids INSERT pipeline for plugin-driven
catalogs:
1. **Plan nodes**: UnboundConnectorTableSink, LogicalConnectorTableSink,
PhysicalConnectorTableSink — generic nodes that work with any
PluginDrivenExternalCatalog without connector-specific plan classes.
2. **Optimizer rules**: BINDING_INSERT_CONNECTOR_TABLE (BindSink),
LogicalConnectorTableSinkToPhysicalConnectorTableSink (implementation),
ExpressionRewrite for LogicalConnectorTableSink.
3. **Insert executor**: PluginDrivenInsertExecutor delegates begin/commit/
abort to ConnectorWriteOps SPI. PluginDrivenInsertCommandContext provides
the context wrapper.
4. **Transaction manager**: PluginDrivenTransactionManager provides
lightweight transaction lifecycle bookkeeping; actual commit/rollback
is handled by ConnectorWriteOps in the insert executor.
5. **Wiring**: PlanType enum entries, RuleType entries, SinkVisitor methods,
RuleSet registration, UnboundTableSinkCreator dispatch,
PhysicalPlanTranslator visitor, InsertIntoTableCommand executor selection.
None
- Test: No need to test — infrastructure code, no connector plugins exercise
this path yet
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Activate CatalogFactory SPI path with build integration and connectivity testing
Issue Number: close #xxx
Problem Summary: The connector SPI plugins were built but not deployed to
the output directory, making them undiscoverable at runtime. Additionally,
PluginDrivenExternalCatalog lacked connectivity testing support.
This commit:
1. Adds connector module compilation to build.sh FE module list
2. Deploys connector plugin ZIPs to output/fe/plugins/connector/<name>/
(mirroring the filesystem plugin deployment pattern)
3. Creates ConnectorTestResult value object in fe-connector-api
4. Adds Connector.testConnection(session) default method
5. Overrides checkWhenCreating() in PluginDrivenExternalCatalog to delegate
connectivity testing to the connector SPI
6. Adds ConnectorFactory.getRegisteredTypes() for diagnostics
7. Enhances logging in CatalogFactory and Env for SPI path transparency
None
- Test: No need to test (build infrastructure + SPI wiring, no behavioral change)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 24: ES full migration to connector SPI
Issue Number: close #xxx
Problem Summary: Migrate the ES (Elasticsearch) connector scan path from
legacy fe-core EsScanNode to the plugin-driven connector SPI. This enables
the ES connector to run as an isolated plugin while producing the same
ES_HTTP_SCAN_NODE Thrift types expected by BE.
Key changes:
- Enhanced ConnectorScanPlanProvider with getScanRangeType(),
getScanNodeProperties(), and getScanNodeMapProperties() to support
non-file scan types (ES, JDBC) that need custom Thrift structures
- Created EsQueryDslBuilder: ports QueryBuilders from legacy Expr-based
filter conversion to ConnectorExpression-based ES Query DSL generation
- Created EsScanPlanProvider: generates per-shard scan ranges with ES
host routing, query DSL, auth info, and field context maps
- Created EsScanRange: ConnectorScanRange impl for ES shard routing
- Created PluginDrivenEsScanNode: extends ExternalScanNode to produce
TPlanNodeType.ES_HTTP_SCAN_NODE with TEsScanNode/TEsScanRange Thrift
- Wired EsConnector.getScanPlanProvider() to return real provider
- Added EsConnector.testConnection() for catalog connectivity testing
- Fixed BindRelation missing PLUGIN_EXTERNAL_TABLE case (critical:
without this fix, no plugin-driven table can be queried)
- PhysicalPlanTranslator dispatches to PluginDrivenEsScanNode when
getScanRangeType() == ES_SCAN
- Removed "es" case from CatalogFactory switch (SPI handles it)
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 25: JDBC full migration (scan only) to connector SPI
Issue Number: close #xxx
Problem Summary: Migrate JDBC connector scan path to the connector SPI
framework, enabling JDBC catalogs to run through the PluginDrivenScanNode
pipeline instead of the legacy JdbcScanNode code path.
Key changes:
- JdbcQueryBuilder: converts ConnectorExpression to SQL WHERE clauses with
DB-specific date formatting (Oracle to_date, Trino date literal, SQL Server
CONVERT) and LIMIT syntax (MySQL LIMIT, Oracle ROWNUM, SQL Server TOP N)
- JdbcScanRange: ConnectorScanRange with table_format_type="jdbc" and Builder
pattern for all BE-expected jdbc_params (url, user, password, driver, query_sql,
connection pool settings)
- JdbcScanPlanProvider: builds SQL query via JdbcQueryBuilder, returns 1 scan
range with all JDBC connection parameters
- JdbcColumnHandle: carries localName + remoteName for SELECT clause column
quoting via JdbcIdentifierQuoter
- JdbcConnectorMetadata.getColumnHandles(): returns column handle map for
PluginDrivenScanNode column selection
- JdbcDorisConnector: wires getScanPlanProvider() and testConnection()
- ConnectorScanRange.getTableFormatType(): default "plugin_driven", JDBC
overrides to "jdbc" so BE routes to JdbcJniReader
- ConnectorScanPlanProvider: 5-arg planScan() overload with limit parameter
for JDBC LIMIT pushdown
- PluginDrivenScanNode: uses scanRange.getTableFormatType() instead of
hardcoded constant; builds column handles via metadata.getColumnHandles();
passes limit to 5-arg planScan()
- CatalogFactory: removed "jdbc" case from fallback switch (all JDBC catalogs
now go through SPI path)
None
- Test: Manual test (compilation + checkstyle verified)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Step 26: MaxCompute full migration (scan only) to connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Migrates MaxCompute scan planning from fe-core to the
fe-connector-maxcompute plugin module. This removes the "max_compute" fallback
case from CatalogFactory, routing MaxCompute catalog creation exclusively
through the SPI connector path (PluginDrivenExternalCatalog).
Key changes:
- MaxComputeScanPlanProvider: creates ODPS TableBatchReadSession, generates
splits (byte_size and row_offset strategies), supports limit optimization
- MaxComputePredicateConverter: converts ConnectorExpression to ODPS Predicate
(CompoundPredicate, RawPredicate, UnaryPredicate) with datetime timezone
conversion
- MaxComputeScanRange: ConnectorScanRange carrying serialized session, split
params, and timeout config with tableFormatType="max_compute"
- MaxComputeColumnHandle: tracks partition vs data column distinction
- MCConnectorEndpoint: region-to-timezone mapping for datetime pushdown
- PluginDrivenScanNode: format-type dispatch in setScanParams() -
"max_compute" builds TMaxComputeFileDesc, others use generic jdbc_params map
- CatalogFactory: removed "max_compute" fallback case and
MaxComputeExternalCatalog import
- Added odps-sdk-table-api dependency for table read session API
None
- Test: No need to test - structural migration only, MaxCompute not available
in dev environment
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Migrate Trino connector scan planning to connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Trino connector scan planning was still handled by the legacy
TrinoConnectorScanNode in fe-core with direct Trino SPI type dependencies.
This commit migrates the full scan flow to the connector SPI plugin module
(fe-connector-trino), keeping all Trino SPI types inside the plugin boundary.
Key changes:
- TrinoScanPlanProvider: core scan planning (beginQuery → applyFilter/Limit/
Projection → getSplits → JSON-serialize → TrinoScanRange list → cleanupQuery)
- TrinoJsonSerializer: ObjectMapperProvider + HandleJsonModule + BlockJsonSerde
for serializing Trino SPI objects to JSON
- TrinoPredicateConverter: ConnectorExpression → Trino TupleDomain<ColumnHandle>
- TrinoScanRange: ConnectorScanRange carrying pre-serialized JSON properties
- TrinoColumnHandle: ConnectorColumnHandle for projection pushdown
- PluginDrivenScanNode: "trino_connector" dispatch filling TTrinoConnectorFileDesc
from properties map (no Trino types needed in fe-core)
- CatalogFactory: removed "trino-connector" legacy fallback case
- TrinoBootstrap: added getHandleResolver()/getTypeRegistry() accessors
- TrinoConnectorDorisMetadata: added getColumnHandles() method
- TrinoDorisConnector: replaced stub with real getScanPlanProvider() + testConnection()
None
- Test: No need to test - structural migration, scan logic ported from existing
TrinoConnectorScanNode with no behavioral changes
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Hive scan planning to connector plugin (Step 31)
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Adds scan planning capability to the Hive connector plugin
module (fe-connector-hive), enabling file-based scan range generation
through the ConnectorScanPlanProvider SPI. This is the scan-only phase of
the Hive full migration to the connector plugin architecture.
New files in fe-connector-hive:
- HiveScanPlanProvider: Core scan planning - partition resolution via HMS,
file listing via Hadoop FS, file splitting by configurable target size
- HiveColumnHandle: ConnectorColumnHandle wrapping column name/type/isPartKey
- HiveFileFormat: Enum mapping InputFormat/SerDe classes to format strings
- HiveScanRange: ConnectorScanRange with Builder for file split descriptors
- HiveTextProperties: Extracts text format props from HMS SerDe parameters
Enhanced existing files:
- HiveTableHandle: Rewritten with Builder pattern, now carries inputFormat,
serializationLib, location, partitionKeyNames, sdParameters, tableParameters,
and prunedPartitions for scan planning
- HiveConnectorMetadata: Added getColumnHandles(), applyFilter() with
partition pruning (equality + IN predicates on partition columns)
- HiveConnector: Added getScanPlanProvider() returning HiveScanPlanProvider
- HmsTableInfo: Added sdParameters field for SerDe parameters
- ThriftHmsClient: Populates sdParameters from SerDeInfo
- PluginDrivenScanNode: Major enhancement with property-driven overrides for
getFileFormatType, getPathPartitionKeys, getLocationProperties, getFileAttributes,
plus hive/transactional_hive dispatch in setScanParams
Scope: Non-ACID Hive tables (Parquet, ORC, Text). ACID tables, file listing
cache, and table sampling are deferred to future steps.
None
- Test: No need to test (scan planning code path not yet wired to production
query flow; requires HMS + Hadoop cluster for integration testing)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Hudi scan planning to connector plugin (Step 32)
Issue Number: close #xxx
Problem Summary: Adds scan planning capability to the fe-connector-hudi plugin
module, following the same pattern established in Step 31 (Hive scan). This
enables the Hudi connector plugin to generate scan ranges that BE can consume,
supporting both COW (native Parquet/ORC reader) and MOR (JNI reader with delta
log merging) table types.
New files:
- HudiScanPlanProvider: Core scan planning — builds MetaClient, resolves
partitions via HoodieTableMetadata API, generates COW splits (base files
only) and MOR splits (base + delta logs with dynamic native downgrade)
- HudiScanRange: ConnectorScanRange implementation with Builder pattern,
carrying both native reader fields and JNI metadata (instant_time, serde,
delta_logs, column_names/types)
- HudiColumnHandle: Column handle with name, typeName, isPartitionKey
Modified files:
- HudiTableHandle: Rewritten with Builder pattern, added scan-related fields
(inputFormat, serdeLib, partitionKeyNames, tableParameters, prunedPaths)
- HudiConnectorMetadata: Enhanced getTableHandle with scan fields from HMS,
added getColumnHandles() and applyFilter() for partition pruning
- HudiConnector: Added getScanPlanProvider()
- PluginDrivenScanNode: Added "hudi" case in setScanParams dispatch with
setHudiParams() method creating THudiFileDesc for both native/JNI paths
None
- Test: No need to test — plugin scan planning mirrors existing HudiScanNode
logic; will be validated when end-to-end regression tests are enabled
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[feature](fe) Add Paimon scan planning to connector plugin (Step 33)
Problem Summary: Migrate Paimon scan planning logic from fe-core PaimonScanNode
to the fe-connector-paimon plugin module, following the same pattern established
by Hive (Step 31) and Hudi (Step 32) scan migrations.
This step adds:
- PaimonColumnHandle: Column handle with name and field index in RowType
- PaimonPredicateConverter: Converts ConnectorExpression to Paimon Predicate
(PredicateBuilder-based, supports AND/OR/comparison/IN/IS NULL/LIKE prefix)
- PaimonScanRange: Builder-pattern scan range supporting JNI, native, and
COUNT pushdown paths with dual serialization
- PaimonScanPlanProvider: Core scan planning using Paimon SDK (ReadBuilder →
TableScan → Split), converting to ConnectorScanRange with JNI/native dispatch
- Modified PaimonTableHandle to carry transient Table reference
- Modified PaimonConnectorMetadata to store Table in handle and add getColumnHandles()
- Modified PaimonConnector to add getScanPlanProvider()
- Modified PluginDrivenScanNode to add paimon dispatch (setPaimonParams),
getSerializedTable() override, and scan-level paimon params (predicate + options)
None
- Test: No need to test (compile-only migration, runtime wiring not yet active)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove old internal ES table code and clean up references
Issue Number: close #xxx
Problem Summary: The old internal Elasticsearch table implementation (EsScanNode,
EsExternalCatalog, EsExternalTable, etc.) has been superseded by the ES Catalog
connector plugin. This commit removes the legacy ES code from fe-core and cleans
up all references throughout the codebase.
Changes:
- Delete 21 old ES datasource files from datasource/es/
- Delete 3 nereids plan classes (LogicalEsScan, PhysicalEsScan, LogicalEsScanToPhysicalEsScan)
- Delete 6 old ES test files from external/elasticsearch/
- Clean up ES references in 13+ files (Env.java, ExternalCatalog.java, ESCatalogAction.java,
PhysicalPlanTranslator.java, BindRelation.java, RuleSet.java, RelationVisitor.java,
StatsCalculator.java, CostModel.java, ChildOutputPropertyDeriver.java,
TopnFilterPushDownVisitor.java, GsonUtils.java, TableIf.java, etc.)
- Preserve EsTable.java and EsResource.java for persistence compatibility
- Add ES stub in SHOW CREATE TABLE for deprecation notice
None
- Test: Manual test - fe-core compiles successfully, ES connector tests pass
- Behavior changed: Yes - old internal ES table queries will no longer work;
users should use ES Catalog instead
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Add metadata methods to JDBC connector for migration parity
Issue Number: close #xxx
Problem Summary: During JDBC connector modular migration from fe-core to
fe-connector-jdbc, several metadata methods were missing in the new connector
client and SPI interfaces. This commit adds the missing methods to achieve
parity with the old JdbcClient.
None
- Test: No need to test - pure method additions with no behavior change,
existing integration paths still use old JdbcClient
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Implement JDBC write path through connector plugin pipeline
Issue Number: close #xxx
Problem Summary: The JDBC INSERT INTO write path was not wired through the
new connector plugin pipeline. This commit implements the complete JDBC write
path so that INSERT INTO on PluginDriven JDBC tables produces the correct
TJdbcTableSink thrift structure, matching the old JdbcTableSink behavior.
Changes:
- JdbcConnectorMetadata now implements ConnectorWriteOps with getWriteConfig(),
beginInsert(), finishInsert() — builds INSERT SQL via JdbcIdentifierQuoter
and populates all JDBC connection/pool properties
- JdbcIdentifierQuoter gains buildInsertSql() for parameterized INSERT SQL
- PluginDrivenTableSink.bindJdbcWriteSink() now populates ALL TJdbcTable
fields: catalog_id, driver_checksum, table_name, resource_name, and all
connection pool settings; also sets TOdbcTableType on TJdbcTableSink
- PluginDrivenInsertExecutor resolves write type from connector metadata and
returns TransactionType.JDBC for JDBC writes (was hardcoded to HMS)
- PhysicalPlanTranslator passes actual column list to getWriteConfig()
(was passing empty list)
- ConnectorSessionBuilder now propagates enable_odbc_transcation session var
- Explain output enhanced for JDBC write sinks (shows table type, SQL, txn)
None
- Test: No need to test - infrastructure wiring, no end-to-end JDBC catalog
available in unit test environment; will be tested with regression tests
after full migration
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Port JDBC function pushdown and ClickHouse FINAL to connector module
Issue Number: close #xxx
Problem Summary:
The JDBC connector module (fe-connector-jdbc) lacked function pushdown support
and ClickHouse FINAL query support. These features existed only in the old
fe-core JdbcScanNode code path. This commit ports them to the connector SPI
pipeline so PluginDrivenScanNode-based JDBC scans have feature parity.
Key changes:
- New JdbcFunctionPushdownConfig: per-DB function whitelist/blacklist/replacement
rules with time arithmetic rewriting and JSON-configurable overrides
- Extended JdbcQueryBuilder: ConnectorFunctionCall/Like/Between to SQL conversion,
conjunct pushdown guards (Oracle NULL, cast, function-containing expressions),
ClickHouse SETTINGS final=1 support
- Updated JdbcScanPlanProvider to create pushdown config and pass session props
- Plumbed 4 session variables through ConnectorSessionBuilder:
enable_ext_func_pred_pushdown, jdbc_clickhouse_query_final,
enable_jdbc_oracle_null_predicate_push_down, enable_jdbc_cast_predicate_push_down
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Route JDBC TVF and CallExecuteStmt through connector SPI
Issue Number: close #xxx
Problem Summary:
JdbcQueryTableValueFunction and CallExecuteStmtFunc only supported the old
JdbcExternalCatalog code path. Newly created JDBC catalogs use
PluginDrivenExternalCatalog via the SPI connector, so these features were
broken for new catalogs.
Key changes:
- New PassthroughQueryTableHandle in connector API for raw query TVF scans
- JdbcQueryTableValueFunction: dual path — uses ConnectorMetadata for schema
discovery and PluginDrivenScanNode for scan planning when catalog is
PluginDrivenExternalCatalog
- CallExecuteStmtFunc: routes to connector metadata.executeStmt() for
PluginDrivenExternalCatalog, falls back to old JdbcExternalCatalog path
- QueryTableValueFunction factory: detects PluginDrivenExternalCatalog
- JdbcScanPlanProvider: handles PassthroughQueryTableHandle for TVF queries
None
- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[improvement](fe) Migrate JDBC identifier mapping to connector module
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
The JDBC identifier mapping logic (lower_case_meta_names, meta_names_mapping,
lower_case_table_names) was tightly coupled to fe-core via JdbcIdentifierMapping
which depends on Jackson ObjectMapper. This commit migrates the functionality
to the connector-jdbc module as part of the SPI modularization effort.
Key changes:
- Add ConnectorIdentifierOps SPI interface in connector-api with 3 name mapping
methods (fromRemoteDatabaseName, fromRemoteTableName, fromRemoteColumnName)
- ConnectorMetadata now extends ConnectorIdentifierOps
- Create JdbcIdentifierMapper in connector-jdbc: pure Java reimplementation
with regex-based JSON parsing (no Jackson dependency), full validation with
case-conflict detection
- Wire identifier mapping through getColumnHandles() and getWriteConfig() so
column names are properly mapped in both read and write paths
- Add lower_case_table_names from GlobalVariable to ConnectorSessionBuilder
- PluginDrivenExternalCatalog overrides fromRemoteDatabaseName/TableName to
delegate to connector metadata
- PluginDrivenExternalTable.initSchema() applies fromRemoteColumnName mapping
None
- Test: Manual test / Compilation verified
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Remove old JDBC connector code after migration to plugin-driven architecture
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: After migrating JDBC connector to fe-connector-jdbc module with
plugin-driven architecture (PluginDrivenExternalCatalog/Database/Table), the old
JDBC code in fe-core/datasource/jdbc/ is dead code. This commit:
1. Adds Gson compatible subtype adapters so persisted metadata with old class names
(JdbcExternalCatalog/Database/Table) deserializes correctly as PluginDriven* classes
2. Deletes 21 old JDBC files: JdbcExternalCatalog, JdbcExternalDatabase,
JdbcExternalTable, JdbcNameUtil, JdbcSchemaCacheValue, JdbcTableSink, JdbcScanNode,
JdbcSplit, JdbcFunctionPushDownRule, IdentifierMapping, JdbcIdentifierMapping,
UnboundJdbcTableSink, LogicalJdbcTableSink, PhysicalJdbcTableSink,
JdbcInsertExecutor, JdbcInsertCommandContext,
LogicalJdbcScanToPhysicalJdbcScan, LogicalJdbcTableSinkToPhysicalJdbcTableSink,
LogicalJdbcScan, PhysicalJdbcScan
3. Cleans up all references in 23 files across nereids visitors, derivers, rules,
plan translator, expression rewrite, and external function rules
4. Inlines function pushdown constants from deleted JdbcFunctionPushDownRule into
ExternalFunctionRules
JdbcClient hierarchy (16 files) is kept for CDC binlog functionality.
None
- Test: No need to test - pure code deletion of dead code after migration
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix connector plugin packaging and ES scan node bugs
Issue Number: close #xxx
Problem Summary:
Three issues causing CI regression test failures (build 926124):
1. **JDBC/Hive/MaxCompute plugin JARs not discovered at runtime**
The assembly descriptors for these connectors used `includeBaseDirectory=true`,
creating a nested subdirectory inside the zip (e.g., `fe-connector-jdbc/lib/*.jar`).
When build.sh unzips into `plugins/connector/jdbc/`, the actual path becomes
`plugins/connector/jdbc/fe-connector-jdbc/lib/*.jar`, but DirectoryPluginRuntimeManager
expects JARs at `plugins/connector/jdbc/*.jar` and `plugins/connector/jdbc/lib/*.jar`.
Fix: Set `includeBaseDirectory=false` and place plugin JAR at root + deps in lib/,
matching the working ES/iceberg/paimon assembly layout.
2. **ES scan node URL parsing fails for http:// prefixed hosts**
PluginDrivenEsScanNode.convertToThrift() split host strings by ":" naively,
so `http://172.16.0.98:29200` was parsed as host="http", port="//172.16.0.98"
causing NumberFormatException. Fix: Strip scheme prefix before parsing host:port.
3. **ES JSONB type not recognized by ConnectorColumnConverter**
ES connector returns JSONB type but ScalarType.createType() only handles "JSON".
ConnectorColumnConverter now maps JSONB to JSON explicitly.
None
- Test: Manual verification of zip layout; CI rerun pending
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Resolve JDBC driver_url via ConnectorContext environment
Problem Summary: JDBC connector plugin fails to load driver classes because
plain driver filenames (e.g., "mysql-connector-j-8.4.0.jar") are not resolved
to absolute file:// URLs using Config.jdbc_drivers_dir.
The fix adds a generic getEnvironment() method to ConnectorContext SPI,
which fe-core populates with system configs (jdbc_drivers_dir, doris_home).
The JDBC connector uses this environment to resolve driver URLs, keeping
fe-core free of JDBC-specific logic.
None
- Test: Manual test - verified driver URL resolution logic
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Implement toThrift() for PluginDrivenExternalTable via connector SPI
Problem Summary: PluginDrivenExternalTable.toThrift() inherited the base
ExternalTable implementation which returns null, causing NPE during query
fragment serialization when BE needs TTableDescriptor.
The fix adds:
1. ConnectorTableOps.getTableDescriptorProperties() SPI method that lets
connectors declare the BE table descriptor type and properties
2. JdbcConnectorMetadata returns JDBC_TABLE type with all connection/pool
properties needed by TJdbcTable
3. EsConnectorMetadata returns ES_TABLE type
4. PluginDrivenExternalTable.toThrift() reads these properties and
constructs the appropriate typed TTableDescriptor (TJdbcTable/TEsTable)
None
- Test: Manual test — JDBC catalog query no longer hits null TTableDescriptor
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[refactor](fe) Connector directly builds TTableDescriptor via fe-thrift
Issue Number: close #xxx
Problem Summary: Previously PluginDrivenExternalTable.toThrift() used a Map-based
getTableDescriptorProperties() intermediary to build TTableDescriptor, which required
fe-core to contain connector-specific logic for constructing TJdbcTable/TEsTable.
This refactoring lets each connector module directly depend on fe-thrift (provided scope)
and build its own TTableDescriptor in buildTableDescriptor(), making PluginDrivenExternalTable
a pure delegator with zero connector-specific code.
Key changes:
- ConnectorTableOps: replaced getTableDescriptorProperties() with buildTableDescriptor()
returning TTableDescriptor directly
- JdbcConnectorMetadata: builds TJdbcTable + TTableDescriptor directly
- EsConnectorMetadata: builds TEsTable + TTableDescriptor directly
- PluginDrivenExternalTable.toThrift(): simplified to pure delegation
- Added fe-thrift as provided dependency to fe-connector-api, fe-connector-jdbc,
fe-connector-es (Maven provided scope is NOT transitive)
- All 4 plugin assembly ZIPs exclude fe-thrift and libthrift
None
- Test: Manual test (compilation verified, runtime pending BIT/BOOLEAN display fix)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix SPI column attributes, type mapping, EXPLAIN format, and DB/schema mapping
Issue Number: close #xxx
Problem Summary: Fix 13 CI test failures in the External Regression pipeline
caused by the connector migration. This batch addresses 4 root cause categories:
1. **isKey lost (5 tests)**: ConnectorColumn had no isKey field; convertColumn()
hardcoded false. Added isKey to ConnectorColumn, propagated through
ConnectorColumnConverter, populated from JDBC getPrimaryKeys().
2. **Type mapping (3 tests)**: HLL/BITMAP/QUANTILE_STATE types were not handled
in JdbcMySQLConnectorClient.mapSignedType(), falling to UNSUPPORTED. Added
the missing cases for Doris-to-Doris JDBC catalog support.
3. **DB/schema mapping (3 tests)**: ClickHouse connector ignored databaseterm
URL parameter; OceanBase only delegated type mapping (not metadata methods);
PostgreSQL missing bit/varbit/hstore type mappings.
4. **EXPLAIN format (2 tests)**: PluginDrivenScanNode did not output QUERY: line
in EXPLAIN. Added getNodeExplainString() override that shows pushed-down SQL
from JdbcScanPlanProvider.getScanNodeProperties().
None
- Test: Regression test (pending CI run)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Wire write path, statistics, row count, and test expectations through connector SPI
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Fix 5 regression test failures in the connector SPI migration (Batch 2):
1. InsertUtils write sink: Add UnboundConnectorTableSink handling in
getTargetTableQualified() for write transaction support.
2. Statistics collection: Override createAnalysisTask() in
PluginDrivenExternalTable to return ExternalAnalysisTask.
3. Row count estimation: Add fetchRowCount() to PluginDrivenExternalTable
delegating to ConnectorMetadata.getTableStatistics(). Add
JdbcConnectorMetadata.getTableStatistics() and MySQL-specific
getRowCount() querying INFORMATION_SCHEMA.
4. CallExecuteStmt test: Update expected exception message from
"Only support JDBC catalog" to "executeStmt not supported" since
HMS catalogs now route through PluginDrivenExternalCatalog.
5. Oracle CHAR padding: Update test_oracle_jdbc_catalog.out to match
current OracleTypeHandler behavior which trims CHAR trailing spaces.
This is a pre-existing behavior change from master commit 9770ecf8a59
(JdbcTypeHandler refactor) where the .out file was never updated.
None
- Test: Regression test expectations updated
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ES URL parsing, predicate pushdown, date compat, and TVF classloader
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Four bugs introduced during the connector SPI migration:
1. **ES double-port URL** (test_es_query_no_http_url): EsNodeInfo seed constructor
produced "host:port:80" for inputs like "host:9200" without a scheme prefix.
Fixed by correctly parsing host and port when split(":") yields 2 parts.
2. **ES LIKE not pushed down** (test_es_query): EsQueryDslBuilder.likeToDsl()
checked column2typeMap BEFORE resolving keyword sub-fields from fieldsContext.
Text fields with .keyword sub-fields were rejected. Fixed by checking
fieldsContext.containsKey() first, matching the old QueryBuilders behavior.
3. **ES date equality returns empty** (test_es_query_nereids): compatDefaultDate()
used ZoneOffset.UTC, but the old Joda-Time code used the system default timezone.
When JVM runs in CST (UTC+8), "2022-08-08 08:00:00" should map to midnight UTC,
not 8am UTC. Fixed by using ZoneId.systemDefault().
4. **TVF cross-catalog ClassCastException** (test_query_tvf_cross_catalog,
test_query_tvf_auth): PassthroughQueryTableHandle (in org.apache.doris.connector.api)
was loaded by both app classloader and ChildFirstClassLoader, causing instanceof
to fail. Fixed by adding "org.apache.doris.connector.api." to the parent-first
package list in ChildFirstClassLoader.
None
- Test: Regression test (CI pipeline)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](be) Fix CTAS ArrayStoreException for MySQL YEAR column
Issue Number: close #xxx
Problem Summary: MySQL YEAR columns mapped to Doris SMALLINT caused
ArrayStoreException: java.sql.Date in JdbcJniScanner.getNext().
Root cause: MySQLTypeHandler.getColumnValue() used untyped
rs.getObject(columnIndex) for TINYINT/SMALLINT, which returns
java.sql.Date for YEAR columns when MySQL Connector/J default
yearIsDateType=true is in effect. The SMALLINT converter then could
not handle java.sql.Date (falls through to return input as-is),
and storing java.sql.Date into Short[] caused ArrayStoreException.
Fix: Use typed rs.getObject(columnIndex, Integer.class) for TINYINT
and SMALLINT to force the JDBC driver to convert to Integer regardless
of the underlying MySQL column type.
None
- Test: Regression test (test_mysql_all_types_ctas)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](build) Change mvn package to install to fix MDEP-187 reactor dependency issue
Problem Summary: The fe-core module copy-dependencies goal fails with
MDEP-187 error when new reactor modules (fe-connector-api,
fe-connector-spi) have not been installed to the local Maven
repository. This is a known Maven limitation where copy-dependencies
cannot resolve reactor artifacts during the same mvn package run.
Fix: Change build.sh from mvn package to mvn install so all reactor
artifacts are installed to the local repo before copy-dependencies runs.
None
- Test: Manual test (build.sh --fe)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](build) Add Apache license headers to SPI service files
Problem Summary: ConnectorProvider service files in META-INF/services/
were missing Apache license headers, causing the license checker to fail.
None
- Test: No need to test (license headers only)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](es) Fix ES date predicate pushdown timezone conversion
Problem Summary: ES date equality queries returned empty results because
compatDefaultDate() could not parse ISO-formatted datetime strings produced
by extractLiteralValue(). The extractLiteralValue() method formats
LocalDateTime as "2022-08-08T08:00:00" (ISO with T separator), but
compatDefaultDate() only tried to parse "yyyy-MM-dd HH:mm:ss" (space
separator). The parse failure caused the value to be sent without timezone
conversion, so ES interpreted it as UTC instead of the local timezone.
Fix: Add ISO_LOCAL_DATE_TIME parsing as the first attempt in
compatDefaultDate(), before falling back to space-separated format and
date-only format.
None
- Test: Regression test (test_es_query_nereids sql63)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix JDBC connector type mapping, auth, and EXPLAIN regressions
Issue Number: close #62183
Problem Summary: Fix multiple regression failures in the connector-based
JDBC/ES external catalog migration. This commit addresses 8 root causes
covering 20+ test failures from CI External Regression pipeline build 926236.
Fixes include:
- detectDoris() initialization ordering: moved from constructor to
postInitialize() hook so data source is live before detection runs.
Added dorisTypeToConnectorType() for Doris-to-Doris type mapping
including HLL, BITMAP, QUANTILE_STATE types.
- Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match
"TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc.
- Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0
for integer branch, and match old boundary thresholds.
- ClickHouse DB listing: use SHOW DATABASES for full listing;
fix databaseTermIsCatalog inversion for old drivers.
- DATETIMEV2 precision: ConnectorColumnConverter reads precision
(not scale) for datetime fractional seconds, matching connector
encoding convention.
- EXPLAIN cache: invalidate scanNodeProperties after convertPredicate()
so pushed conjuncts are reflected in EXPLAIN output.
- ExecutionAuthenticator: call initPreExecutionAuthenticator() in
PluginDrivenExternalCatalog.initLocalObjectsImpl().
- PassthroughQueryTableHandle: add instanceof guards in
JdbcConnectorMetadata to prevent ClassCastException for TVF.
- isKey: hardcode true for all columns matching legacy behavior.
None
- Test: Regression test (CI External Regression pipeline)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ES LIKE pushdown for Nereids planner
Problem Summary: Nereids planner translates Like/Regexp expressions to
FunctionCallExpr (via ExpressionTranslator.visitScalarFunction) rather
than LikePredicate. ExprToConnectorExpressionConverter only checked for
instanceof LikePredicate, so the LIKE predicate was converted to a
ConnectorFunctionCall instead of ConnectorLike. This caused ES query DSL
builder to not recognize it as a LIKE and fall back to match_all,
returning all documents instead of only those matching the wildcard.
Fix: Detect FunctionCallExpr with function name "like" or "regexp" in
ExprToConnectorExpressionConverter and convert to ConnectorLike.
None
- Test: Regression test (test_es_query sql_5_29/sql_6_29)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Add JDBC URL normalization for connector migration
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The connector migration moved JDBC catalog logic from
fe-core to fe-connector-jdbc, but the URL normalization step was lost.
The old path called JdbcResource.handleJdbcUrl() which added critical
JDBC driver parameters. Without these, OceanBase tests fail with
timezone shifts (8h offset), lost fractional seconds, BOOLEAN display
as true/false instead of 1/0, and DOUBLE precision loss.
The fix adds JdbcUrlNormalizer in fe-connector-jdbc that replicates the
same parameter injection logic:
- MySQL/OceanBase: yearIsDateType=false, tinyInt1isBit=false,
useUnicode=true, characterEncoding=utf-8, rewriteBatchedStatements=true
- OceanBase additionally: useCursorFetch=true
- PostgreSQL: reWriteBatchedInserts=true
- SQL Server: useBulkCopyForBatchInsert=true
The normalization is applied once in JdbcDorisConnector constructor,
so all downstream consumers (client, metadata, scan plan provider)
use the normalized URL.
None
- Test: Regression test (test_oceanbase_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Stop filtering ClickHouse system database from catalog listing
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The new JdbcClickHouseConnectorClient incorrectly filtered
the ClickHouse "system" database in getFilterInternalDatabases(). The old
JdbcClickHouseClient never overrode this method, so it used the base class
default which only filters information_schema, performance_schema, and mysql.
The system database contains useful tables (query_log, processes, etc.) and
should remain accessible to users.
None
- Test: Regression test (test_clickhouse_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
[fix](fe) Fix ClickHouse JDBC catalog regression issues in plugin-driven architecture
Issue Number: close #xxx
Problem Summary: Multiple regression issues found after migr…
Contributor
Author
|
run buildall |
2 similar comments
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I found 2 blocking issues.
-
Resource-backed JDBC/ES catalogs cannot be reinitialized after restart.
PluginDrivenExternalCatalog.initLocalObjectsImpl()reconstructs the connector only fromcatalogProperty["type"], but legacy resource-backed catalogs derive their type from the serialized resource and may not persist atypeproperty. After checkpoint/restart this path passes an empty catalog type tocreateConnector(), so FE can no longer initialize those catalogs. -
Replay/startup now depends on ES/JDBC plugin load success.
CatalogFactoryremoved thees/jdbcbuilt-in fallback, butinitConnectorPluginManager()explicitly allows partial plugin-load failure and only logs warnings. A missing or broken connector zip now makescreateFromLog()throwUnknown catalog typefor existing ES/JDBC catalogs during replay.
Critical checkpoint conclusions:
- Goal of current task: Partially accomplished. The PR adds the FE connector API/SPI skeleton, but the migration path for persisted JDBC/ES catalogs is not safe yet.
- Is the change small/clear/focused: No. The PR also flips live ES/JDBC catalogs onto the new plugin path, and that is where the compatibility regressions are introduced.
- Concurrency: No blocking concurrency issue found in the reviewed paths.
- Lifecycle/static initialization: No blocking SIOF issue found; the main lifecycle problem is connector re-creation compatibility after restart.
- Configuration items:
connector_plugin_rootis added and startup wiring is present; no blocking dynamic-config issue found. - Incompatible changes / rolling upgrade: Blocking. Persisted ES/JDBC catalog compatibility is incomplete for both resource-backed catalogs and missing-plugin scenarios.
- Parallel code paths: Not fully covered. The old ES/JDBC fallback path was removed without an equivalent compatibility path when plugin loading fails.
- Special conditional checks: No additional blocking issue found.
- Test coverage: Insufficient for the migration risk. I did not see restart/image-replay coverage for old ES/JDBC catalogs or coverage for missing plugin artifacts.
- Observability: Startup logging is acceptable, but it does not mitigate the replay break.
- Transaction and persistence: Blocking. FE image/replay compatibility for external catalog metadata is not preserved end to end.
- Data writes/modifications: Not applicable in the reviewed paths.
- New FE/BE variable passing: Not applicable.
- Performance: No blocking issue found in the reviewed paths.
- Other issues: None beyond the blockers above in the reviewed areas.
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Findings
- The new Hive SPI connector drops the existing HMS Kerberos/doAs authentication path, so secured metastores will fail once this implementation is exercised.
- The new Hive SPI connector ignores the configured HMS type-mapping options and always falls back to default mappings.
Critical Checkpoints
- Goal of the task: Partially met. The PR wires the FE connector skeleton and SPI path, but the new Hive connector implementation does not yet preserve existing HMS compatibility behavior.
- Modification size/focus: No. This is a broad change spanning API, build, plugin packaging, and multiple connector implementations, which increases migration risk.
- Concurrency: Reviewed. The lazy-init paths use
volatileplussynchronized, and I did not find a blocking lock-order or thread-safety issue. - Lifecycle/static initialization: Reviewed. Connector/plugin lifecycle is mostly clear; no static initialization issue found.
- Configuration items:
connector_plugin_rootis added as a startup config; non-dynamic behavior is acceptable here. - Incompatible changes / parallel paths: Not preserved for Hive. The new SPI path diverges from the built-in HMS path for Kerberos auth and type mapping.
- Special conditional checks: No blocking issue found.
- Test coverage: Insufficient for the compatibility-sensitive Hive paths. ES unit tests exist, but there is no coverage for Hive Kerberos parity or mapping-option parity.
- Observability: Basic logs are present; no blocking observability issue found.
- Transaction / persistence / data-write correctness: Not applicable to the reviewed changes.
- FE-BE variable passing: No blocking issue found in the reviewed paths.
- Performance: No blocking performance issue found.
- Other issues: None beyond the findings above.
Because of the two compatibility regressions above, I can't approve this PR in its current state.
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: close #xxx Problem Summary: When initLocalObjectsImpl() re-creates the connector (e.g., after ALTER CATALOG or first makeSureInitialized()), the old connector injected by CatalogFactory during checkWhenCreating() was silently replaced without closing. This leaked the HikariCP connection pool and classloader from the old connector. Fix: Save the old connector reference before assigning the new one, then close the old connector after successful replacement. ### Release note None ### Check List (For Author) - Test: Unit Test (existing PluginDrivenExternalCatalogConcurrencyTest covers lifecycle) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### What problem does this PR solve? Issue Number: close #xxx Problem Summary: JdbcQueryBuilder.literalToSql() unconditionally rendered Boolean values as "1"/"0", which fails on PostgreSQL and Trino where boolean columns have strict typing and WHERE "flag" = 1 produces a type mismatch error. The old code (ExprToSqlVisitor.visitBoolLiteral) always used TRUE/FALSE keywords. Fix: Add formatBooleanLiteral() with database-specific rendering following the existing date literal pattern. Oracle/SQL Server/DB2 use 1/0 (integer boolean columns), all others use TRUE/FALSE keywords. Added 6 unit tests. ### Release note None ### Check List (For Author) - Test: Unit Test (6 new boolean literal tests across MySQL/PostgreSQL/Trino/Oracle/SQL Server/OceanBase Oracle) - Behavior changed: No (restores old behavior) - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ibility ### What problem does this PR solve? Issue Number: close #xxx Problem Summary: The SPI runtime path resolveDriverUrl() only checked DORIS_HOME/plugins/jdbc_drivers for bare driver filenames. Users who have driver JARs in the old default directory DORIS_HOME/jdbc_drivers (prior to the directory change) would get "driver not found" errors at runtime, even though creation-time validation (which delegates to JdbcResource.getFullDriverUrl) succeeded. Fix: When jdbc_drivers_dir equals the default (DORIS_HOME/plugins/jdbc_drivers) and the file is not found there, fall back to checking DORIS_HOME/jdbc_drivers, matching the old JdbcResource.checkAndReturnDefaultDriverUrl() behavior. ### Release note None ### Check List (For Author) - Test: Manual test (path resolution logic) - Behavior changed: No (restores old behavior) - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### What problem does this PR solve? Issue Number: close #xxx Problem Summary: The BE connectivity test sent SELECT 1 for all database types except Oracle. DB2 requires SELECT 1 FROM SYSIBM.SYSDUMMY1 and SAP HANA requires SELECT 1 FROM DUMMY. This caused DB2 catalog creation to fail with SQLCODE=-104 (syntax error). ### Release note None ### Check List (For Author) - Test: Regression test (doc/external/jdbc/ibmdb2.md.groovy) - Behavior changed: No (restores old behavior) - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
|
OpenCode automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
FE Regression Coverage ReportIncrement line coverage |
morningman
changed the title
Contributor
Author
|
/review |
suxiaogang223
approved these changes
Apr 18, 2026
morrySnow
approved these changes
Apr 18, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
OpenCode automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Author
|
/review |
Contributor
|
skip buildall |
CalvinKirs
approved these changes
Apr 18, 2026
Contributor
|
OpenCode automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
|
OpenCode automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
hubgeter
approved these changes
Apr 18, 2026
morningman
added a commit
that referenced
this pull request
Apr 22, 2026
…inating PluginDrivenEsScanNode (#62602) Related PR: #62183 Problem Summary: Before this change, ES catalog queries went through a dedicated `PluginDrivenEsScanNode` → `ES_HTTP_SCAN_NODE` → `EsScanner` code path, while JDBC catalogs used `PluginDrivenScanNode` → `FILE_SCAN_NODE` → `FileScanner`. The two paths had completely separate plan-generation logic, Thrift structures, and BE execution flows, making the connector SPI harder to maintain and extend. This PR routes ES catalog scans through the same `FILE_SCAN_NODE` path that JDBC already uses, achieving a single unified scan-node implementation for all plugin-driven connectors. The key changes are: **Thrift layer** - Add `FORMAT_ES_HTTP = 19` to `TFileFormatType` - Add `es_params` (per-shard map) to `TTableFormatFileDesc` - Add `es_properties`, `es_docvalue_context`, `es_fields_context` to `TFileScanRangeParams` (shared across shards) **BE — EsHttpReader (new, native C++)** - New `GenericReader` subclass in `be/src/format/table/es_http_reader.{h,cpp}` - Wraps existing `ESScanReader` (HTTP scroll) and `ScrollParser` (JSON→columns) - No JVM overhead — pure C++ HTTP reader, unlike JniReader used by some formats - Registered in `FileScanner::_get_next_reader()` for `FORMAT_ES_HTTP` - Implements `get_columns()` and `fill_all_columns() = true` **FE — Connector SPI extension** - Add `ScanNodePropertiesResult` class: typed wrapper carrying both scan properties and a `notPushedConjunctIndices` set with explicit `hasConjunctTracking` flag (distinguishes "no tracking" from "all pushed") - Add `getScanNodePropertiesResult()` to `ConnectorScanPlanProvider` **FE — ES connector adaptation** - `EsScanRange`: changed to `FILE_SCAN` range type, added `getPath()`, `getTableFormatType()`, `getFileFormat()`, BE-compatible property keys - `EsScanPlanProvider`: builds scan node properties with query_dsl, doc_values_mode, auth, docvalue/fields context serialization - `EsConnectorMetadata`: added `getColumnHandles()` using `NamedColumnHandle` **FE — PluginDrivenScanNode enhancement** - Added ES format mapping (`es_http` → `FORMAT_ES_HTTP`) - Added `setEsParams()` / `setEsScanLevelParams()` for ES Thrift fields - Added conjunct pruning via `ScanNodePropertiesResult` (removes pushed-down filters including `esquery()` fake function) - Added ES-specific EXPLAIN output **FE — Cleanup** - Deleted `PluginDrivenEsScanNode.java` (390 lines) - Simplified `PhysicalPlanTranslator`: removed ES-specific branch, all plugin-driven connectors now go through `PluginDrivenScanNode` ### Release note ES catalog scans now use the unified FILE_SCAN execution path shared with JDBC and other plugin-driven connectors. This is an internal architectural change — query behavior and results are unchanged. The `esquery()` function and doc-value optimization continue to work as before. ### Check List (For Author) - Test: Regression test — all 7 ES test suites pass - test_es_query - test_es_query_nereids - test_es_flatten_type - test_es_keyword_array_type - test_es_query_no_http_url - test_es_query_predicate_correctness - test_es_catalog_http_open_api - Behavior changed: No - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
The existing external catalog implementation in Doris has type-specific classes
scattered throughout fe-core (JdbcExternalCatalog, EsExternalCatalog,
JdbcExternalTable, EsExternalTable, JdbcScanNode, PhysicalJdbcTableSink, etc.),
making it difficult to add new connector types and leading to duplicated logic
across catalog implementations.
This PR introduces a Service Provider Interface (SPI) framework for external
connectors and migrates JDBC and Elasticsearch catalogs to the new plugin-driven
architecture. The key changes are:
New
fe-connectormodule hierarchy:fe-connector-api: Core interfaces (Connector, ConnectorMetadata,ConnectorCapability, ConnectorPushdownOps, ConnectorWriteOps,
ConnectorSession, pushdown expression tree, handle types)
fe-connector-spi: Service provider loading (ConnectorProvider,ConnectorContext, CatalogFactory)
fe-connector-jdbc: Full JDBC connector implementation migrated from fe-corefe-connector-es: Elasticsearch connector implementation migrated from fe-coreUnified catalog/table/scan classes:
Capability-based feature detection: Connectors declare capabilities
(SUPPORTS_INSERT, SUPPORTS_PARALLEL_WRITE, SUPPORTS_PASSTHROUGH_QUERY) so the
optimizer can make correct distribution and planning decisions without
type-checking specific catalog classes.
GSON backward compatibility: GsonUtils.registerCompatibleSubtype remaps
persisted JdbcExternalCatalog/EsExternalCatalog metadata to
PluginDrivenExternalCatalog for seamless upgrade from older Doris versions.
JDBC-specific behavioral preservation:
Release note
Introduce a pluggable connector SPI framework (fe-connector) that allows
external data source integrations to be developed as independent modules.
JDBC and Elasticsearch catalogs are migrated to this new architecture.
Existing catalogs created on older versions are automatically upgraded --
no user action is required. All existing JDBC and ES catalog functionality
is preserved, including predicate pushdown, identifier mapping, property
validation, and transactional write semantics.
Check List (For Author)