[DISCUSS] Apache Doris support for the Gravitino Spark connector

[jiangxt2]

Summary

This discussion asks whether Apache Doris support should be contributed to Apache Gravitino's Spark connector, and what the appropriate contribution and architecture path should be.

This is an independent reference implementation, not an official Apache Gravitino feature:

https://github.com/jiangxt2/gravitino-doris-spark-connector

This discussion focuses on governed batch reads.

This is not a request to merge the current implementation as-is. The question is whether the validated read capabilities should live in the generic Spark JDBC connector, a dedicated Doris adapter, a shared Gravitino extension point, or an external connector.

Motivation

Apache Gravitino 1.3.0 documents generic Spark JDBC support for MySQL and PostgreSQL and provides a separate jdbc-doris Server Catalog for Doris metadata. In the tested configuration, the native composition can initialize a catalog and read a scalar Doris table, but this is feasibility evidence rather than a complete Doris Spark support contract. Doris-specific schema types, global query semantics, tablet reads, authorization ordering and fail-closed connection security need an explicit design.

The independent connector combines Gravitino authorization and credential vending with the official Doris tablet reader and Spark's JDBC V2 planner. It is offered as a reference for identifying the Doris-specific capabilities that may supplement the native connector, not as a predetermined upstream design.

What the reference implementation adds

The reference implementation is not presented as a replacement for Gravitino's native Spark connector. It makes the Doris-specific gap explicit by adding a data plane and support contract around the existing Gravitino governance model:

Area	Native Gravitino composition today	What the reference implementation explores	Potential upstream boundary
Doris data plane	The generic Spark JDBC path is documented for MySQL/PostgreSQL, while jdbc-doris is a Doris Server Catalog. A basic scalar read gate has passed experimentally, but no complete Doris-specific Spark read contract is established.	A Doris-aware detail-read path using the official Doris tablet reader for eligible scans, alongside SQL reads.	A dedicated Doris adapter or an extension seam; not necessarily the generic JDBC layer.
Doris schema and types	No complete Spark-side Doris type contract is established; JDBC metadata can be lossy for Doris-specific or complex types.	Explicit hybrid type normalization and fallback representations, with a verified matrix for the tested types.	A shared schema/type extension or Doris-specific adapter.
Global query semantics	The generic JDBC planner provides a baseline path, but full Doris parity for aggregates, Top-N, limit, offset and partitioned reads is not independently established.	Reuse Spark JDBC V2 for semantics-sensitive SQL operations and select native tablet reads only for eligible detail scans; differential tests compare results.	Capability/lane policy in a Doris adapter or shared connector seam.
Governance and connection security	Gravitino already provides authorization and credential-vending concepts, but Doris-specific authorization-before-I/O and fail-closed connection behavior are not independently established in the native path.	Make SELECT_TABLE ordering, unsafe-configuration rejection, redaction and strict JDBC TLS behavior explicit and testable.	Shared governance hooks plus Doris-specific validation, subject to maintainer design.

These are candidate contribution boundaries, not claims that the native connector is unusable or that every reference capability is ready for upstream adoption.

Goals

Goal	Description
Clarify the contribution path	Decide whether this work should become an Apache Gravitino contribution, an external connector, or a staged combination.
Keep Doris semantics explicit	Define supported types, planner operations, partitions and transports instead of relying on generic JDBC behavior implicitly.
Keep governance auditable	Define authorization order, credential handling, configuration validation, redaction and trust boundaries.
Establish a maintainable test contract	Agree on Spark, Gravitino, Doris and executor/classpath compatibility gates before upstream implementation.

Current evidence

The evidence below is deliberately separated between the native Gravitino path and the independent reference implementation.

Area	Native Gravitino Spark connector	Independent connector read path
Basic catalog and scalar read	An experimental basic catalog/scalar gate passed on Doris 3.0.6.2 and 4.0.6 in the tested Spark 3.5.8 configuration; this is feasibility evidence, not complete Doris Spark support	A selected 17-method read-only targeted subset passed on both Doris versions; this is not the complete current integration suite
Schema, planner and types	Scalar schema, aggregate/Top-N and a numeric predicate have preliminary evidence; the native special-type matrix is not complete	Hybrid type normalization, SQL/native lane selection and direct JDBC differential tests exist
Partitioning, security and failures	Not independently validated as a native matrix	Read-path tests cover partitioning, authorization ordering, configuration rejection, schema cache and strict JDBC TLS
Performance	Preliminary local observations only; no release-level ranking	Governed SQL is compared with direct JDBC; native tablet and generic JDBC remain different data planes

The reference tests use Gravitino 1.3.0, Spark 3.5.8, and Doris 3.0.6.2/4.0.6. These versions describe the reference environment, not upstream support commitments.

The native special-type, partition, failure-path and security areas remain not yet verified. The reference implementation is not a substitute for native Gravitino evidence.

Candidate contribution shapes

1. Extend the generic Spark connector. Reuse the existing catalog facade and add Doris-specific provider, type and capability behavior. This keeps the user entry point unified, but may push tablet planning and Doris-specific schema semantics into a generic abstraction.
2. Add a dedicated Doris Spark adapter. Give the Doris data plane a clear home while keeping generic JDBC generic. This is clearer architecturally but adds modules, dependencies and maintenance surface.
3. Contribute a shared extension seam first. Upstream the smallest reusable provider, credential, schema or capability hooks and keep the Doris reader external until the boundary is accepted. This is a possible low-risk first step; keeping the complete connector external remains another valid outcome.

Reference architecture under discussion

The current project uses the following read-path separation as a reference, not as a predetermined upstream design:

Spark application
       |
       v
Gravitino Spark integration
       |
       +-- authorization and credential vending
       +-- governed physical-schema contract
       |
       +-- SQL lane: Spark JDBC V2 planner for global SQL semantics
       |
       +-- native lane: official Doris tablet reader for eligible detail scans

The architectural question is where this lane selection and its support contract should live.

Required validation before an upstream implementation claim

Before an upstream support claim, the community should agree on a minimum gate covering:

• native scalar, schema, special-type and partitioned-read behavior;
• aggregate, Top-N, global limit and offset result parity;
• authorization-before-I/O, unsafe configuration rejection and sensitive-data redaction;
• TLS success/failure paths and executor dependency visibility;
• Doris/Spark compatibility versions and a cleanly isolated performance comparison.

Suggested contribution path

1. Use this Discussion to confirm whether Doris Spark support is in scope for Gravitino.
2. Let maintainers choose the ownership boundary: generic connector, dedicated adapter, shared seam or external project.
3. Write a design document if the accepted scope crosses modules, then split implementation and Doris-backed tests into reviewable contributions.
4. Keep unverified native behavior and project-only features out of the official support statement until the corresponding upstream tests pass.

Questions for Gravitino maintainers

1. Does the community want Apache Doris support in the Gravitino Spark connector?
2. Should this begin as a generic Spark connector extension, a dedicated Doris adapter, or an external connector with a smaller upstream seam?

References

• Apache Gravitino Discussions
• Gravitino Discussion #11929: Iceberg V3 type support
• Gravitino Discussion #11295: Long-term architecture for Lance support
• Apache Gravitino RFC directory
• Apache Gravitino design documents
• Gravitino 1.3.0 Spark JDBC Catalog
• Gravitino 1.3.0 Doris Catalog
• Ray Issue #65022: RFC for a Doris datasource
• Independent reference connector

[jerryshao]

Hi @jiangxt2 this is a good idea. I think it is valuable to add to the Spark connector support. You can create issues and work on this. @yuqi1129 @diqiu50 can help you to review and merge this feature.