Spark query and write serving endpoint for hudi-lakehouse

[cshuo]

Summary

We would like to discuss adding a persistent Spark serving endpoint under hudi-lakehouse, providing interactive query and write access to Hudi tables.

The initial lakehouse implementation provides Trino-based query serving and Spark writer examples. A shared Spark endpoint would expose Hudi's Spark SQL and DataFrame capabilities without requiring each client to configure and launch its own Spark application.

Rather than simply adding a Spark Thrift Server chart, we should evaluate the available serving approaches and keep the design friendly to Spark Connect, ADBC, and potentially Substrait.

The goal of this discussion is to collect requirements and compare options, not to require every interface in the first release.

Goals

• Provide an optional Spark serving component and Helm chart under hudi-lakehouse.
• Support interactive queries and Hudi writes, including table creation, INSERT, UPDATE, DELETE, and MERGE INTO.
• Support writes through direct Spark endpoint clients. Agent gateway write tools are out of scope.
• Reuse the catalog and object-store configuration of the lakehouse deployment.

The exact operation set and production requirements for the first version remain open for discussion.

Architecture layers

The relevant technologies address different layers and are not necessarily mutually exclusive:

Layer	Possible interfaces
Client API	JDBC/ODBC, Spark Connect clients, ADBC
Transport	HiveServer2/Thrift, Spark Connect gRPC, Arrow Flight SQL
Query/plan representation	SQL text, Spark logical plans, optional Substrait plans
Execution	Spark SQL with the Hudi bundle and Spark SQL extensions
Data and catalog	Hudi, HMS/Glue-compatible catalog, object storage

In particular:

• Spark Thrift Server exposes SQL through HiveServer2 and provides broad JDBC/ODBC compatibility.
• Spark Connect provides a decoupled Spark client/server model, using protobuf logical plans over gRPC and Arrow for result transfer.
• ADBC is a client API rather than a server protocol. An ADBC client can connect through an ADBC driver for Spark, or through the ADBC Flight SQL driver when the server exposes a Flight SQL endpoint.
• Substrait is a portable plan representation rather than a transport or server implementation.

Supporting one does not automatically provide the others, but they are not necessarily mutually exclusive.

Options to consider

Option A: Spark Thrift Server

Deploy Spark Thrift Server with the Hudi bundle, Spark SQL extensions, and shared catalog/storage configuration.

Advantages:

• Mature JDBC/ODBC and SQL-tool compatibility.
• Straightforward deployment using an existing Spark server implementation.

Concerns:

• SQL-only interface; does not expose Spark Connect DataFrame APIs.
• Does not directly address Flight SQL or Substrait.
• Session isolation, metadata freshness, dependency configuration, and long-running server stability require careful validation.

Option B: Spark Connect Server

Deploy Spark Connect as the primary Spark endpoint.

Advantages:

• Supports remote SQL and DataFrame applications.
• Provides client/server separation and Arrow-based result transport.
• Can also serve JDBC/ADBC clients through a compatible Spark driver.

Concerns:

• ODBC clients need an additional frontend or adapter.
• Hudi writes, SQL extensions, and Spark/client version compatibility must be validated specifically in Connect mode.
• Authentication, session routing, and multi-tenant resource isolation require an explicit deployment design.

Option C: Kyuubi or another Spark SQL gateway

Use a gateway to manage Spark engines and client sessions.

Advantages:

• Established JDBC/ODBC integration.
• Provides authentication, multi-tenancy, and engine lifecycle management.
• Supports resource isolation through separately managed Spark engines.

Concerns:

• Adds another major runtime dependency and operational layer.
• Spark Connect support and its maturity must be evaluated separately.
• Does not automatically provide Flight SQL, ADBC compatibility, or Substrait support.

Option D: Multi-protocol Spark serving

Share a Hudi-enabled Spark image and catalog/storage configuration, while exposing independently enabled endpoints:

• Spark Connect for remote SQL and DataFrame clients.
• Optional Thrift/HS2 for JDBC/ODBC tools.
• ADBC access through an existing Spark ADBC driver, without requiring a new Flight SQL server.

This option combines existing serving interfaces rather than introducing a new server implementation.

Advantages:

• Supports different client ecosystems without committing to a single protocol.
• Reuses common runtime packaging and catalog/storage configuration.
• Allows interfaces to be introduced incrementally using existing servers and drivers.

Concerns:

• Multiple endpoints increase deployment, testing, and maintenance costs.
• Separate Spark applications increase runtime overhead and do not share sessions, temporary views, or caches.
• Sharing one driver instead requires additional compatibility and lifecycle validation.
• Flight SQL/Substrait adapters remain additional work. The Spark Substrait Gateway is an architecture reference, not a drop-in Hudi solution.

Initial validation

Before selecting an approach, we should validate:

• Hudi query and write correctness through direct clients.
• Spark/Hudi/client version compatibility.
• Concurrent sessions, cancellation, and visibility of external commits.
• Authentication, resource isolation, and long-running stability.

The first deliverable could then be one optional endpoint with a query/write quickstart, while keeping additional interfaces independently extensible.

Questions for the community

1. What should the first version support?
   Which clients and workloads are essential: JDBC/ODBC tools, remote SQL, or Spark Connect DataFrame applications? Which Hudi write operations are required on day one?

2. Which serving approach best fits these requirements?
   Should we start with Spark Connect, Spark Thrift Server, or a gateway such as Kyuubi? Experience from existing deployments would be helpful.

3. What isolation and security model is needed initially?
   Is a shared Spark application with multiple sessions sufficient, or do we need per-user engines, stronger resource isolation, and multi-tenant authentication from the beginning?

4. What level of ADBC support should we target?
   Is a compatible driver or adapter sufficient initially, or is there a concrete need for an Arrow Flight SQL endpoint?

5. What concrete Hudi use case would benefit from accepting Substrait plans?

Feedback on requirements, protocol preferences, and operational experience would be very helpful.

Related work

• Agentic lakehouse: built-in AI agent serving and operations for Hudi #19256 — Agentic lakehouse initiative.
• feat: agentic lakehouse v1 - hudi-lakehouse charts and hudi-agent-gateway #19265 — Initial hudi-lakehouse implementation.
• feat(gateway): add Spark Thrift Server as a selectable lakehouse engine #19411 — Gateway connectivity to an existing Spark Thrift Server; this does not deploy the server itself.
• Native SQL DDL support for Hudi table creation across engines (Trino, Presto etc.) #19484 — Native SQL DDL support across engines.
• Support for DDL/DML in Hudi connector #15527 — Trino Hudi connector DDL/DML support.

[danny0405]

I would start with Option A (Spark Thrift Server) for v1, while keeping the Hudi-enabled image and catalog/storage configuration reusable for Spark Connect later. If remote DataFrame applications are a day-one requirement, my preference would change to Option B.

To make the proposal easier to read and respond to, I suggest:

1. Lead with the user problem and the decision requested: which clients need a persistent endpoint, which operations they need, and whether v1 must support remote DataFrame APIs. Move the architecture-layer explanation after the proposed scope.
2. Separate v1 requirements from future extensions. Propose an initial client/operation set and ask the community to adjust it. Keep Flight SQL and Substrait in future work until there is a concrete consumer.
3. Use one comparison table with consistent dimensions: client coverage, gateway integration, isolation, operational cost, and compatibility gaps. Consolidate shared validation concerns into one checklist.
4. Treat Option D as an incremental expansion strategy. It combines the other endpoints rather than representing a separate server choice. Similarly, distinguish the client-protocol decision from the engine-management/isolation decision that motivates Kyuubi.
5. Focus the community questions on required clients, required Hudi operations, and required isolation. These answers should determine the implementation.

My reasons for preferring A initially are:

• feat(gateway): add Spark Thrift Server as a selectable lakehouse engine #19411 already proposes gateway integration through Spark Thrift Server. Although that PR is still open, deploying the server would complement a concrete integration path and should reduce the first release's scope.
• The listed operations—table creation, INSERT, UPDATE, DELETE, and MERGE INTO—can be expressed through SQL. These requirements alone do not establish a need for remote DataFrame APIs; Hudi correctness still needs to be validated through the chosen endpoint.
• ADBC does not require choosing Connect or implementing Flight SQL. The linked Spark ADBC driver supports both Thrift and Connect backends. We should state support in terms of a tested driver/backend/version combination, rather than assuming driver availability establishes Hudi write compatibility.

One scope clarification would help: a write-capable Spark endpoint does not automatically make the agent gateway write-capable. #19411 retains SELECT-only guardrails. Are the proposed writes intended for direct clients, or should this work also include gateway write tools?

For validation, I would explicitly include external-commit visibility, concurrent writers, and the outcome of a write when the client times out or cancels. Session separation and engine resource isolation should also be evaluated separately.

I would choose B first if notebooks or applications need remote DataFrame transformations immediately, and C if separately managed tenant engines are required from the beginning. Otherwise, one optional Thrift endpoint, a pinned Hudi/Spark combination, and a query/write quickstart seem like a focused first deliverable, with additional endpoints introduced as client demand is demonstrated.

This is an architectural preference; I have not validated Hudi compatibility against these server options.

[cshuo]

Thanks, Danny. Agreed on narrowing v1 and validating concrete driver/backend/version combinations.

Given Vinoth's operational concerns and the available Connect JDBC/ADBC drivers, I'm leaning toward Connect first, provided it covers our initial clients—not just for remote DataFrame support.

I'd propose keeping writes scoped to direct clients, with gateway write tools as separate work.

[vinothchandar]

Is there not a easy way to do JDBC/ODBC/ADBC https://github.com/HyukjinKwon/adbc-driver-spark and bet on the spark connect route.

thrift-server has so many issues at scale. We just rewrote the layer from scratch for Onehouse's query engine (lakegres).
I'd love to avoid thrift server altogether for a new feature like this.

[cshuo]

Thanks, Vinoth. Spark Connect looks viable:

• JDBC: Available from Apache Spark 4.1.
• ADBC: Both the driver you linked and the Foundry driver support Connect.
• ODBC: I haven't found a driver that directly supports generic Spark Connect endpoints.

I'd suggest starting with Connect under hudi-lakehouse and validating Hudi reads/writes through JDBC/ADBC. Do we have any ODBC-only clients that must be supported initially?

[vinothchandar]

If something like Apache Superset or some OSS BI stack works - end to end, we can pick and decide..

Can we do some research on what tools can work with JDBC/ADBC..