Native SQL DDL support for Hudi table creation across engines (Trino, Presto etc.)

[OjashKush]

Problem

Today, table creation via CREATE TABLE DDL is fully supported only through Spark SQL (via Hudi's Spark SQL extensions). Other engines — Trino, Hive, etc. — either don't support creating Hudi tables via SQL at all, or only have partial/inconsistent support (e.g. querying existing tables works, but creating new ones doesn't, or the config keys/syntax diverge from what Spark SQL expects).

In practice, this means anyone creating a new Hudi table today has to go through Spark SQL, regardless of which engine they'll actually read/write with day-to-day.

Motivation

• Multi-engine adoption: Hudi's core value proposition is being a multi-engine format (Trino, Presto, Hive, Spark). Table creation shouldn't be gated behind a single engine.
• Spark as a mandatory bootstrap step: Teams that are Trino-first for their actual workloads still have to stand up Spark just to create a table. That's an unnecessary dependency and operational overhead.
• Consistency of schema/config handling: Each engine currently has its own (or no) way of expressing Hudi-specific table properties — record key, partition path, precombine field, etc. — at creation time. This creates room for subtle misconfiguration when a table created in one context is later used from another engine.
• Accessibility for analysts: Analysts frequently need to create ad-hoc tables and are SQL-heavy by default. Analysts and other less Spark-fluent users are often intimidated by Spark SQL / Spark session setup, but still need to create Hudi tables to get started. Trino has no CREATE TABLE path for Hudi today, so when analysts need a quick ad-hoc table, the practical workaround is to create it in Iceberg instead. Supporting CREATE TABLE natively in Trino would close that gap directly — no new tooling for analysts to learn, and no reason to default to Iceberg just to get a table created.

Ask

Opening this up for discussion:

• Is there existing work or interest in closing this gap for specific engines (e.g. Trino DDL)?
• What's the right shared abstraction here - should DDL translate through a common catalog/metaclient path, so config handling stays consistent across engines rather than each engine reinventing its own CREATE TABLE handling?
• Given the breadth of per-engine catalog/DDL work involved, should this be scoped as a formal RFC?

cc @vinothchandar - flagging this as a discussion per your suggestion. Curious to hear from others who've hit this gap too.

[danny0405]

Flink already support create table though Flink SQL with a configurable catalog, did you try that:

https://hudi.apache.org/docs/flink-quick-start-guide#create-table
https://hudi.apache.org/docs/sql_ddl/#create-table-1

[OjashKush]

Yes, thanks for Flink part - but the issue is that both Flink and Spark require session/cluster initialization, which is fine for automated jobs but adds real friction for interactive, ad-hoc SQL. Our use case is Trino specifically: we want to create, insert etc Hudi tables directly through Trino without bootstrapping Flink or Spark first.

[danny0405]

sounds like you want some function similar with the catalog, it actually makes sense:

Trino supports table-creation APIs through its connector SPI, but the current Hudi connector does not implement them.

Today, the Trino Hudi connector is read-only and expects Hudi tables to already exist and be synchronized to Hive Metastore. [Trino’s documentation says this explicitly](https://trino.io/docs/current/connector/hudi.html). The local implementation confirms it:

• HudiMetadata.java implements only read/metadata operations—no createTable() or beginCreateTable().
• HudiConnector.java provides a page source but no page sink, so it cannot write data.
• HudiTableProperties.java currently exposes only location and partitioned_by.

It is feasible to add empty-table creation using Trino SQL. The connector would implement Trino’s ConnectorMetadata.createTable() and:

1. Validate the schema and Hudi properties.
2. Initialize the physical Hudi table and .hoodie metadata.
3. Register the table in HMS.
4. Roll back initialization if catalog registration fails.

The prospective Trino syntax would look like:

CREATE TABLE hudi.sales.trips (
    id BIGINT,
    event_time TIMESTAMP(6),
    rider VARCHAR,
    city VARCHAR
)
WITH (
    location = 's3://warehouse/sales/trips',
    table_type = 'COPY_ON_WRITE',
    primary_key = ARRAY['id'],
    precombine_field = 'event_time',
    partitioned_by = ARRAY['city']
);

Those additional properties are illustrative—they would need to be added to the connector. Trino does not use Spark’s USING hudi syntax; the hudi catalog identifies the connector.

A sensible staged design would be:

• First: empty CREATE TABLE and perhaps register_table for an existing Hudi location.
• Later: CREATE TABLE AS SELECT and INSERT, which additionally require beginCreateTable(), a page-sink implementation, and Hudi commit/write handling. [Trino documents those write-side SPI requirements here](https://trino.io/docs/current/develop/insert.html).

So the short answer is: Trino has the necessary catalog SPI, and we can make this work, but the current Hudi connector does not support it yet.

cc @voonhous for the support for Trio Hudi SQL table creation/management.

[voonhous]

Thanks for the ping @danny0405, and thanks for writing this up @OjashKush -- the gap is real and your description of it is accurate. Trino has no CREATE TABLE path for Hudi today, and Danny's read of the connector is correct: HudiMetadata implements read-only ConnectorMetadata, there's no page sink, and HudiTableProperties exposes only location and partitioned_by.

Existing tracking

This is already filed as #15527 (from HUDI-5115) and upstream as trinodb/trino#14433. Both have been open since 2022 with no implementation attached. Worth linking this discussion to them so the context lives in one place rather than forking a third thread.

Unblocking you in the meantime

Separate from the long-term design, you probably don't need Spark for the specific thing you described. Table creation is engine-agnostic already:

• HoodieTableMetaClient.newTableBuilder().setTableType(..).setTableName(..).setRecordKeyFields(..).setPartitionFields(..).setTableCreateSchema(..).initTable(storageConf, basePath) is pure hudi-common. No Spark engine, no session. That's roughly 15 lines in any JVM process, and it's exactly what hudi-cli's create command calls. (Caveat so you're not surprised: the shipped hudi-cli-with-bundle.sh wrapper still puts Spark jars on the classpath, so if you want a genuinely Spark-free binary today, the small JVM shim is the cleaner route.)
• HMS registration is run_sync_tool.sh, which is a plain java -cp ... org.apache.hudi.hive.HiveSyncTool invocation. Hive and Hadoop jars only, no Spark.

That is not a substitute for native DDL and I'm not offering it as one. But if the immediate pain is "we have to stand up Spark just to bootstrap a table," those two steps remove it today, and it also lets you validate the property and schema mapping you'd eventually want Trino to emit.

The technical map, if someone picks this up

Sharing this in full because the shape of the work is not obvious from the outside, and whoever drives it shouldn't have to rediscover it.

Context that changes the calculus: the connector source moved into apache/hudi as the hudi-trino module (RFC-105, #18837, merged 2026-07-26), with a thin shim on the Trino side. DDL work now lands in this repo on Hudi's cadence rather than being gated on a Trino release plus trinodb's bar for turning a read-only connector writable. That was the main structural reason #14433 sat for three years.

Sensible staging:

0. register_table / unregister_table procedures. The Iceberg connector ships system.register_table. Table already exists on storage, you just want it queryable without running Hive Sync separately. Smallest useful change, and it covers a real slice of the bootstrap pain on its own.
1. Empty CREATE TABLE + DROP TABLE. Metaclient init plus catalog registration.
2. INSERT / CTAS / MERGE. A different order of magnitude, see below.

Non-obvious parts of stage 1:

• Two schemas, not one. HudiMetadata#getColumnHandles derives columns from the HMS table via hiveColumnHandles(table, ...), not from the Hudi schema. So createTable() must write a correct Avro schema into hoodie.table.create.schema and a consistent HMS storage descriptor. [SUPPORT] Trino can't read tables created by Flink Hudi conector #9435 is what happens when those diverge: a table registered with no HMS columns, and Trino returns SELECT * not allowed from relation that has no columns. Good news is hudi-hive-sync is already a compile-scope dependency of hudi-trino, so reusing HiveSyncTool / HiveSchemaUtil for the catalog side is the natural path and makes Trino-created tables register identically to Spark/Flink-synced ones.
• Trino type -> Hudi type is a new direction. Today the connector only maps Hive/Hudi -> Trino. The ambiguous cases need explicit decisions: TIMESTAMP(6) vs (3), VARCHAR(n)/CHAR, DECIMAL fixed vs bytes, UUID, nullability inside ROW/ARRAY/MAP, and the newer custom types. None of these fail at CREATE TABLE. They fail later, when another engine writes into the table. [BUG] hudi-trino: pushed-down predicate on a float -> double evolved column fails the query #19457 (pushdown on a float -> double evolved column) is a live example of how cheaply these assumptions drift.
• Pin the created table version deliberately. Whatever hoodie.table.version gets initialized has to be one the connector can still read after someone else writes to it. Concretely: the connector cannot read RFC-103 native log blocks today (HudiTrinoIOFactory#getFileFormatUtils throws). Creating an empty MOR table at v10 from Trino would succeed, and then the first Spark or Flink delta write produces logs Trino can't read. Stage 1 should pin the version or restrict to COW until that closes.
• Small property allowlist plus an escape hatch. Hudi has hundreds of table configs. Enumerate only what must be right at init (table_type, primary_key, precombine_field, partitioned_by, location, key generator, hive-style partitioning, record merge mode) and add a passthrough map for arbitrary hoodie.* instead of growing a bespoke Trino property per config. Aliases should match what Spark SQL accepts (HoodieOptionConfig: primaryKey, preCombineField, type) so the same table doesn't describe itself differently depending on where you run SHOW CREATE TABLE.
• Failure ordering. On Danny's rollback point: init .hoodie first, then register. On registration failure, clean up the base path only if the connector created it, never a pre-existing directory.

On your shared-abstraction question

Yes, and most of it exists. HoodieTableMetaClient.newTableBuilder() + HoodieTableConfig is already the engine-neutral sink, and Spark's HoodieCatalog and Flink's HoodieHiveCatalog both funnel into it. What's missing is the layer directly above: DDL options -> validated HoodieTableConfig. Spark's lives in HoodieOptionConfig (hudi-spark-common, Scala), Flink's in hudi-flink, and neither is reusable. Lifting that normalization and validation into hudi-common, and standardising on hudi-hive-sync for registration, is what would make a Trino createTable() genuinely thin. Worth saying plainly: adding createTable() to Trino alone, without that, would add a fourth dialect rather than fix the consistency problem you describe.

On RFC scoping

I'd split it three ways instead of one, because they have very different costs:

• register_table + empty CREATE/DROP: no RFC needed. A design sketch on Support for DDL/DML in Hudi connector #15527 and a PR against hudi-trino is enough.
• Shared DDL normalization in hudi-common: short RFC or design doc, since it moves shared code and can change Spark and Flink behavior.
• INSERT / CTAS / MERGE from Trino: RFC, definitely. This makes Trino a second full writer, so commit and rollback, markers, index selection, MDT updates, OCC and locking, key generation, precombine, file sizing, and who runs compaction/clean/clustering all need answers. The mechanics aren't hypothetical -- hudi-trino already uses HoodieJavaWriteClient at test scope to build fixtures -- but mapping Trino's sink model (workers emit fragments, coordinator commits) onto a Hudi commit needs deliberate design, and hudi-java-client's index and table-service coverage relative to the Spark client is the real open question.

Follow-up on process and next steps in a separate comment, to keep this one focused on the technical shape.

[voonhous]

Following up on the scoping question above, on process and ownership.

On who drives it

Being straightforward about how this works, since you asked whether to formalize it: the RFC process expects a champion who carries the proposal through design and implementation. Our RFC table has an ABANDONED status for a reason -- proposals without someone driving them stall regardless of merit. Committer attention here is finite and tends to follow whatever has an owner, so the honest answer to "should this be an RFC" is that it's worth one exactly when someone intends to build it.

If that's you or your team, I'd genuinely encourage starting at stage 0. It's well-scoped, it's a real improvement on its own, and it's a reasonable first contribution to the module. I can commit to design review, PR review, and helping you navigate the module specifics -- the build is profile-gated and JDK 25 (mvn -Phudi-trino -pl hudi-trino compile, tests behind the additional hudi-trino-tests profile), which is the sort of thing that costs a newcomer a day otherwise.

One scoping note: I'd keep this Trino-only. Presto's Hudi connector lives in prestodb/presto and shares none of this code, so folding it in would make the proposal unactionable.

[vinothchandar]

@OjashKush any interest in driving trino DDL support in the connector?

[OjashKush]

@vinothchandar Yes, I can pick this up. I will start with stage 0 (register_table + empty CREATE/DROP) as @voonhous us laid out. Thanks for the direct ask.

[vinothchandar]

@OjashKush Sounds good.. It might be worth doing some research around how/if this write path can be supported.
https://hudi.apache.org/docs/write_operations#write-path

Ultimately, we need to take the incoming (parallel) record streams, shuffle them for indexing/precombining/optimizing layout, then write.

[vinothchandar]

Quick Trino connector SPI research says,

source SELECT ──┐
                ├── join on merge condition        ← engine-planned exchanges (index lookup
target scan ────┘   (t.uuid = s.uuid)                 can be done by pushing filter to target scan)
        │
  merge processor: tags each row INSERT / DELETE / UPDATE(→delete+insert), attaches $row_id
        │
  ═══ remote exchange: merge partitioning scheme ═══   ← Only one connector-controlled shuffle allowed
        │   (our challenge is doing optimizing file size/layout and writing MDT RLI etc from here)
        ▼
  ConnectorMergeSink per writer task  → writes files → finish() → fragments → sent to coordinator for commit

I think we cannot easily support RLI or SI writes in Trino. We should probably design the V1 without those. Then get creative later on.

Hope that helps! this was the limitation I was referring to at the Hudi meetup talk.