docs(sql-ddl): partition columns must be declared last in CREATE TABLE

[deepakpanda93]

Describe the issue this Pull Request addresses

Closes #17357. (JIRA: HUDI-8827, a subtask of HUDI-9109 "Bridging Hudi Spark SQL behavior gaps".)

Spark moves partition columns to the end of the table schema. If a partition column is declared anywhere else in
CREATE TABLE, the stored column order silently differs from what was written, and a positional
INSERT INTO ... SELECT assigns values to the wrong columns. The reporter hit this as a baffling cast error on a ts
column they had explicitly cast to bigint. Nothing on the SQL DDL page mentions the constraint.

The example in the Create partitioned table section had exactly that shape, so following it produced the bug.

Summary and Changelog

Two changes to the Create partitioned table section:

1. Fixed the example. It declared (id, name, dt, hh) with PARTITIONED BY (dt), leaving hh after the partition
   column. Changed the clause to PARTITIONED BY (dt, hh). The schema is untouched — with both columns partitioned they
   are already the trailing columns, in declaration order, so nothing is reordered.
2. Added a :::caution stating that partition columns must be declared last, with the resulting schema, both
   failure modes, and the explicit-column-list alternative.

The existing multi-field :::note is unchanged; testing confirmed it was already correct.

Applied to website/docs/sql_ddl.md (next) and website/versioned_docs/version-1.2.0/sql_ddl.md (current released
docs), per the next-plus-current convention used in #19473 and #19551. The section is byte-identical in 1.0.0–1.1.1 and
carries the same example; those were left alone — happy to widen if preferred.

Reproduction

Spark 3.5.7, hudi-spark3.5-bundle_2.12:1.2.0.

The issue as reported. Declaring (id, name, price, dt, ts) with PARTITIONED BY (dt) stores the table as
(id, name, price, ts, dt), and a positional insert fails:

insert into t select 1, 'a1', 10.0, '2021-03-21', 1L;

[INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST]
Cannot safely cast `ts` "STRING" to "BIGINT"

The worse case, which the issue does not mention. When the shifted columns happen to be type-compatible there is no
error at all. Declaring (id, a, dt, b) as strings and inserting 1,'VALUE_A','VALUE_DT','VALUE_B' returns exit 0 and
stores b='VALUE_DT' with dt='VALUE_B' — the partition value written into a data column and a data value used as the
partition. That is why the new text is a caution and calls out the silent case explicitly.

All four variants of the example's own schema (id BIGINT, name STRING, dt STRING, hh STRING), inserting
1,'n1','2024-01-01','10':

PARTITIONED BY	Stored column order	Result
(dt) — what the page had	id, name, **hh, dt**	❌ silently stores hh='2024-01-01', dt='10', exit 0
(dt, hh) — what the page now has	id, name, dt, hh (unchanged)	✅ dt='2024-01-01', hh='10'
(hh)	id, name, dt, hh (unchanged)	✅ correct — hh is already last
(hh, dt)	—	❌ rejected at analysis: Partition schema fields order does not match the table schema fields order, tableSchemaFields: (dt, hh), partitionFields: (hh, dt) (HoodieSchemaUtils.checkPartitionSchemaOrder)

Two things follow. Only the first row was broken, which is why this PR changes one line rather than restructuring the
section. And the (hh) row shows the constraint is "partition columns must be trailing" in general, not something
specific to dt.

Also verified that naming the columns explicitly avoids the mismatch even on a mis-declared table:
INSERT INTO t (id, name, price, dt, ts) SELECT ... lands every value correctly. That is why the caution offers it as an
alternative.

Worth noting the last row for contrast: getting the PARTITIONED BY order wrong fails loudly, whereas getting the
declaration order wrong — the subject of this PR — is checked by nothing.

A note on scope

The referenced PR #12577 was closed unmerged; it only added a test reproducing the problem, with all three insert
variants commented out as "None of these queries work". So there is no code fix to describe here — this documents
current behaviour, which is what the issue asks for.

Site verification

npm run build passes with the warning set byte-identical to a baseline build of the same base commit — no new broken
links or anchors. /docs/sql_ddl and /docs/next/sql_ddl were loaded from npm run serve and the code block renders as
PARTITIONED BY (dt, hh); with the schema unchanged, the caution renders as an admonition, and the section anchors are
intact. /docs/1.1.1/sql_ddl still shows the old text, as intended.

Impact

Documentation only. No code, config, or behaviour change.

Risk Level

none

Documentation Update

This PR is the documentation update — the SQL DDL page, /docs/sql_ddl and /docs/next/sql_ddl.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-agent]

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the docs update! The change correctly documents that Spark stores partition columns at the end of the table schema and fixes the "Create partitioned table" example so its declaration order matches, with a clear caution covering both the cast-error and silent-misalignment failure modes. The content aligns with the existing :::note and with Spark's actual behavior; please still route to a Hudi committer or PMC member (e.g. @yihua) for final confirmation.

cc @yihua

[deepakpanda93]

Widened the scope in c6f6796, taking up the "happy to widen if preferred" offer in the description. Now 6 files instead of 2.

Why

Auditing the other versioned copies showed every one of the ten carries the identical broken example — (id, name, dt, hh) with PARTITIONED BY (dt). The 0.x and 1.x sections differ only by trailing whitespace on two lines, so this is one defect duplicated ten times, not a next-only problem.

That puts it in the correction bucket rather than the addition bucket. The next-plus-current convention (#19473) is for additive guidance; a wrong example that silently corrupts data is closer to #19459, which backported a stale default to every affected 1.x version. The cause is Spark's partition-column reordering rather than anything Hudi-version-specific, so the example is wrong on every release that ships it, and those pages are all live.

What changed

The corrected example and the caution are now applied to next, 1.2.0, 1.1.1, 1.0.2, 1.0.1 and 1.0.0. All six Create partitioned table sections are byte-identical after the change.

0.14.0/1/2 and 0.15.0/1 are deliberately left alone as end-of-life — flagged here rather than left as a silent gap, since they carry the same broken example. Happy to include them if the project would rather every live page be correct.

Verified on 1.0.2 before backporting

Since the change now touches older releases, I re-ran the decisive pair against hudi-spark3.5-bundle_2.12:1.0.2 rather than carrying the 1.2.0 result across by inference:

PARTITIONED BY	Stored column order	Positional insert 1,'n1','2024-01-01','10'
(dt)	id, name, hh, dt	❌ dt='10', hh='2024-01-01' — silent, exit 0
(dt, hh)	id, name, dt, hh (unchanged)	✅ dt='2024-01-01', hh='10'

Identical to 1.2.0. (The first 1.0.2 run also threw an unrelated FileAlreadyExistsException from the metadata-table writer on local FS; it retried and the write completed. I re-ran with hoodie.metadata.enable=false for a clean result, which is what is quoted above.)

Also

Bumped last_modified_at on the two copies that carry a value — I had missed that in the first commit. The four older copies have the field present but blank in this repo, so I left them blank rather than invent dates for historical releases.

Build is clean with the warning set byte-identical to a baseline of the same base commit. Rendering was checked on /docs/sql_ddl, /docs/next/sql_ddl, and the 1.1.1, 1.0.2, 1.0.1, 1.0.0 pages — all show PARTITIONED BY (dt, hh); with the caution; 0.15.1 still shows the old form, as intended.

[hudi-agent]

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the docs update! This change fixes the Create partitioned table example (declaring hh as a partition column so no silent schema reordering occurs) and adds a well-scoped :::caution explaining Spark's partition-column-last behavior with both the cast-error and silent-wrong-column failure modes. The technical claims match the documented reproduction and the fix is applied consistently across the next and versioned docs; please route to a Hudi committer or PMC member for final confirmation.

cc @yihua