[SPARK-58460][SQL] Do not leak generated column expressions into DataFrame schemas and derived tables

[szehon-ho]

What changes were proposed in this pull request?

A generated column's generation expression was carried as GENERATION_EXPRESSION metadata on a relation's output attributes. This PR treats that key as internal metadata, so it never reaches a relation's output, and reworks the write path to take the expression from the table's V2 columns instead:

• GENERATION_EXPRESSION is added to INTERNAL_METADATA_KEYS, so removeInternalMetadata strips it. DataSourceV2Relation.create already ran the table schema through that, so batch relations are clean; the StreamingRelationV2 paths built attributes directly from table.columns.toAttributes and now use a new toOutputAttributes, which applies the same strip (keeping field IDs, which the column-ID feature deliberately exposes).
• ResolveOutputRelation attaches the expressions to the expected output from table.columns() (the persisted source of truth) via GeneratedColumn.attachGenerationExpressions, only for as long as resolving the write takes, so TableOutputResolver can still auto-fill missing generated columns.
• Instead of removing metadata from auto-filled columns to suppress validation, the columns Spark computed are now marked with a new internal key, __auto_filled_generated_column, and ResolveTableConstraints skips those. Validation is therefore the default: any value that did not come from the generation expression gets a CheckInvariant, as before.

Why are the changes needed?

Because the expression rode along on the relation's output, it leaked in two user-visible ways.

It showed up in a DataFrame's schema:

sql("CREATE TABLE testcat.t (id INT, doubled INT GENERATED ALWAYS AS (id * 2)) USING foo")
spark.table("testcat.t").schema("doubled").metadata
// {"GENERATION_EXPRESSION":"id * 2"}

And, since a table created from a query derives its columns from that schema, the generated column was promoted back into a real generated column on the new table:

sql("CREATE TABLE testcat.dst USING foo AS SELECT * FROM testcat.t")
// dst.doubled is a generated column copied from the source

spark.readStream.table("testcat.t").writeStream.toTable("testcat.dst2")
// fails: the target is created with a generated column, and streaming writes to
// tables with generated columns are unsupported

Generated column values on writes were added recently (SPARK-57644) and are not in any release, so this corrects the behavior of an unreleased feature.

Does this PR introduce any user-facing change?

Yes, within unreleased branches only. GENERATION_EXPRESSION no longer appears in the metadata of a DataFrame's schema (batch reads, readStream, and SELECT), and CREATE TABLE AS SELECT / writeStream.toTable now create ordinary columns rather than inheriting the source's generated columns. Auto-filling and validation of generated columns on writes to an existing table are unchanged.

How was this patch tested?

GeneratedColumnWriteSuite (72 tests, all passing), with three new cases:

• generation expression is not exposed in the read schema, covering spark.table, spark.read.table, SELECT, and readStream.table
• CTAS from a table with generated columns does not create generated columns, and the resulting column behaves as an ordinary one
• streaming write to a new table does not create generated columns

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor 2.5, Opus 5

[cloud-fan]

0 blocking, 0 non-blocking, 0 nits.
The metadata lifetime is now bounded to write analysis, and the read, derived-table, auto-fill, and validation paths remain internally consistent.

Verification

I traced the metadata from V2 table columns through relation construction, temporary write-time attachment, output resolution, and constraint selection. Read outputs remove both internal generated-column keys, while write resolution restores the persisted expression only on its expected attributes; auto-filled target attributes are marked after character/varchar metadata cleanup, and unmarked generated columns still receive invariants.

[cloud-fan]

Merge Summary:

• merged into master 9586a2a
• merged into branch-4.x 0197331

Posted by merge_spark_pr.py