[SPARK-58419][SQL] Support ANSI SQL UNNEST in the FROM clause

[ganeshashree]

What changes were proposed in this pull request?

This PR adds support for the ANSI SQL UNNEST collection derived table in the FROM clause:

UNNEST ( expression [ , ... ] ) [ WITH ORDINALITY ] [ table_alias ]

UNNEST expands one or more arrays into a relation, producing one row per
element:

• Multiple arrays are expanded in parallel: the number of output rows equals the length of the longest array, and shorter arrays are padded with NULLs.
• A NULL array is treated as an empty array and contributes no rows (a NULL element, by contrast, produces a NULL row).
• WITH ORDINALITY appends a trailing 1-based BIGINT position column.
• Correlated arrays are supported via LATERAL (FROM t, LATERAL UNNEST(t.arr)), reusing the existing lateral-join
  machinery, and LEFT JOIN LATERAL UNNEST(...) ON true preserves outer rows whose array is empty or NULL.
• Only ARRAY arguments are accepted. Each array contributes exactly one output column holding its element as-is; unlike inline, an array of structs is not expanded into one column per field.

The semantics follow PostgreSQL and Trino (parallel multi-array expansion with NULL padding, 1-based WITH ORDINALITY). BigQuery's single-array UNNEST and 0-based WITH OFFSET are intentionally not adopted.

References:

• PostgreSQL: https://www.postgresql.org/docs/17/queries-table-expressions.html
• Trino: https://trino.io/docs/current/sql/select.html
• BigQuery: https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax

Implementation notes:

• Grammar: UNNEST and ORDINALITY are added as non-reserved keywords (like PIVOT/UNPIVOT), so existing identifiers named unnest/ordinality keep working. A dedicated relationPrimary alternative (#unnestTable) and an unnest production are added to SqlBaseParser.g4, with matching lexer tokens. A table-valued function named unnest can still be invoked by quoting the name: unnest(...).
• A new Unnest generator expression implements the zip-and-pad plus ordinality semantics. AstBuilder.visitUnnestTable desugars UNNEST into a Generate over a OneRowRelation, reusing the shared FROM-clause aliasing helper (mayApplyAliasPlan). UnnestTableContext is whitelisted in the two LATERAL validation sites so correlated and LEFT JOIN LATERAL forms work.
• Unnest uses interpreted evaluation (CodegenFallback) with a lazy iterator (one output row built per pull), consistent with other non-CollectionGenerator generators such as ReplicateRows. A dedicated whole-stage-codegen path is left as future work (tracked separately) and is documented in the class scaladoc.

Example:

-- multiple arrays, parallel expansion with NULL padding, plus ordinality

SELECT * FROM UNNEST(array(1, 2), array(10, 20, 30)) WITH ORDINALITY AS t(a, b, ord);

+----+---+---+
|   a|  b|ord|
+----+---+---+
|   1| 10|  1|
|   2| 20|  2|
|NULL| 30|  3|
+----+---+---+

Why are the changes needed?

UNNEST is part of the SQL:2016 standard and is available in PostgreSQL, Trino/Presto, and BigQuery (and, via FLATTEN, Snowflake). Spark has no equivalent ANSI syntax today: users must rewrite UNNEST(...) to LATERAL VIEW EXPLODE or the DataFrame explode API. This is a recurring pain point in migrations when porting queries from other engines. Array expansion is an everyday operation, so this closes a genuine standards-compliance gap with broad reach.

Does this PR introduce any user-facing change?

Yes. UNNEST(...) [WITH ORDINALITY] can now be used in the FROM clause. This is a new feature, not a change to existing behavior; UNNEST and ORDINALITY remain valid as regular (non-reserved) identifiers, so existing queries continue
to parse unchanged. New user-facing SQL reference documentation is added under docs/sql-ref-syntax-qry-select-unnest.md.

How was this patch tested?

Added new tests:

• GeneratorExpressionSuite — unit tests for the Unnest generator: single and multiple arrays, WITH ORDINALITY, empty/NULL arrays, column naming/nullability, lazy evaluation (elements read only as rows are pulled), string/EXPLAIN representation, and type-check errors.
• PlanParserSuite — parser tests: single/multiple arrays, WITH ORDINALITY, table and column aliases, LATERAL correlation, non-reserved keyword backwards compatibility (unnest/ordinality as identifiers), and the quoted-name TVF escape hatch.
• SQLQueryTestSuite golden-file unnest.sql — end-to-end coverage including parallel padding, NULL vs empty arrays, NULL elements, nested arrays, LEFT JOIN LATERAL outer-row preservation, arrays of structs, and error cases (non-array and MAP arguments).

Regression: existing suites that reference unnest/ordinality (e.g. postgreSQL/with.sql, which uses ordinality as a CTE and table name) pass without golden-file regeneration, and generators.sql, generators-resolution-edge-cases.sql, join-lateral.sql, and postgreSQL/join.sql all pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

[cloud-fan]

0 blocking, 0 non-blocking, 3 nits.
The implementation is coherent; three documentation and comment nits should be cleaned up.

Nits: 3 minor items (see inline comments).

Verification

I traced the new grammar alternative through AstBuilder.visitUnnestTable, generator resolution, and GenerateExec. The generator's declared schema matches its emitted rows: multi-array columns are conservatively nullable, ordinality is non-nullable BIGINT, null arrays contribute zero elements, and padding occurs to the longest input. The SQL golden coverage exercises both direct and correlated execution paths.

[cloud-fan]

Merge Summary:

• merged into master fafb64c
• merged into branch-4.x c10fe90

Posted by merge_spark_pr.py