[SPARK-58366][SQL] Support JSON_TABLE table-valued function

[ganeshashree]

What changes were proposed in this pull request?

Add the ANSI SQL:2016 JSON_TABLE table-valued function, which shreds a JSON document into a relational table. A row path selects a sequence of JSON items and a COLUMNS clause projects a typed value out of each item into a column.

Syntax (flat, non-nested subset):

JSON_TABLE(json_expr, row_path
  COLUMNS (
    col1 FOR ORDINALITY,
    col2 <type> [PATH '<json_path>'],
    col3 <type> EXISTS [PATH '<json_path>']
  )
  [ { NULL | ERROR } ON ERROR ]
) [AS] alias

• A row path ending in [*] expands a JSON array into one row per element; a non-wildcard path yields a single row for the matched value.
• FOR ORDINALITY: a 1-based BIGINT row counter.
• Value columns are extracted and cast to the declared type; the path may be explicit (PATH '...') or implicit ('$.<columnName>').
• EXISTS columns are a presence test cast to the declared type; a present-but-null JSON value counts as existing, only an absent path is false.
• { NULL | ERROR } ON ERROR: NULL ON ERROR (default) yields no rows on null/malformed input; ERROR ON ERROR raises.
• Usable in a comma join and with LATERAL.

Implementation notes:

• A JsonTable Generator expression is wrapped by the existing Generate operator, so no new execution operator is introduced.
• A dedicated grammar production handles the COLUMNS(...) clause; AstBuilder.visitJsonTableRelation builds the plan. ERROR, ORDINALITY, and JSON_TABLE are added as non-reserved keywords.
• A token-aware navigator (JsonTableEvaluator) extracts values so that a missing path, a JSON null, and the literal string "null" are all distinguished (unlike get_json_object). Non-trailing wildcard paths are rejected during analysis via DATATYPE_MISMATCH.INVALID_JSON_TABLE_PATH.
• Only the flat subset is implemented; NESTED PATH, FORMAT JSON query columns, and DEFAULT ... ON EMPTY are left as follow-ups.

Why are the changes needed?

JSON_TABLE is the SQL-standard way to turn JSON into rows and columns, and is supported by Oracle, DB2, MySQL 8, PostgreSQL 17, Snowflake, and Trino. Spark previously required chaining from_json + explode/inline + get_json_object to achieve the same result. This folds that into one declarative, standard construct and eases migration from those systems.

Does this PR introduce any user-facing change?

Yes. It adds the JSON_TABLE SQL table-valued function and a SQL reference documentation page. ERROR, ORDINALITY, and JSON_TABLE are added as non-reserved keywords, so existing queries using them as identifiers continue to parse. There is no change to existing behavior.

How was this patch tested?

New end-to-end suite JsonTableSuite covering array expansion, ordinality, explicit/implicit paths, missing-vs-null semantics for value and EXISTS columns, NULL/ERROR ON ERROR, [*] over a non-array, mid-path wildcard rejection, non-explode string row items, arrays of scalars/strings, duplicate keys, structure-valued columns, LATERAL joins, nested-field extraction, and column aliases. Existing JsonExpressionsSuite, JsonFunctionsSuite, GeneratorFunctionSuite, PlanParserSuite, DDLParserSuite, SQLKeywordSuite, and SparkThrowableSuite continue to pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

[cloud-fan]

0 blocking, 1 non-blocking, 1 nit.
The semantics are coherent and well covered, but wide projections should avoid reparsing every row once per output column.

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:548: JSON_TABLE reparses each row item once per projected value or EXISTS column, so wide projections multiply parsing work by the column count. -- see inline

Nits: 1 minor item (see inline comments).

Verification

I traced the new grammar through AstBuilder into JsonTable and the existing Generate execution path. I also checked path validation, missing-versus-null handling, ANSI cast-mode capture, lazy parser cleanup, lateral activation, and the corresponding focused tests.

[cloud-fan]

can we fix merge conflicts?

[ganeshashree]

can we fix merge conflicts?

Fixed merge conflicts.

[cloud-fan]

2 addressed, 0 remaining, 5 new. (1 newly introduced, 4 late catches, 0 previously raised.)
1 blocking, 1 non-blocking, 3 nits.
The earlier whole-row reparsing concern is addressed, but exact numeric preservation needs correction before merge; the remaining items are localized efficiency and documentation cleanup.

Correctness (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:675: JSON_TABLE can round high-precision fractional values while serializing a matched fragment, before Spark casts it. -- see inline

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:564: Each projected value still creates a Jackson parser even when it is not a JSON string. -- see inline

Nits: 3 minor items (see inline comments).

Verification

I traced the new grammar through AstBuilder, JsonTable analysis, and the existing Generate execution path, including path validation, missing-versus-null handling, ANSI cast-mode capture, and parser cleanup. I also checked the Jackson 2.22 source contract for token traversal and structure copying; that contract exposes the floating-point exactness issue reported below.

[HyukjinKwon]

0 blocking, 1 non-blocking, 0 nits.
Solid, well-tested feature; one non-blocking per-row efficiency cleanup in projectRow before it compounds on wide projections.

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:553: projectRow does O(n^2) positional List access per row over the column count -- see inline

Verification

Traced the new grammar production through AstBuilder.visitJsonTableRelation into the JsonTable generator and the existing Generate path; checked path validation (INVALID_JSON_TABLE_PATH), missing-vs-null handling, ANSI cast-mode capture at plan construction, the Jackson parser ownership/cleanup lifecycle (transfer to the streaming iterator; finally close otherwise), and lateral-join activation. Confirmed the projectRow finding is distinct from the already-fixed whole-row reparsing concern (thread r3675017828): the trie fixed per-column reparsing, but the projection loop still does positional List access.

[cloud-fan]

5 addressed, 0 remaining, 2 new. (0 newly introduced, 2 late catches, 0 previously raised.)
1 blocking, 0 non-blocking, 1 nit.
The prior performance and documentation concerns are addressed; one identifier-contract defect should be fixed before merge, plus one comment nit.

Design / architecture (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:3273: JSON_TABLE duplicate-name validation ignores Spark's case-sensitive resolver setting and rejects identifiers that should remain distinct. -- see inline

Nits: 1 minor item (see inline comments).

Verification

I traced the new grammar through AstBuilder.visitJsonTableRelation, JsonTable analysis/evaluation, and the existing Generate execution path. I checked path validation, missing/null/EXISTS states, ANSI cast capture, parser ownership and early termination, keyword metadata, and the focused end-to-end tests. I also compared duplicate-name normalization with Spark's existing caseSensitiveAnalysis parser behavior.

[cloud-fan]

2 addressed, 0 remaining, 0 new.
0 blocking, 0 non-blocking, 0 nits.
The prior concerns are addressed, and the current parser, generator, streaming evaluator, documentation, and test portfolio are coherent.

Verification

I traced the dedicated grammar through AstBuilder into the JsonTable generator and the existing Generate path. I checked row and column path validation, missing-versus-JSON-null handling, result schema nullability, ANSI and case-sensitivity branches, Jackson token positioning and parser ownership, early iterator termination, keyword metadata, and the focused end-to-end tests. The four selected scanners found no correctness, link, or prose defects; their two local efficiency observations and four wording observations are below the review bar and recorded in the audit.

[cloud-fan]

Merge Summary:

• merged into master ce0578d
• merged into branch-4.x 3ef4617
• merged into branch-4.3 1387595

Posted by merge_spark_pr.py