[SPARK-51975][SQL] Add variant_from_arrays and variant_from_entries

[SreeramaYeshwanthGowd]

What changes were proposed in this pull request?

Add two built in scalar SQL functions that construct a Variant object directly from collections:

• variant_from_arrays(keys, values) builds a Variant object from an array of string keys and an array of values.
• variant_from_entries(entries) builds a Variant object from an array of key/value struct entries (a two field struct per entry).

> SELECT variant_from_arrays(array('a', 'b'), array(1, 2));
 {"a":1,"b":2}
> SELECT variant_from_entries(array(struct('a', 1), struct('b', 2)));
 {"a":1,"b":2}

API surface added:

• SQL: variant_from_arrays, variant_from_entries
• Scala DataFrame: functions.variant_from_arrays(keys, values), functions.variant_from_entries(entries)
• PySpark, classic and Spark Connect: pyspark.sql.functions.variant_from_arrays / variant_from_entries

Implementation notes:

• Both are native expressions in variantExpressions.scala next to to_variant_object, delegating (eval and codegen) to two new VariantExpressionEvalUtils helpers that build the object in a single pass with VariantBuilder (addKey / finishWritingObject) and reuse the existing buildVariant for each value. This avoids materializing an intermediate MapType, which is the point of the ticket.
• Semantics, chosen to match the existing to_variant_object and map_from_* precedents:
  • Object keys must be non-null strings. A non-string key type is rejected at analysis (as to_variant_object does for non-string-keyed maps, it is not implicitly cast); a null key raises NULL_MAP_KEY.
  • Duplicate keys raise VARIANT_DUPLICATE_KEY, matching to_variant_object (Variant objects cannot hold duplicate keys). This is stricter than map_from_* under the non default spark.sql.mapKeyDedupPolicy=LAST_WIN.
  • Null values are kept as Variant null. A null array argument yields NULL, and for variant_from_entries a null entry yields NULL (as map_from_entries does).
  • For variant_from_arrays, a keys/values length mismatch reuses the same error map_from_arrays raises.

Why are the changes needed?

Today, creating a Variant object from arrays or entries requires a two step expression: to_variant_object(map_from_arrays(...)) or to_variant_object(map_from_entries(...)). That materializes an intermediate MapType only to immediately convert it. variant_from_arrays / variant_from_entries build the Variant object directly, reducing allocation and CPU, which matters for large maps and high volume workloads. They also mirror Spark's own map_from_arrays / map_from_entries naming, so the Variant surface stays consistent.

Does this PR introduce any user-facing change?

Yes. It adds two new built in SQL functions and their Scala and PySpark DataFrame API entries. No existing behavior changes.

How was this patch tested?

• Catalyst unit tests in VariantExpressionSuite covering object construction, key sorting, empty input, null values, nested values, null array / null entry inputs returning null, and the null key / duplicate key / length mismatch errors.
• An end to end test in VariantEndToEndSuite exercising the DataFrame API under both CODEGEN_ONLY and NO_CODEGEN over a non foldable relation.
• SQL golden tests in variant/variant-from-arrays-entries.sql, covering each JSON type, null handling, and the null key / duplicate key / length mismatch / non-string key / unsupported value type errors.
• PySpark doctests and a test_functions.py test, and regenerated sql-expression-schema.md.

Was this patch authored or co-authored using generative AI tooling? No

[uros-b]

Please note a minor e2e test coverage gap: the SQL golden input documents -- Duplicate keys are rejected for variant_from_arrays (select variant_from_arrays(array('a', 'a'), array(1, 2))) but has no analogous duplicate-key case for variant_from_entries; the entries cases stop at null-key / wrong-type / uncastable-value.

[SreeramaYeshwanthGowd]

@uros-b Thanks, Added a duplicate-key case for variant_from_entries and regenerated the golden output so it mirrors the variant_from_arrays case.

[HyukjinKwon]

Can we have arguments docs?

[SreeramaYeshwanthGowd]

@HyukjinKwon, added arguments docs for both functions. Thank you!

[SreeramaYeshwanthGowd]

@cloud-fan Would you have a moment to review this when you get a chance? Thank you!

[cloud-fan]

0 blocking, 1 non-blocking, 0 nits.
The feature is coherent and well covered; one small hot-path allocation cleanup is worth making before completion.

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:297: These helpers execute once per input row, so allocating Scala Range/foreach machinery adds row-cardinality overhead. Please cache numElements() and use index-based while loops in both helpers, matching the surrounding Catalyst evaluation style. -- see inline

Verification

I traced both public entry forms through FunctionRegistry, Catalyst type checking, interpreted and generated evaluation, and the shared VariantBuilder conversion. The new paths preserve the established map/to_variant_object behavior for valid inputs, nulls, unsupported types, and errors, with the intentional stricter duplicate-key behavior required by Variant objects.

[cloud-fan]

Please cache numElements() and use index-based while loops here and in the matching entries loop at VariantExpressionEvalUtils.scala:322. These helpers execute once per input row, so 0 until ... adds avoidable Range/foreach allocation in a Catalyst hot path.

[SreeramaYeshwanthGowd]

@cloud-fan Thanks, cached numElements() and switched both loops (variantFromArrays and the matching variantFromEntries loop at line 322) to index-based while loops.

[cloud-fan]

1 addressed, 0 remaining, 1 new to this AI review. (0 newly introduced, 1 late catch, 0 previously raised.)
0 blocking, 1 non-blocking, 0 nits.
The prior hot-loop allocation issue is fixed; one additional single-pass efficiency cleanup remains.

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:323: When the entries element type permits nulls, every non-null row scans the full array once here and then scans it again in variantFromEntries. Fold the null-entry check into the construction loop so this row-cardinality hot path remains single-pass. -- see inline

Verification

I traced both public entry forms through FunctionRegistry, Catalyst type checking and nullability, interpreted and generated evaluation, and VariantBuilder object finalization. The current head preserves the documented null, duplicate-key, length, and unsupported-type behavior, and the previous Range/foreach allocation concern is resolved. For nullable entry arrays, however, nullSafeEval still pre-scans the full array before variantFromEntries performs its construction scan.

[cloud-fan]

Fold this null-entry check into variantFromEntries and propagate a null helper result into the expression's null flag. With containsNull=true, this loop walks every non-null row's full array, then the helper immediately walks it again to build the object, leaving this hot path with two cardinality-scaled traversals.

[SreeramaYeshwanthGowd]

@cloud-fan Thanks, folded the null-entry check into variantFromEntries's existing loop. It now returns null on a null entry, and both nullSafeEval and codegen just propagate that result instead of pre-scanning, so the whole path is single-pass again.

[cloud-fan]

1 addressed, 0 remaining, 1 new to this AI review. (1 newly introduced, 0 late catches, 0 previously raised.)
1 blocking, 0 non-blocking, 0 nits.
The public surface and implementation are coherent, but the single-pass nullable-entry change alters error ordering for values before a later null entry.

Correctness (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:328: Preserve null-entry dominance before converting preceding values; the new single-pass loop can throw before reaching a later null entry. -- see inline

Verification

I traced both public entry forms through registration, type checking, interpreted/code-generated evaluation, recursive value conversion, and VariantBuilder object finalization. The new single-pass entries loop converts each value before inspecting later entries, while the established map_from_entries analogue pre-scans nullable entries before processing values; this makes a later null no longer dominate an earlier conversion failure.

[cloud-fan]

Please preserve the documented null-entry dominance before converting any values. In this single-pass loop, an earlier nested value can fail during buildVariant (for example, a struct with duplicate field names) before a later null entry is reached, so the call throws instead of returning null as map_from_entries does; retaining the pre-scan and adding a mixed failing-value/null test would keep the contract intact.

[SreeramaYeshwanthGowd]

@cloud-fan Thanks, good catch. Restored the null-only pre-scan in variantFromEntries so a null entry always dominates an earlier conversion failure, matching map_from_entries, and added a test for that case.

[cloud-fan]

1 addressed, 0 remaining, 0 new to this AI review.
0 blocking, 0 non-blocking, 0 nits.
The previously reported null-entry error-ordering regression is fixed, and the current public API, implementation, generated-code path, and test portfolio are coherent.

Verification

I traced both public entry forms from FunctionRegistry and language wrappers through Catalyst type checking, interpreted evaluation, generated evaluation, recursive Variant conversion, and object finalization. I also compared nullable-entry behavior with map_from_entries: both pre-scan for null entries before processing values, and the added focused test covers a conversion failure before a later null entry. The selected prose, contract, and local-efficiency scans reported no current findings.