[SPARK-58307][SQL][PYTHON] Add json_typeof function

[SreeramaYeshwanthGowd]

What changes were proposed in this pull request?

Add a built in json_typeof(json) scalar function that returns the type of the outermost value in a JSON string, as one of object, array, string, number, boolean, or null. It returns SQL NULL when the input is not a valid JSON string or is an empty string.

API surface added:

• SQL: json_typeof(json)
• Scala DataFrame: functions.json_typeof(col)
• PySpark, classic and Spark Connect: pyspark.sql.functions.json_typeof(col)

Implementation notes:

• Modeled on the existing json_object_keys function: a RuntimeReplaceable expression whose replacement is a StaticInvoke into JsonExpressionUtils, using the same Jackson parser (CreateJacksonParser / SharedFactory) that already backs json_array_length and json_object_keys. No new dependency.
• Like those functions, malformed input returns NULL: json_typeof reads the outermost value and consumes it, so a structurally invalid JSON value returns NULL (consistent with json_object_keys).
• Both integer and floating point JSON numbers map to number, matching PostgreSQL and BigQuery.

Why are the changes needed?

Spark can already ask a JSON string for its array length (json_array_length) and its object keys (json_object_keys), but not for the type of its outermost value. json_typeof completes that family. It is a lightweight top level type probe: unlike schema_of_json, which does full recursive schema inference and returns a DDL string such as ARRAY<BIGINT>, json_typeof returns a single simple type token. It is provided by PostgreSQL (json_typeof and jsonb_typeof) and BigQuery (JSON_TYPE), so it also improves parity with the engines Spark users migrate from.

Does this PR introduce any user-facing change?

Yes. It adds a new built in SQL function json_typeof and the corresponding Scala and PySpark DataFrame API entries. No existing behavior changes.

Example:

spark-sql> SELECT json_typeof('{"a": 1}');
object
spark-sql> SELECT json_typeof('[1, 2, 3]');
array
spark-sql> SELECT json_typeof('123');
number
spark-sql> SELECT json_typeof('not json');
NULL

How was this patch tested?

Added catalyst unit tests in JsonExpressionsSuite covering each JSON type (object, array, string, number, boolean, null), invalid and empty inputs returning null, and null propagation, and a DataFrame API test in JsonFunctionsSuite. Added a PySpark doctest and regenerated sql-expression-schema.md.

Was this patch authored or co-authored using generative AI tooling? No

[uros-b]

PlanGenerationTestSuite.scala - no Spark Connect Scala functionTest("json_typeof") and no function_json_typeof.{json,proto.bin,explain} query-test goldens are added. All seven sibling JSON functions on master (get_json_object, json_tuple, from_json, schema_of_json, to_json, json_array_length, json_object_keys) carry a functionTest plus goldens (verified against upstream/master); this new functions.scala entry adds none. It is not CI-blocking; the suite lists functions explicitly rather than auto-enumerating, and the Scala Connect client works because it reuses the shared sql/api functions.scala; but it is a uniform convention gap on a new public API path that committers typically request. Add the functionTest and regenerate the goldens.

[SreeramaYeshwanthGowd]

@uros-b Thanks for the review. Added the functionTest and regenerated the json, proto.bin, and explain goldens for json_typeof.

[SreeramaYeshwanthGowd]

@cloud-fan Would you have a moment to review this when you get a chance? Thank you!

[SreeramaYeshwanthGowd]

@uros-b Updated since to 4.4.0, since branch-4.3 is now cut.

[cloud-fan]

1 blocking, 1 non-blocking, 0 nits.
The API integration is coherent, but the invalid-input contract needs one correctness fix; the hot path also has an avoidable allocation.

Correctness (1)

• sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionUtils.java:102: Trailing content after a valid first JSON value is accepted, contradicting the public invalid-input contract. -- see inline

Suggestions (1)

• sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionUtils.java:103: The per-row helper repeatedly allocates encoded wrappers for a fixed six-value result set. -- see inline

Verification

I traced the SQL/Python/Scala entry points through JsonTypeof.replacement to JsonExpressionUtils.jsonTypeof and compared the structure with json_object_keys. The helper calls skipChildren() and returns immediately, so it never advances once more to establish end-of-input; consequently, two adjacent JSON values can be accepted as the type of the first value. I also verified that the previously requested Spark Connect function test and generated artifacts are present in the current tree.

[cloud-fan]

Please verify that the parser reached end-of-input before returning the type. An input containing two JSON values such as 123 true is not a valid single JSON document, but this path returns number instead of the documented null; call nextToken() after consuming the value, require EOF, and add a trailing-content test.

[cloud-fan]

Please cache the six possible UTF8String results and return those constants from the switch. This currently re-encodes a fixed literal and allocates a byte array plus wrapper for every evaluated row.

[SreeramaYeshwanthGowd]

@cloud-fan Thanks, fixed. json_typeof now checks for EOF after the value so trailing content is rejected, added a test for that, and cached the six UTF8String results to avoid the per-row allocation.

[cloud-fan]

2 addressed, 0 remaining, 0 new.
0 blocking, 0 non-blocking, 0 nits.
The current revision is clean; both concerns from the prior review are addressed.

Verification

I traced the SQL registry and Scala/Python entry points through JsonTypeof.replacement to JsonExpressionUtils.jsonTypeof. The helper now consumes the selected value, advances once more to require EOF, and returns cached UTF8String constants. The focused 123 true test verifies the former trailing-content failure, while the Connect function test and generated artifacts remain present.

[SreeramaYeshwanthGowd]

@cloud-fan CI is green. Thans for the review.

[cloud-fan]

Merge Summary:

• merged into master b24247b
• merged into branch-4.x 8faa524

Posted by merge_spark_pr.py