Feature request: infer field names in json_tuple

[jxu]

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.json_tuple.html

Currently, if you want new columns matching the field names, you have to specify the field names you want twice: first for extracting the fields and second for aliasing them from the default "c0", "c1", etc.

json_tuple(df.jstring, 'f1', 'f2').alias("f1", "f2")

You can put the field names into a tuple and unpack them with *, but it would be a nice convenience if the column names took on what the fields were automatically. Also if it inferred the schema with JSON data types. Maybe this is already handled by schema_of_json.

[dkranchii]

Thanks for the suggestion. Spark has a tool that does both of the things you're asking for — it's just from_json (with schema_of_json for inference), not json_tuple.

Use from_json for named, typed columns

from pyspark.sql.functions import from_json, schema_of_json, col, lit

# Option 1: explicit schema — named columns, real types
df.select(from_json("jstring", "f1 STRING, f2 INT").alias("j")) \
  .select("j.f1", "j.f2")

# Option 2: inferred schema from a sample row — also named, also typed
sample = df.select("jstring").first()[0]
df.select(from_json("jstring", schema_of_json(lit(sample))).alias("j")) \
  .select("j.*")    # expands to f1, f2, ... with inferred types

[jxu]

yeah, it's a little inconvenient to have to infer the schema when most JSON parsers can map JSON types to native types. Nice tip.