PySpark StructType Examples

PySpark StructType Examples

Status

Supported for common scalar fields. Nested arrays, maps, and structs have limited generation support.

What it is and when to use it

StructType is Spark's programmatic schema. Use it in Databricks, Microsoft Fabric, Synapse Spark, EMR, Dataproc, or local PySpark tests.

Example

from pyspark.sql import types as T
from great_generator import generate_from_schema

schema = T.StructType(
    [
        T.StructField("customer_id", T.StringType(), False),
        T.StructField("customer_name", T.StringType(), True),
        T.StructField("age", T.IntegerType(), True),
        T.StructField("email", T.StringType(), True),
        T.StructField("balance", T.DoubleType(), True),
        T.StructField("created_at", T.TimestampType(), True),
    ]
)

spark_df = generate_from_schema(schema, rows=1000, engine="spark")
spark_df.show(5, truncate=False)

Sample output preview

+-----------+-------------+---+-----------------------+-------+-------------------+
|customer_id|customer_name|age|email                  |balance|created_at         |
+-----------+-------------+---+-----------------------+-------+-------------------+
|CUST000001 |Ava Johnson  |34 |ava.johnson@example.com|4812.37|2024-08-19 10:00:00|
+-----------+-------------+---+-----------------------+-------+-------------------+

Write output

spark_df.write.mode("overwrite").parquet("/data/synthetic/customers")
spark_df.write.format("delta").mode("overwrite").save("/mnt/delta/customers")

Notes and limitations

An active SparkSession is discovered automatically in many notebooks. Pass spark=spark when discovery is unavailable. This API currently generates values locally before creating the Spark DataFrame, so row count is driver-memory sensitive. Spark-native arbitrary-schema generation is planned.