Spark Engine

Spark Engine

Use the Spark engine for distributed DataFrames, lakehouse exports, Databricks notebooks, large datasets, and performance demos.

data = generate_domain(
    "banking",
    engine="spark",
    scale="large",
    realism="realistic",
)

transactions = data["transactions"]
transactions.printSchema()

In active Spark notebooks, Great Generator can usually infer the Spark session. In scripts, pass spark=spark explicitly if needed.

Spark output is a dictionary of PySpark DataFrames.

data["transactions"].write.mode("overwrite").parquet("/tmp/banking/transactions")

Spark realistic values are generated with deterministic Spark-native expressions and curated reference values, avoiding driver-side collection for large tables.