Databricks and Delta Lake Usage

Databricks and Delta Lake Usage

Great Generator works well in Databricks notebooks because it can return Spark DataFrames and write Parquet or Delta outputs.

Generate Spark data

from great_generator import generate_domain

data = generate_domain(
    "banking",
    engine="spark",
    scale="large",
    realism="realistic",
)

Write Parquet

data["transactions"].write.mode("overwrite").partitionBy("event_date").parquet(
    "/Volumes/demo/synthetic/banking/transactions"
)

Write Delta

data["transactions"].write.format("delta").mode("overwrite").partitionBy("event_date").save(
    "/Volumes/demo/synthetic/banking_delta/transactions"
)

Convenience export

generate_domain(
    "banking",
    engine="spark",
    scale="large",
    realism="realistic",
    output_path="/Volumes/demo/synthetic/banking_delta",
    output_format="delta",
    partition_by=["event_date"],
)

Notes

• Databricks Runtime usually includes Delta Lake support.
• Use Unity Catalog volumes or governed external locations when possible.
• Great Generator does not create credentials, storage accounts, IAM roles, or catalogs.
• Avoid using real company/workplace data in demos.