Lakehouse and Warehouse Examples

Lakehouse and Warehouse Examples

Spark schema generation

spark_df = generate_from_schema(
    "customer_id string, customer_name string, balance double, created_at timestamp",
    rows=1000,
    engine="spark",
)

Delta and Databricks

spark_df.write.format("delta").mode("overwrite").save("/mnt/delta/customers")
spark_df.write.format("delta").mode("overwrite").saveAsTable("dev.synthetic_customers")

Microsoft Fabric Lakehouse

spark_df.write.format("delta").mode("overwrite").save("Tables/synthetic_customers")

Snowflake, PostgreSQL, Azure SQL, or SQL Server

import os
from sqlalchemy import create_engine

engine = create_engine(os.environ["DATABASE_SQLALCHEMY_URL"])
pandas_df.to_sql("synthetic_customers", engine, if_exists="replace", index=False)

Cloud storage

Use the URI and connector configured by the platform, such as s3a://, abfss://, gs://, DBFS, or a Unity Catalog volume path.

Great Generator returns the DataFrame. The platform remains responsible for credentials, permissions, drivers, catalogs, and storage configuration.