Writing Data to Files and Databases

Writing Data to Files and Databases

Great Generator returns DataFrames so storage remains an explicit user decision.

Pandas files

df.to_csv("customers.csv", index=False)
df.to_json("customers.json", orient="records", lines=True, date_format="iso")
df.to_parquet("customers.parquet", index=False)

Spark Parquet and Delta

spark_df.write.mode("overwrite").parquet("/data/synthetic/customers")
spark_df.write.format("delta").mode("overwrite").save("/mnt/delta/customers")

Databricks table

spark_df.write.format("delta").mode("overwrite").saveAsTable(
    "dev.synthetic_customers"
)

Microsoft Fabric Lakehouse

spark_df.write.format("delta").mode("overwrite").save(
    "Tables/synthetic_customers"
)

SQL databases with Pandas and SQLAlchemy

import os
from sqlalchemy import create_engine

engine = create_engine(os.environ["DATABASE_SQLALCHEMY_URL"])
df.to_sql("synthetic_customers", engine, if_exists="replace", index=False)

PySpark or Databricks to Snowflake

Use Snowflake's Spark connector for Spark DataFrame transfers:

def secret(key: str) -> str:
    return dbutils.secrets.get(scope="great-generator", key=key)


sf_options = {
    "sfURL": secret("snowflake-url"),
    "sfUser": secret("snowflake-user"),
    "sfPassword": secret("snowflake-password"),
    "sfDatabase": secret("snowflake-database"),
    "sfSchema": secret("snowflake-schema"),
    "sfWarehouse": secret("snowflake-warehouse"),
    "sfRole": secret("snowflake-role"),
}

(
    spark_df.write.format("net.snowflake.spark.snowflake")
    .options(**sf_options)
    .option("dbtable", "SYNTHETIC_CUSTOMERS")
    .mode("overwrite")
    .save()
)

Install a connector compatible with the Spark and Scala runtime when it is not already bundled.

PySpark or Databricks to Azure SQL with JDBC

def secret(key: str) -> str:
    return dbutils.secrets.get(scope="great-generator", key=key)


server = secret("azure-sql-server")
database = secret("azure-sql-database")
jdbc_url = (
    f"jdbc:sqlserver://{server}:1433;"
    f"databaseName={database};"
    "encrypt=true;trustServerCertificate=false;"
    "hostNameInCertificate=*.database.windows.net;loginTimeout=30;"
)

(
    spark_df.coalesce(4)
    .write.format("jdbc")
    .mode("overwrite")
    .option("url", jdbc_url)
    .option("dbtable", "dbo.synthetic_customers")
    .option("user", secret("azure-sql-user"))
    .option("password", secret("azure-sql-password"))
    .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
    .option("batchsize", "1000")
    .save()
)

Reduce partitions when needed to avoid opening too many concurrent database connections. The Microsoft SQL Server JDBC driver must be present on the Spark runtime.

The URL and installed driver determine whether the destination is Snowflake, Azure SQL, SQL Server, PostgreSQL, or another SQLAlchemy-supported database.

SQLite needs no external server:

engine = create_engine("sqlite:///synthetic_data.db")
df.to_sql("synthetic_customers", engine, if_exists="replace", index=False)

Cloud object storage

df.to_parquet("s3://bucket/synthetic/customers.parquet")
df.to_parquet("gs://bucket/synthetic/customers.parquet")
df.to_parquet("abfss://container@account.dfs.core.windows.net/synthetic/customers.parquet")

Spark commonly uses s3a://, gs://, and abfss:// paths configured by the runtime.

Security and platform configuration

Install the required connector and configure authentication separately. Use managed identity, IAM roles, workload identity, environment variables, or a secret manager. Do not hardcode credentials. Great Generator does not configure cloud permissions, database drivers, catalogs, or external locations.