CDC Simulation

CDC Simulation

CDC means Change Data Capture. CDC records describe changes to source tables as event streams, often with insert, update, and delete operations.

Synthetic CDC data is useful for:

• lakehouse ingestion tests
• merge/upsert logic
• deduplication tests
• event ordering validation
• late-arriving data handling
• incremental ETL demos

Example

from great_generator import generate_cdc

cdc = generate_cdc(
    "banking",
    table="customers",
    rows=10000,
    operations=["insert", "update", "delete"],
    late_arrival_rate=0.02,
    duplicate_rate=0.005,
    seed=42,
)

CDC output includes:

• operation type
• before values where applicable
• after values where applicable
• event timestamp
• ingestion timestamp
• sequence number
• source system
• late-arriving indicator
• duplicate indicator

CDC data is especially useful for Databricks, Delta Lake, Spark Structured Streaming demos, and table merge validation.