ETL and Pipeline Testing

ETL and Pipeline Testing

Schema-driven generation fits pipeline tests because the generated input starts from the expected contract.

from great_generator import generate_from_schema

source = generate_from_schema(
    "transaction_id string, account_id string, transaction_amount double, transaction_date date, transaction_type string",
    rows=10000,
    domain="banking",
)

source.to_parquet("landing/transactions.parquet", index=False)

Tests to build

• schema and type compatibility
• required field handling
• date parsing and ordering
• amount and quantity transformations
• join coverage and key assumptions
• incremental loads and idempotency
• reject paths and quarantine tables
• clean-data validation before anomaly scenarios

For multiple related custom tables, use generate_relational. For CDC records, use generate_cdc. For ready-made anomaly scenarios, use domain generation with explicit anomaly rates.