Lower Environment Testing

Lower Environment Testing

Goal

Build test datasets from the same schema expected by a lower environment without copying production records.

from great_generator import generate_from_schema

schema = {
    "customer_id": "string",
    "customer_name": "string",
    "email": "string",
    "date_of_birth": "date",
    "age": "int",
    "created_at": "timestamp",
    "updated_at": "timestamp",
    "account_status": "string",
}

df, report = generate_from_schema(
    schema,
    rows=5000,
    validate=True,
    return_report=True,
)

assert report["passed"], report["errors"]
df.to_parquet("lower_env_customers.parquet", index=False)

Recommended workflow

1. Export or express only the target schema, not production records.
2. Add explicit rules for business-specific fields.
3. Inspect explain_generation_plan for ambiguous columns.
4. Generate and validate a small sample.
5. Scale within the memory and compute limits of the chosen engine.
6. Write through the destination's normal secure connector.

Synthetic data reduces reliance on copied records but does not itself guarantee compliance. Follow organizational policy.