Data Quality and Edge Cases

Data Quality and Edge Cases

Good synthetic test data should be fake, schema-aligned, type-correct, logically consistent, and easy to inspect.

Clean defaults

Realistic schema generation currently handles:

• aliases such as cust_name, emp_name, email_id, and txn_amt
• age ranges and age/date-of-birth alignment
• non-future birth and historical dates
• updated_at >= created_at
• end_date >= start_date
• delivery and payment dates after order dates
• status-aware nulls for pending payments and orders
• unique non-null ID-like fields
• positive quantities
• total amount calculations where quantity and price fields are recognized

Inspect inference

from great_generator import explain_generation_plan

plan = explain_generation_plan(
    {"cust_nm": "string", "txn_amt": "double", "created_ts": "timestamp"}
)
print(plan)

Validate output

from great_generator import validate_generated_data

report = validate_generated_data(df, schema=schema, rules=custom_rules)
assert report["passed"], report["errors"]

Boundaries

Semantic inference cannot know organization-specific meanings from every field name. Add custom rules for abbreviations or business logic that is local to your system. Spark validation currently avoids collecting distributed results and returns a note instead of a local report.