Architecture

Architecture

Great Generator keeps schema understanding, planning artifacts, and data generation separate.

schema -> advisor -> plan -> generation -> data

Core layers

• Schema normalization turns mappings, compact DDL, Pandas schemas, PySpark schemas, and TableSchema objects into common metadata.
• Semantic generation maps column names and data types to deterministic value generators.
• Relational generation creates parent and child tables with valid keys.
• Export helpers write generated data to local paths, cloud paths, lakehouse formats, or user-selected destinations.

Advisor layer

The advisor layer is optional and runs before generation. It can create:

• GenerationPlan
• ColumnTags
• RealismReport

Advisors never create row data in v1. They produce JSON artifacts that users can inspect, edit, save, and review.

Deterministic generation boundary

Generation does not call an advisor. It only consumes a plan when you pass plan=....

from great_generator import generate_from_schema, infer_generation_plan

schema = "customer_id int, customer_name string"
plan = infer_generation_plan(schema)
df = generate_from_schema(schema, rows=1000, plan=plan)

This boundary keeps test runs reproducible. The same schema, plan, seed, and arguments produce the same data.

Query-aware generation flow

flowchart LR
    A[Schema or relational schemas] --> B[Existing generation logic]
    C[Optional required_values] --> D[Query-aware planner]
    E[Optional partition_by] --> D
    F[Optional target_selectivity] --> D
    G[Optional ensure_join_coverage] --> D
    D --> B
    B --> H[Synthetic data]
    H --> I[Optional coverage validation report]

Loading

Query-aware generation is an optional layer. It does not replace the existing generation engine.

When no query-aware arguments are provided, Great Generator uses the same generation path as before.

When query-aware arguments are provided, the planner applies constraints only to the named columns, tables, partitions, or relationships. All other generated values continue to use the existing generation logic.