Advanced Capabilities

Advanced Capabilities

Great Generator is more than fake values. It can generate realistic enterprise-style systems, then shape those systems for data-quality benchmarks, temporal history demos, analytics modeling, and lakehouse architecture examples.

Included in 0.1.1

Capability	API	Use case
Labeled anomaly ground truth	generate_domain(..., return_labels=True)	Data-quality benchmarks and QA validation
SCD2 temporal history	generate_domain(..., history="scd2"), generate_history(...)	dbt snapshots, Delta merge, time travel demos
CLI	great-generator ...	Shell and demo usage without writing Python
Dataset recipes	generate_from_recipe(path), great-generator run recipe.yaml	Reproducible datasets for demos, labs, and papers
Dimensional models	generate_dimensional_model(...)	Facts and dimensions for BI and analytics engineering
Data Vault models	generate_data_vault_model(...)	Hubs, links, and satellites for lakehouse modeling

Labeled anomaly ground truth

from great_generator import generate_domain

data = generate_domain(
    "ecommerce",
    scale="small",
    anomalies={
        "null_rate": 0.02,
        "duplicate_rate": 0.01,
        "orphan_fk_rate": 0.001,
        "invalid_status_rate": 0.005,
    },
    return_labels=True,
)

labels = data["_anomaly_labels"]

The label table includes table name, row index, primary key, affected column, anomaly type, original value, and corrupted value.

SCD2 history

from great_generator import generate_domain, generate_history

banking = generate_domain("banking", scale="small", history="scd2")
customers_history = banking["customers_history"]

single_table_history = generate_history("ecommerce", table="customers")

History output includes effective_from, effective_to, and is_current. Great Generator keeps at most one current row per natural key and avoids overlapping intervals.

CLI

great-generator list-domains
great-generator describe banking
great-generator gen banking --scale medium --out ./data/banking --format parquet
great-generator run banking_recipe.yaml

Recipes

Recipes make generated data reproducible and easy to cite.

kind: domain
domain: banking
engine: pandas
scale: small
realism: realistic
anomalies:
  null_rate: 0.01
  duplicate_rate: 0.005
output:
  path: ./generated/banking
  format: parquet

Supported recipe formats are JSON, TOML, and simple YAML.

Modeling outputs

For analytics modeling and architecture demos, see Dimensional and Data Vault Modeling.

Design principle

Advanced features should stay developer-first, deterministic when a seed is supplied, offline by default, and lightweight in the core package.