FAQ

FAQ

Is this a production-data anonymization tool?

No. Great Generator creates synthetic data and does not transform production records.

What should industry users start with?

Start with generate_from_schema when you already know the expected table structure.

Which schema inputs work today?

Plain mappings, Pandas dtype mappings and DataFrames, compact DDL, PySpark StructType and DataFrames, TableSchema, and DomainSchema. See the support matrix.

Does full SQL CREATE TABLE work?

Yes, for the documented SQL DDL subset. Use parse_ddl(...) for SQL CREATE TABLE contracts, or use compact column DDL and Python mappings for simpler schemas.

Are JSON Schema and YAML schema files supported?

Not as generate_from_schema inputs. They are planned. JSON, TOML, and simple YAML dataset recipes are a separate supported feature.

Can it return Spark DataFrames?

Yes, when Spark context is available or engine="spark" is selected. Single-table arbitrary-schema generation currently creates values locally before creating the Spark DataFrame.

Can I write to CSV, Parquet, Delta, databases, S3, ADLS, or GCS?

Yes, through normal Pandas or Spark writers and separately installed connectors. Great Generator does not configure credentials or platform permissions.

Is a seed mandatory?

No. Use a seed only when repeatable output matters for a test, benchmark, or experiment.

Can it generate millions of rows?

Row-count capability depends on engine, schema, memory, compute, and storage. Use chunking or Spark-native domain generation for very large workloads and benchmark in your own environment.

When should I use domain packs?

Use generate_domain for ready-made demos, learning, tutorials, and examples where you do not already have a schema.

Does Great Generator send my schema to an LLM by default?

No. The default is advisor="none". It makes no model calls, reads no API keys, and sends nothing over the network.

Can I use the advisor layer offline?

Yes. Use Ollama with a local model. llama.cpp is reserved as a clear stub for later implementation.

Is generation reproducible when using an advisor?

Yes. Advisors produce artifacts such as GenerationPlan. Generation consumes the plan. The same plan, seed, schema, and arguments produce the same output.

Does query-aware generation change default behavior?

No. Query-aware generation is optional. Existing generation behavior is unchanged unless you provide required_values, partition_by, target_selectivity, ensure_join_coverage, or query_profile.

Does Great Generator parse my SQL queries?

Not in this version. You provide the required values, partition dates, and selectivity targets manually. SQL parsing may be added later as an optional feature.

Does query-aware generation reproduce production performance?

No. It helps synthetic data match query values, partition dates, and join paths. It does not guarantee identical production performance because file layout, table statistics, clustering, caching, concurrency, warehouse size, and query engine configuration also affect runtime.

What is the difference between required values and filters?

required_values means the generated data must contain those values. It does not mean Great Generator is executing a SQL filter. The term is used because it is easier to understand for test-data generation.

Can required values include other random values too?

Yes. Required values make sure specific values appear. Other generated values may also appear unless the user explicitly configures otherwise.

Can I use query-aware generation with relational data?

Yes. Use table-qualified names such as dim_member.region or dim_product.product_type. Use ensure_join_coverage=True when you need matching fact rows for required dimension values.