Benchmarking

Benchmarking

Great Generator includes a lightweight local pandas benchmark script in the repository:

python benchmarks/benchmark_pandas_generation.py

Planned benchmark categories:

• small dataset generation
• medium dataset generation
• large dataset generation
• Pandas mode
• Spark mode
• CSV export
• Parquet export
• Delta export

Benchmark results depend on:

• domain pack
• scale profile
• Python version
• pandas/numpy versions
• machine memory and CPU
• Spark cluster size
• partition count
• storage system

Spark and Delta benchmarks should be measured in the target environment because cluster size and storage throughput dominate performance.