v4.0.0

v4.0.0 is the first stable release with TPC-DS support.

• tpcgen-cli is the new CLI for both TPC-H (tpcgen-cli tpch) and TPC-DS (tpcgen-cli tpcds).
• tpchgen-cli is kept for backwards compatibility. It uses the same code as tpcgen-cli tpch.

Run it with uv, no install needed:

uvx tpcgen-cli tpch parquet -s 10 --output-dir=/tmp/tpch
uvx tpcgen-cli tpcds parquet -s 10 --output-dir=/tmp/tpcds

Also available via pip install tpcgen-cli, cargo install tpcgen-cli and brew install tpcgen-cli.

Packages

All packages now share one version, 4.0.0 (#546).

Package	Registry	Description
tpcgen-cli	PyPI	New. CLI for TPC-H and TPC-DS
tpcgen-cli	crates.io	New. CLI for TPC-H and TPC-DS
tpchgen-cli	PyPI	TPC-H CLI, same as tpcgen-cli tpch
tpchgen-cli	crates.io	TPC-H CLI, same as tpcgen-cli tpch
tpcdsgen	crates.io	New. TPC-DS data generator library
tpcdsgen-arrow	crates.io	New. TPC-DS data as Arrow record batches
tpchgen	crates.io	TPC-H data generator library
tpchgen-arrow	crates.io	TPC-H data as Arrow record batches

Highlights

• TPC-DS generation in DAT, CSV and Parquet (#329, #331), generated in parallel (#360, #361, #463), with --parts/--part (#406) and --stdout (#482). Output matches the C dsdgen by default; use --compat trino to match Trino (#628).
• Parquet options: per-column encodings with --column-encoding (#405) and human-readable --row-group-bytes such as 128MB (#486).
• Parquet field IDs are written by default. Use --field-ids=false to turn them off (#509, #537).
• --overwrite replaces existing output files (#494).
• Progress bars show ETA and write throughput, and render more cleanly (#356, #386, #393, #399, #507).
• Logging: clearer per-table messages, a summary of sizes and throughput at the end of each run, and readable error messages (#501, #504, #506, #520).
• Input validation: invalid options are rejected before any output is created (#519), and TPC-H table names are case-insensitive (#426).
• Arrow 59 and 60 support: tpchgen-arrow and tpcdsgen-arrow use Arrow 60 by default, or Arrow 59 with the arrow_59 feature (#473).
• Minimum supported Rust version is declared as 1.88 for all crates (#583).
• schema_ref() on every tpchgen-arrow and tpcdsgen-arrow reader returns a table's Arrow schema without creating a generator (#427).
• New home: the project moved to datafusion-contrib/tpcgen-rs (#407).

Breaking changes

• --delimiter only accepts comma, pipe, tab and semicolon (#500).
• --parts 0 is rejected (#519).
• tpchgen-cli flags deprecated in 2.x are removed (#421). Use the tbl, csv or parquet subcommand instead of --format, and parquet --compression / parquet --row-group-bytes instead of --parquet-compression / --parquet-row-group-bytes.
• TPC-H --parts N --part K writes the nation and region tables only for part 1, so combining all parts no longer repeats them (#641).
• tpchgen-arrow: generators implement Arrow's RecordBatchReader instead of the crate's own RecordBatchIterator trait (#345).
• tpchgen-cli library: tpchgen_cli::TpchGenerator moved to tpcgen_cli::tpch_cli::TpchGenerator (#296).

All changes

• refactor: move TPCH CLI implementation into tpcgen by @alamb in #296
• Run clippy on all targets and features by @alamb in #315
• [tpcgen]: move TPCH CLI tests into tpcgen (unified cli) by @alamb in #313
• [TPC-DS]: remove tpcdsgen CLI binary by @alamb in #303
• [tpcdsgen]: remove chrono dependency from tpcdsgen by @alamb in #312
• refactor: Move some output configuration and writing from tpcdsgen by @alamb in #317
• rename tpcgen crate and binary to tpcgen-cli by @alamb in #314
• Make progress tracker API always available by @kevinjqliu in #323
• [tpcdsgen]: add validated SessionBuilder by @alamb in #318
• Fix release workflow triggers and permissions by @kevinjqliu in #322
• Improve python CLI quickstart documentation by @kevinjqliu in #321
• Add all new crates to the publishing workflow by @kevinjqliu in #326
• Simplify progress tracker plumbing by @kevinjqliu in #325
• [tpcdsgen]: move output options out of Session by @alamb in #319
• [tpcgen-cli]: split CLI integration tests by @alamb in #328
• [tpcgen-cli]: Integrate tpcds dat status output and logging by @alamb in #330
• [tpcgen-cli]: support TPC-DS parquet generation by @alamb in #329
• [tpcgen-cli]: support TPC-DS CSV generation by @kevinjqliu in #331
• Refactor progress tracker by @kevinjqliu in #327
• chore(deps): bump crate-ci/typos from 1.47.2 to 1.48.0 by @dependabot[bot] in #333
• Refactor TPC-DS output dispatch by @kevinjqliu in #332
• [tpcgen-cli] Enable TPC-DS parquet logging by @alamb in #340
• [tpcgen-cli] Respect TPC-DS parquet row group bytes by @alamb in #341
• Minor: Add hyperfine benchmarking instructions to contributing by @alamb in #343
• [tpcgen-cli] Remove dead code in TPCH parquet generation by @alamb in #344
• [tpcdsgen-arrow]: add source row range generation by @alamb in #346
• [tpcgen-cli]: extract shared parquet writer from tpch generation by @alamb in #342
• [arrow] Consolidate to use Arrow RecordBatchReader by @alamb in #345
• chore: remove generate chunks progress wrapper by @kevinjqliu in #347
• [tpcgen-cli] Propagate parquet reader errors by @alamb in #348
• [tpcgen-cli] Validate TPC-DS row group bytes with clap by @alamb in #349
• refactor: simplify TPC-DS inventory DAT generation by @kevinjqliu in #351
• Wire progress tracker with dat/csv/parquet by @kevinjqliu in #350
• [tpcgen-cli] Fix TPC-DS dat table selection by @alamb in #352
• refactor: share TPC-DS sales returns DAT generation by @kevinjqliu in #353
• feat: Per-table row count scaling cache by @clflushopt in #357
• feat: use fmt::Display buffering for Dat output by @clflushopt in #359
• [tpcgen-cli]: parallel TPC-DS parquet generation (10x faster) by @alamb in #360
• [tpcgen-cli]: reject --num-threads=0 for TPC-H generation too by @alamb in #364
• [tpcgen-cli]: extract WorkerQueue into a shared module by @alamb in #362
• cleanup: remove redundant generate_parquet function by @alamb in #363
• Generate TPC-DS parquet tables concurrently (4.5%-35% faster) by @alamb in #361
• feat: Consolidate CI scripts by @clflushopt in #366
• feat: Use fmt::Display for DAT override for all tables. by @clflushopt in #365
• chore: remove standalone TPC-DS benchmark binary by @alamb in #370
• refactor(tpcds): remove unused OutputOptions from DAT output by @alamb in #371
• refactor: construct DAT fields via impls on DatField/NullLiteralField by @alamb in #372
• chore:: remove TableRow::write_to and supporting code by @alamb in #375
• fix: remove invalid Maven cache setup by @kevinjqliu in #376
• fix TPC-DS conformance script validation by @kevinjqliu in https://github.com/data...

v4.0.0-rc.1

What's Changed

• refactor: move TPCH CLI implementation into tpcgen by @alamb in #296
• Run clippy on all targets and features by @alamb in #315
• [tpcgen]: move TPCH CLI tests into tpcgen (unified cli) by @alamb in #313
• [TPC-DS]: remove tpcdsgen CLI binary by @alamb in #303
• [tpcdsgen]: remove chrono dependency from tpcdsgen by @alamb in #312
• refactor: Move some output configuration and writing from tpcdsgen by @alamb in #317
• rename tpcgen crate and binary to tpcgen-cli by @alamb in #314
• Make progress tracker API always available by @kevinjqliu in #323
• [tpcdsgen]: add validated SessionBuilder by @alamb in #318
• Fix release workflow triggers and permissions by @kevinjqliu in #322
• Improve python CLI quickstart documentation by @kevinjqliu in #321
• Add all new crates to the publishing workflow by @kevinjqliu in #326
• Simplify progress tracker plumbing by @kevinjqliu in #325
• [tpcdsgen]: move output options out of Session by @alamb in #319
• [tpcgen-cli]: split CLI integration tests by @alamb in #328
• [tpcgen-cli]: Integrate tpcds dat status output and logging by @alamb in #330
• [tpcgen-cli]: support TPC-DS parquet generation by @alamb in #329
• [tpcgen-cli]: support TPC-DS CSV generation by @kevinjqliu in #331
• Refactor progress tracker by @kevinjqliu in #327
• chore(deps): bump crate-ci/typos from 1.47.2 to 1.48.0 by @dependabot[bot] in #333
• Refactor TPC-DS output dispatch by @kevinjqliu in #332
• [tpcgen-cli] Enable TPC-DS parquet logging by @alamb in #340
• [tpcgen-cli] Respect TPC-DS parquet row group bytes by @alamb in #341
• Minor: Add hyperfine benchmarking instructions to contributing by @alamb in #343
• [tpcgen-cli] Remove dead code in TPCH parquet generation by @alamb in #344
• [tpcdsgen-arrow]: add source row range generation by @alamb in #346
• [tpcgen-cli]: extract shared parquet writer from tpch generation by @alamb in #342
• [arrow] Consolidate to use Arrow RecordBatchReader by @alamb in #345
• chore: remove generate chunks progress wrapper by @kevinjqliu in #347
• [tpcgen-cli] Propagate parquet reader errors by @alamb in #348
• [tpcgen-cli] Validate TPC-DS row group bytes with clap by @alamb in #349
• refactor: simplify TPC-DS inventory DAT generation by @kevinjqliu in #351
• Wire progress tracker with dat/csv/parquet by @kevinjqliu in #350
• [tpcgen-cli] Fix TPC-DS dat table selection by @alamb in #352
• refactor: share TPC-DS sales returns DAT generation by @kevinjqliu in #353
• feat: Per-table row count scaling cache by @clflushopt in #357
• feat: use fmt::Display buffering for Dat output by @clflushopt in #359
• [tpcgen-cli]: parallel TPC-DS parquet generation (10x faster) by @alamb in #360
• [tpcgen-cli]: reject --num-threads=0 for TPC-H generation too by @alamb in #364
• [tpcgen-cli]: extract WorkerQueue into a shared module by @alamb in #362
• cleanup: remove redundant generate_parquet function by @alamb in #363
• Generate TPC-DS parquet tables concurrently (4.5%-35% faster) by @alamb in #361
• feat: Consolidate CI scripts by @clflushopt in #366
• feat: Use fmt::Display for DAT override for all tables. by @clflushopt in #365
• chore: remove standalone TPC-DS benchmark binary by @alamb in #370
• refactor(tpcds): remove unused OutputOptions from DAT output by @alamb in #371
• refactor: construct DAT fields via impls on DatField/NullLiteralField by @alamb in #372
• chore:: remove TableRow::write_to and supporting code by @alamb in #375
• fix: remove invalid Maven cache setup by @kevinjqliu in #376
• fix TPC-DS conformance script validation by @kevinjqliu in #377
• Improve indicatif progress bar UI by @kevinjqliu in #356
• feat: introduce ProgressHandle for easier plumbing by @kevinjqliu in #378
• refactor: let ProgressHandle handle increments by @kevinjqliu in #379
• feat: list TPC-DS tables and descriptions in --tables help by @alamb in #382
• refactor: remove the unused TableRow trait from tpcdsgen by @alamb in #385
• feat: Use fmt::Display CSV wrappers for TPC-DS CSV output (1.8x - 2.0x faster) by @alamb in #373
• chore: Add tpcdsgen-arrow CI coverage, use cargo nextest by @alamb in #387
• Fix incomplete and noisy progress rendering by @kevinjqliu in #386
• chore: update Rust toolchain to 1.97.1 by @alamb in #388
• test: cover CSV in the TPC-DS reparse tests by @alamb in #383
• let dependabot update rust-toolchain by @kevinjqliu in #390
• chore(deps): bump crate-ci/typos from 1.48.0 to 1.49.0 by @dependabot[bot] in #396
• chore(deps): bump taiki-e/install-action from 2 to 2.85.5 by @dependabot[bot] in #395
• Deduplicate TPC-DS table selections in the CLI by @kevinjqliu in #391
• feat: show ETA in progress bars by @kevinjqliu in #393
• [TPC-H] Deduplicate CLI table selections by @kevinjqliu in #392
• Improve progress ETA display by @kevinjqliu in #399
• chore(deps): bump taiki-e/install-action from 2.85.5 to 2.86.6 by @dependabot[bot] in #403
• feat: improve testing docs by @clflushopt in #400
• Update project references after repository move by @kevinjqliu in #407
• Improve Python package metadata by @kevinjqliu in #410
• ci: audit Rust dependencies by @kevinjqliu in #415
• Publish crates.io packages as a workspace by @kevinjqliu in #413
• build: commit and enforce Cargo.lock by @kevinjqliu in #414
• ci: harden GitHub Actions workflows by @kevinjqliu in #412
• chore(deps): bump taiki-e/install-action from 2.86.6 to 2.87.0 by @dependabot[bot] in #418
• chore(deps): bump crate-ci/typos from 1.49.0 to 1.50.0 by @dependabot[bot] in #416
• chore(deps): bump actions/setup-java from 5.7.0 to 6.0.0 by @dependabot[bot] in #417
• fix: wrap TPC-H long RNG seed advance by @kevinjqliu in #419
• fix: align TPC-H scale limits with TPC-DS by @kevinjqliu in #420
• Prepare CI workflows for merge queue by @kevinjqliu in https://github.com/datafusion-contrib/tpcgen-rs/pu...

v3.0.0

What's Changed

• docs: add installation instructions for uv tool by @mocusez in #211
• docs: fix typo in architecture guide by @Matt711 in #213
• feat: TPC-DS support by @clflushopt in #212
• chore(deps): bump actions/cache from 4 to 5 by @dependabot[bot] in #221
• chore(deps): bump actions/upload-artifact from 4 to 6 by @dependabot[bot] in #222
• chore(deps): bump actions/checkout from 4 to 6 by @dependabot[bot] in #223
• chore(deps): bump actions/setup-java from 4 to 5 by @dependabot[bot] in #224
• Fix more typos by @Matt711 in #217
• cli: warn when skip generation by @kevinjqliu in #220
• Add typos to CI by @Matt711 in #227
• cli: add --quiet flag, enable logging by default by @kevinjqliu in #225
• chore(deps): bump crate-ci/typos from 1.41.0 to 1.42.0 by @dependabot[bot] in #230
• feat: custom CSV delimiter support by @clflushopt in #219
• chore(deps): bump crate-ci/typos from 1.42.0 to 1.42.1 by @dependabot[bot] in #231
• chore(deps): bump crate-ci/typos from 1.42.1 to 1.43.4 by @dependabot[bot] in #239
• chore(deps): bump crate-ci/typos from 1.43.4 to 1.43.5 by @dependabot[bot] in #241
• chore(deps): bump actions/attest-build-provenance from 3 to 4 by @dependabot[bot] in #242
• chore(deps): bump crate-ci/typos from 1.43.5 to 1.44.0 by @dependabot[bot] in #243
• chore(deps): bump actions/download-artifact from 7 to 8 by @dependabot[bot] in #244
• chore(deps): bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #245
• Bump arrow-rs version to 58.0.0 by @AdamGS in #246
• Enable LTO and codegen-units=1 for release builds by @kevinjqliu with @Copilot in #240
• Replace deprecated assert_cmd::Command::cargo_bin in tpchgen-cli integration tests by @kevinjqliu in #248
• chore(deps): bump crate-ci/typos from 1.44.0 to 1.46.0 by @dependabot[bot] in #252
• feat: Add parquet subcommand with format-specific argument validation by @Matt711 in #234
• chore: Follow-up improvements to parquet subcommand CLI by @kevinjqliu in #254
• chore(deps): bump crate-ci/typos from 1.46.0 to 1.46.1 by @dependabot[bot] in #256
• chore: Add tbl/csv subcommands, deprecate --format flag by @kevinjqliu in #255
• feat: dsdgen compatibility mode (with C) by @clflushopt in #253
• Speed up TPC-DS conformance tests (only check Md5 by default) by @alamb in #268
• refactor(tpcdsgen): remove unused StringWriter by @alamb in #267
• fix(tpcdsgen): Fix customer.dat to match C dsdgen by @alamb in #266
• chore: Rename test-all-tables.sh to compare-all-tables.sh by @alamb in #264
• chore(deps): bump crate-ci/typos from 1.46.1 to 1.46.2 by @dependabot[bot] in #269
• chore(deps): bump crate-ci/typos from 1.46.2 to 1.47.2 by @dependabot[bot] in #272
• chore(deps): bump actions/checkout from 6 to 7 by @dependabot[bot] in #277
• Bump Arrow and Parquet to 59.0.0 by @kevinjqliu in #278
• feat: Track table generation with progress bars by @Matt711 in #226
• feat: TPCDS <> Arrow support by @clflushopt in #276
• Add tpcgen CLI and Python package by @kevinjqliu in #280
• refactor: group TPC-DS table modules by @kevinjqliu in #291
• docs: document tpcdsgen Session by @alamb in #285
• refactor: Document and simplify tpcdsgen Options <--> Session conversion by @alamb in #286
• Add tpcdsgen-arrow README by @kevinjqliu in #289
• [TPC-DS]: Add tests for paired fact tables, faster comparison by @alamb in #284
• chore(deps): bump actions/cache from 5 to 6 by @dependabot[bot] in #297
• Wire TPC-DS DAT generation into tpcgen by @kevinjqliu in #290
• chore: unify Arrow workspace dependency (on 59.0.0) by @alamb in #300
• fix(tpcds): use i32/ Date32 for call_center columns by @alamb in #294
• fix(tpcds): use i32 / Date32 / Time32 for dbgen_version create date by @alamb in #295
• Prepare v3.0.0 release versions by @kevinjqliu in #304

New Contributors

• @mocusez made their first contribution in #211
• @Matt711 made their first contribution in #213
• @AdamGS made their first contribution in #246
• @kevinjqliu with @Copilot made their first contribution in #240

Full Changelog: v2.0.2...v3.0.0

v2.0.2

What's Changed

• infra: automatically publish to crates.io on release by @kevinjqliu in #190
• infra: configure dependabot for cargo & github action updates by @pmcgleenon in #191
• chore(deps): bump actions/attest-build-provenance from 2 to 3 by @dependabot[bot] in #192
• chore(deps): bump actions/upload-artifact from 4 to 5 by @dependabot[bot] in #193
• chore(deps): bump actions/checkout from 4 to 5 by @dependabot[bot] in #194
• chore(deps): bump actions/download-artifact from 4 to 6 by @dependabot[bot] in #195
• chore(deps): bump actions/cache from 3 to 4 by @dependabot[bot] in #196
• chore(deps): bump actions/checkout from 5 to 6 by @dependabot[bot] in #203
• feat: tpchgen runners as a lib by @clflushopt in #202
• infra: macos-13 is deprecated, replace with macos-15-intel by @kevinjqliu in #208
• chore(deps): bump actions/upload-artifact from 5 to 6 by @dependabot[bot] in #206
• chore(deps): bump actions/cache from 4 to 5 by @dependabot[bot] in #205
• chore(deps): bump actions/download-artifact from 6 to 7 by @dependabot[bot] in #204
• chore(deps): update parquet/arrow/arrow-csv from 56 to 57.1 by @kevinjqliu in #200
• feat: release v2.0.2 by @clflushopt in #210

New Contributors

• @pmcgleenon made their first contribution in #191
• @dependabot[bot] made their first contribution in #192

Full Changelog: v2.0.1...v2.0.2

v2.0.1

What's Changed

• fix: disable s390x support in tpchgen-cli-publish-pypi.yml by @clflushopt in #181
• tpchgen-cli: get readme/license/authors from Cargo.toml by @kevinjqliu in #182
• infra: add a twine check before uploading to pypi by @kevinjqliu in #183
• infra: add continue-on-error: true to the wheel builds by @kevinjqliu in #184
• CI: ensure python cli wheels and sdist can build by @kevinjqliu in #187
• bump to 2.0.1 to fix README in pypi by @kevinjqliu in #188

Full Changelog: v2.0.0...v2.0.1

v2.0.0

What's Changed

• bump arrow version to match datafusion by @joseph-isaacs in #153
• refactor: Extract plan generation to GenerationPlan, add docs and tests by @alamb in #157
• feat: Add integration tests for tpchgen-cli by @alamb in #156
• Minor: parallelize cli part validation test by @alamb in #162
• feat: Chunkify single parts to generate them in parallel by @clflushopt in #155
• feat: expose answers in tbl format as part of the package by @clflushopt in #159
• refactor: Make --part and --parts Option<i32> and add better error handling by @alamb in #166
• refactor: use RangeInclusive<i32> in GenerationPlan by @alamb in #169
• fix: Create multiple row groups when writing single --parts by @alamb in #168
• Minor: Update ARCHITECTURE.md by @alamb in #171
• feat: make parquet row groups size configurable by @kevinjqliu in #158
• Update Rust to 1.89 by @alamb in #177
• Upgrade to arrow/parquet 56.0.0 (8%-10% faster) by @alamb in #176
• Automatically create multiple files with single --part command by @alamb in #175
• Docs: Readme revamp by @alamb in #179
• Prepare for 2.0.0 (upgrade version) by @alamb in #178

New Contributors

• @joseph-isaacs made their first contribution in #153

Full Changelog: v1.1.1...v2.0.0

v1.1.1

What's Changed

• fix: Inconsistent implementation of IntoIterator by @askyx in #148
• feat: prepare v1.1.1 by @clflushopt in #149

New Contributors

• @askyx made their first contribution in #148

Full Changelog: v1.1.0...v1.1.1

v1.1.0

What's Changed

• Add rust-toolchain.toml to control rust version by @alamb in #120
• Change arrow version by @matthewmturner in #122
• fix: Move README to the top level to fix broken link issue by @clflushopt in #125
• Documentation tweaks by @alamb in #117
• feat: fix CLI option docs by @kevinjqliu in #127
• feat: fix CI and run CI for every PR by @kevinjqliu in #128
• [feat] update tpchgen-cli's README by @kevinjqliu in #129
• add python CLI bindings by @kevinjqliu in #121
• fix: Document via examples the behavior of multipart generation by @clflushopt in #135
• Fix up tpchgen-cli's README by @kevinjqliu in #133
• [feat] pyproject use cargo version by @kevinjqliu in #136
• [feat] publish to pypi by @kevinjqliu in #134
• [feat] skip conformance test for markdown file changes by @kevinjqliu in #137
• Replace ByteArrayBuilder with Vec by @scsmithr in #138
• fix: Drop unused TextPoolGenerator by @clflushopt in #141
• feat: Add TPC-H queries by @clflushopt in #140
• feat: prepare v1.1.0 by @clflushopt in #142
• feat: prepare v1.1.0 by @clflushopt in #143

New Contributors

• @kevinjqliu made their first contribution in #127

Full Changelog: v1.0.0...v1.1.0

v1.0.0

What's Changed

• Move chrono to dev dep by @matthewmturner in #107
• Improve documentation on additional shasum checks by @alamb in #109
• Fix typo by @alamb in #111
• fix: Use lowercase for possible values of table names by @szarnyasg in #114
• refactor: Simplify DuckDB export by @szarnyasg in #112
• feat: Link to @alamb's demo and blog in README by @clflushopt in #115
• feat: Small documentation improvements by @clflushopt in #116
• feat: prepare cargo release for v1.0.0 by @clflushopt in #118

New Contributors

• @matthewmturner made their first contribution in #107
• @szarnyasg made their first contribution in #114

Full Changelog: v0.1.1...v1.0.0

v0.1.1

What's Changed

• Update project documentation and benchmarking by @alamb in #72
• Add as_str helper to OrderStatus by @scsmithr in #95
• fix: pin local crate versions by @clflushopt in #94
• fix: minor README improvements by @clflushopt in #96
• Update with benchmark numbers by @alamb in #98
• feat: add to_unix_epoch helper to TPCHDate by @clflushopt in #97
• chore: tweak doc comments, add inline for good measure by @alamb in #102
• feat: pin chrono package to 0.4.39 by @clflushopt in #105
• fix: bump tpchgen-arrow and tpchgen-cli to 0.1.1 by @clflushopt in #106
• fix: bump tpchgen to 0.1.1 by @clflushopt in #108

Full Changelog: v0.1.0...v0.1.1