chore: Add TPCDS benchmarks #18985

comphead · 2025-11-28T20:12:36Z

Which issue does this PR close?

Closes [EPIC] Support TPC-DS benchmarks #4763.

Rationale for this change

By default DF assesses performance by running TPCH benchmarks howevert TPCDS much more complicated and
can expose the performance issue better especially for join changes like

#18393
#18392

This is initial PR to support TPCDS and I can now do it by following instructions, which are also documented

git clone https://github.com/apache/datafusion-benchmarks

Then run the benchmark with the following command:

DATA_DIR=../../datafusion-benchmarks/tpcds/data/sf1/ ./benchmarks/bench.sh run tpcds

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

xudong963 · 2025-12-03T02:19:11Z

benchmarks/src/tpcds/run.rs

+/// Run the tpcds benchmark.
+#[derive(Debug, StructOpt, Clone)]
+#[structopt(verbatim_doc_comment)]
+pub struct RunOpt {


Do we have a --help to see these commands?

yes, same for others, added the help command to .md

xudong963 · 2025-12-03T02:22:09Z

benchmarks/src/tpcds/run.rs

+        };
+
+        let mut benchmark_run = BenchmarkRun::new();
+        let mut config = self


After the PR #18971, the first round run will have statistics. Or the first round run will spend time fetching statistics. (Maybe some noises)

if thats the case we should probably fix all the benches as all of them comes powered by the same mechanics

Jefffrey

Should this PR close the epic yet, considering it still has some open issues linked?

Jefffrey · 2025-12-03T14:21:18Z

benchmarks/src/tpcds/run.rs

+pub fn get_query_sql(base_query_path: &str, query: usize) -> Result<Vec<String>> {
+    if query > 0 && query < 100 {
+        let filename = format!("{base_query_path}/q{query}.sql");
+        let mut errors = vec![];


nit: errors probably doesn't need to be a vec?

its very similar to current TPCH. but you are right I also feel the benchmark runners can be improved

Jefffrey · 2025-12-03T14:54:01Z

benchmarks/bench.sh

+# Points to TPCDS data generation instructions
+data_tpcds() {
+    echo ""
+    echo "For TPC-DS data generation, please clone the datafusion-benchmarks repository:"


It seems the repo already has the data? So might be confusing to call it data generation

I suggest only showing this message when the directory is not present

When I tried it out, I was confused

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ./benchmarks/bench.sh data tpcds *************************** DataFusion Benchmark Runner and Data Generator COMMAND: data BENCHMARK: tpcds DATA_DIR: /Users/andrewlamb/Software/datafusion/benchmarks/data CARGO_COMMAND: cargo run --release PREFER_HASH_JOIN: true *************************** For TPC-DS data generation, please clone the datafusion-benchmarks repository: git clone https://github.com/apache/datafusion-benchmarks

So I did what the script told me

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ git clone https://github.com/apache/datafusion-benchmarks Cloning into 'datafusion-benchmarks'... remote: Enumerating objects: 283, done. remote: Counting objects: 100% (35/35), done. remote: Compressing objects: 100% (26/26), done. remote: Total 283 (delta 18), reused 9 (delta 9), pack-reused 248 (from 3) Receiving objects: 100% (283/283), 268.89 MiB | 40.49 MiB/s, done. Resolving deltas: 100% (64/64), done.

And then I ran the data command again and got told to get the benchmarking scripts again 🤔

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ./benchmarks/bench.sh data tpcds *************************** DataFusion Benchmark Runner and Data Generator COMMAND: data BENCHMARK: tpcds DATA_DIR: /Users/andrewlamb/Software/datafusion/benchmarks/data CARGO_COMMAND: cargo run --release PREFER_HASH_JOIN: true *************************** For TPC-DS data generation, please clone the datafusion-benchmarks repository: git clone https://github.com/apache/datafusion-benchmarks andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$

I dont know how to do it better tbh, TPCDS generation is quite complicated if you dont own dsdgen which needs to be personally downloaded and built.

the expected flow to run benches with data then with benchmark run and since we dont gen data itself for TPCDS I just left an instructions how to get the data.

Let me know if you feel that other way would more fit

alamb

Thank you @comphead ❤️

I think we already have the tpc-ds queries in the repo here:
https://github.com/apache/datafusion/tree/main/datafusion/core/tests/tpc-ds

Is there any way to reuse the same queries so we are sure we are testing and benchmarking the same thing?

I am also trying it out locally

comphead · 2025-12-03T17:02:26Z

Thank you @comphead ❤️

I think we already have the tpc-ds queries in the repo here: https://github.com/apache/datafusion/tree/main/datafusion/core/tests/tpc-ds

Is there any way to reuse the same queries so we are sure we are testing and benchmarking the same thing?

I am also trying it out locally

Oh, they named 1-N.sql instead of q1-qN, thats why I never found them, checking them now

alamb · 2025-12-03T17:20:43Z

When I ran the command locally

./benchmarks/bench.sh run tpcds

I got a bunch of errors like

Warning registering call_center: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/call_center.parquet

Details

./benchmarks/bench.sh run tpcds
...
     Running `/Users/andrewlamb/Software/datafusion/target/release/dfbench tpcds --iterations 5 --path /Users/andrewlamb/Software/datafusion/benchmarks/data --query_path /Users/andrewlamb/Software/datafusion/benchmarks/queries/tpcds --prefer_hash_join true -o /Users/andrewlamb/Software/datafusion/benchmarks/results/dev2/tpcds_sf1.json`
Running benchmarks with the following options: RunOpt { query: None, common: CommonOpt { iterations: 5, partitions: None, batch_size: None, mem_pool_type: "fair", memory_limit: None, sort_spill_reservation_bytes: None, debug: false }, path: "/Users/andrewlamb/Software/datafusion/benchmarks/data", query_path: "/Users/andrewlamb/Software/datafusion/benchmarks/queries/tpcds", mem_table: false, output_path: Some("/Users/andrewlamb/Software/datafusion/benchmarks/results/dev2/tpcds_sf1.json"), disable_statistics: false, prefer_hash_join: true, enable_piecewise_merge_join: false, sorted: false }
Warning registering call_center: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/call_center.parquet
Warning registering customer_address: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/customer_address.parquet
Warning registering household_demographics: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/household_demographics.parquet
Warning registering promotion: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/promotion.parquet
Warning registering store_sales: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/store_sales.parquet
Warning registering web_page: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/web_page.parquet
Warning registering catalog_page: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/catalog_page.parquet
Warning registering customer_demographics: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/customer_demographics.parquet
Warning registering income_band: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/income_band.parquet
Warning registering reason: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/reason.parquet
Warning registering store: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/store.parquet
Warning registering web_returns: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/web_returns.parquet
Warning registering catalog_returns: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/catalog_returns.parquet
Warning registering customer: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/customer.parquet
Warning registering inventory: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/inventory.parquet
Warning registering ship_mode: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/ship_mode.parquet
Warning registering time_dim: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/time_dim.parquet
Warning registering web_sales: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/web_sales.parquet
Warning registering catalog_sales: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/catalog_sales.parquet
Warning registering date_dim: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/date_dim.parquet
Warning registering item: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/item.parquet
Warning registering store_returns: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/store_returns.parquet
Warning registering warehouse: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/warehouse.parquet
Warning registering web_site: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/web_site.parquet
Query 1 failed: Schema error: No field named sr_returned_date_sk.
Query 2 failed: Schema error: No field named ws_sold_date_sk.
Query 3 failed: Schema error: No field named dt.d_date_sk.
Query 4 failed: Schema error: No field named c_customer_sk.
Query 5 failed: Schema error: No field named ss_store_sk.
Query 6 failed: Schema error: No field named d_year.
Query 7 failed: Schema error: No field named ss_sold_date_sk.
Query 8 failed: Schema error: No field named ca_zip.
Query 9 failed: Schema error: No field named r_reason_sk.
Query 10 failed: Schema error: No field named c.c_customer_sk.
Query 11 failed: Schema error: No field named c_customer_sk.
Query 12 failed: Schema error: No field named ws_item_sk.
Query 13 failed: Schema error: No field named s_store_sk.
Query 14 failed: Schema error: No field named ss_item_sk.
Query 15 failed: Schema error: No field named cs_bill_customer_sk.
Query 16 failed: Schema error: No field named cs1.cs_order_number.
Query 17 failed: Schema error: No field named d1.d_quarter_name.
Query 18 failed: Schema error: No field named cs_sold_date_sk.
Query 19 failed: Schema error: No field named d_date_sk.
Query 20 failed: Schema error: No field named cs_item_sk.
Query 21 failed: Schema error: No field named i_current_price.
Query 22 failed: Schema error: No field named inv_date_sk.
Query 23 failed: Schema error: No field named ss_sold_date_sk.
Query 24 failed: Schema error: No field named ss_ticket_number.
Query 25 failed: Schema error: No field named d1.d_moy.
Query 26 failed: Schema error: No field named cs_sold_date_sk.
Query 27 failed: Schema error: No field named ss_sold_date_sk.
Query 28 failed: Schema error: No field named ss_quantity.
Query 29 failed: Schema error: No field named d1.d_moy.
Query 30 failed: Schema error: No field named wr_returned_date_sk.
Query 31 failed: Schema error: No field named ss_sold_date_sk.
Query 32 failed: Schema error: No field named cs_item_sk.
Query 33 failed: Schema error: No field named i_category.
Query 34 failed: Schema error: No field named store_sales.ss_sold_date_sk.
Query 35 failed: Schema error: No field named c.c_customer_sk.
Query 36 failed: Schema error: No field named d1.d_year.
Query 37 failed: Schema error: No field named i_current_price.
Query 38 failed: Schema error: No field named store_sales.ss_sold_date_sk.
Query 39 failed: Schema error: No field named inv_item_sk.
Query 40 failed: Schema error: No field named cs_order_number.
Query 41 failed: Schema error: No field named i_manufact.
Query 42 failed: Schema error: No field named dt.d_date_sk.
Query 43 failed: Schema error: No field named d_date_sk.
Query 44 failed: Schema error: No field named ss_store_sk.
Query 45 failed: Schema error: No field named i_item_sk.
Query 46 failed: Schema error: No field named store_sales.ss_sold_date_sk.
Query 47 failed: Schema error: No field named ss_item_sk.
Query 48 failed: Schema error: No field named s_store_sk.
Query 49 failed: Schema error: No field named ws.ws_order_number.
Query 50 failed: Schema error: No field named d2.d_year.
Query 51 failed: Schema error: No field named ws_sold_date_sk.
Query 52 failed: Schema error: No field named dt.d_date_sk.
Query 53 failed: Schema error: No field named ss_item_sk.
Query 54 failed: Schema error: No field named cs_sold_date_sk.
Query 55 failed: Schema error: No field named d_date_sk.
Query 56 failed: Schema error: No field named i_color.
Query 57 failed: Schema error: No field named cs_item_sk.
Query 58 failed: Schema error: No field named d_date.
Query 59 failed: Schema error: No field named d_date_sk.
Query 60 failed: Schema error: No field named i_category.
Query 61 failed: Schema error: No field named ss_sold_date_sk.
Query 62 failed: Schema error: No field named d_month_seq.
Query 63 failed: Schema error: No field named ss_item_sk.
Query 64 failed: Schema error: No field named cs_item_sk.
Query 65 failed: Schema error: No field named ss_sold_date_sk.
Query 66 failed: Schema error: No field named ws_warehouse_sk.
Query 67 failed: Schema error: No field named ss_sold_date_sk.
Query 68 failed: Schema error: No field named store_sales.ss_sold_date_sk.
Query 69 failed: Schema error: No field named c.c_customer_sk.
Query 70 failed: Schema error: No field named d_month_seq.
Query 71 failed: Schema error: No field named d_date_sk.
Query 72 failed: Schema error: No field named cs_item_sk.
Query 73 failed: Schema error: No field named store_sales.ss_sold_date_sk.
Query 74 failed: Schema error: No field named c_customer_sk.
Query 75 failed: Schema error: No field named i_item_sk.
Query 76 failed: Schema error: No field named ss_customer_sk.
Query 77 failed: Schema error: No field named ss_sold_date_sk.
Query 78 failed: Schema error: No field named wr_order_number.
Query 79 failed: Schema error: No field named store_sales.ss_sold_date_sk.
Query 80 failed: Schema error: No field named ss_item_sk.
Query 81 failed: Schema error: No field named cr_returned_date_sk.
Query 82 failed: Schema error: No field named i_current_price.
Query 83 failed: Schema error: No field named d_date.
Query 84 failed: Schema error: No field named ca_city.
Query 85 failed: Schema error: No field named ws_web_page_sk.
Query 86 failed: Schema error: No field named d1.d_month_seq.
Query 87 failed: Schema error: No field named store_sales.ss_sold_date_sk.
Query 88 failed: Schema error: No field named ss_sold_time_sk.
Query 89 failed: Schema error: No field named ss_item_sk.
Query 90 failed: Schema error: No field named ws_sold_time_sk.
Query 91 failed: Schema error: No field named cr_call_center_sk.
Query 92 failed: Schema error: No field named ws_item_sk.
Query 93 failed: Schema error: No field named sr_item_sk.
Query 94 failed: Schema error: No field named ws1.ws_order_number.
Query 95 failed: Schema error: No field named ws1.ws_order_number.
Query 96 failed: Schema error: No field named ss_sold_time_sk.
Query 97 failed: Schema error: No field named ss_sold_date_sk.
Query 98 failed: Schema error: No field named ss_item_sk.
Query 99 failed: Schema error: No field named d_month_seq.
Failed Queries: Query 1, Query 2, Query 3, Query 4, Query 5, Query 6, Query 7, Query 8, Query 9, Query 10, Query 11, Query 12, Query 13, Query 14, Query 15, Query 16, Query 17, Query 18, Query 19, Query 20, Query 21, Query 22, Query 23, Query 24, Query 25, Query 26, Query 27, Query 28, Query 29, Query 30, Query 31, Query 32, Query 33, Query 34, Query 35, Query 36, Query 37, Query 38, Query 39, Query 40, Query 41, Query 42, Query 43, Query 44, Query 45, Query 46, Query 47, Query 48, Query 49, Query 50, Query 51, Query 52, Query 53, Query 54, Query 55, Query 56, Query 57, Query 58, Query 59, Query 60, Query 61, Query 62, Query 63, Query 64, Query 65, Query 66, Query 67, Query 68, Query 69, Query 70, Query 71, Query 72, Query 73, Query 74, Query 75, Query 76, Query 77, Query 78, Query 79, Query 80, Query 81, Query 82, Query 83, Query 84, Query 85, Query 86, Query 87, Query 88, Query 89, Query 90, Query 91, Query 92, Query 93, Query 94, Query 95, Query 96, Query 97, Query 98, Query 99
+ set +x
Done

@comphead what would you think about updating the bench.sh data tpcds command so it automatically downloaded the using wget?

For example something like

mkdir -p benchmarks/data/tpcds_sf1
wget https://github.com/apache/datafusion-benchmarks/raw/refs/heads/main/tpcds/data/sf1/call_center.parquet -O benchmarks/data/tpcds_sf1/call_center.parquet

?

comphead · 2025-12-03T17:27:53Z

When I ran the command locally
./benchmarks/bench.sh run tpcds
I got a bunch of errors like

Warning registering call_center: Table file does not exist: /Users/andrewlamb/Software/datafusion/benchmarks/data/call_center.parquet

Details
@comphead what would you think about updating the bench.sh data tpcds command so it automatically downloaded the using wget?

For example something like
mkdir -p benchmarks/data/tpcds_sf1
wget https://github.com/apache/datafusion-benchmarks/raw/refs/heads/main/tpcds/data/sf1/call_center.parquet -O benchmarks/data/tpcds_sf1/call_center.parquet
?

I provided instructions to clone the repo.
I was also confused if the data file is not found then schema cannot be inferred and query parsing failed.

WDYT if I throw an error if no input data exists, pointing them to repo clone?

comphead · 2025-12-03T18:03:46Z

I fixed the scripts, but after latest merge I'm getting

thread 'main' (115794474) panicked at datafusion/physical-expr/src/projection.rs:374:22:
Expected column reference in projection

on Q13, investigating

comphead · 2025-12-03T19:38:39Z

Filed #19075 looks like it is a regression, after the merge the Q13 fails in datafusion-cli

comphead changed the title ~~chore: Add TPCDS benchmarks~~ [WIP]chore: Add TPCDS benchmarks Nov 28, 2025

This was referenced Dec 1, 2025

Fix tpch benchmark harness #19033

Merged

Use tpchgen-cli to generate tpch data in bench.sh #19035

Open

comphead force-pushed the dev2 branch from b378076 to 3bbbac0 Compare December 2, 2025 20:33

github-actions bot added core Core DataFusion crate and removed core Core DataFusion crate labels Dec 2, 2025

comphead changed the title ~~[WIP]chore: Add TPCDS benchmarks~~ chore: Add TPCDS benchmarks Dec 2, 2025

comphead marked this pull request as ready for review December 2, 2025 22:41

comphead requested review from Jefffrey, alamb, andygrove and xudong963 December 2, 2025 22:41

comphead mentioned this pull request Dec 2, 2025

Sort Merge Join: Reduce batch concatenation, use BatchCoalescer, new benchmarks (TPC-H Q21 SMJ up to ~1000x faster) #18875

Open

xudong963 reviewed Dec 3, 2025

View reviewed changes

Jefffrey approved these changes Dec 3, 2025

View reviewed changes

alamb reviewed Dec 3, 2025

View reviewed changes

comphead mentioned this pull request Dec 3, 2025

Benchmarks binary compilation takes too long #19072

Open

comphead added 9 commits December 3, 2025 10:13

chore: Add TPCDS benchmarks

79d1cbc

chore: Add TPCDS benchmarks

ed44d97

chore: Add TPCDS benchmarks

fe7d362

chore: Add TPCDS benchmarks

3ed0e71

chore: Add TPCDS benchmarks

3926849

chore: Add TPCDS benchmarks

0666b31

chore: Add TPCDS benchmarks

dc2a33f

chore: Add TPCDS benchmarks.

e0b5f7d

chore: Add TPCDS benchmarks.

247c2cd

comphead mentioned this pull request Dec 3, 2025

Projection fails on TPCDS Q13 #19075

Open

chore: Add TPCDS benchmarks.

4911eab

comphead force-pushed the dev2 branch from 28939f0 to 4911eab Compare December 3, 2025 21:46

chore: Add TPCDS benchmarks #18985

Are you sure you want to change the base?

chore: Add TPCDS benchmarks #18985

Conversation

comphead commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

comphead commented Dec 3, 2025

Uh oh!

alamb commented Dec 3, 2025

Uh oh!

comphead commented Dec 3, 2025

Uh oh!

comphead commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

comphead commented Nov 28, 2025 •

edited

Loading

comphead commented Dec 3, 2025 •

edited

Loading