Add benchmarks for testing row filtering #3769

thinkharderdev · 2022-10-08T13:41:15Z

Which issue does this PR close?

Rationale for this change

Need a set benchmarks for evaluating performance implications of parquet predicate pushdown. This sets up some very basic benchmarks which can be used for that purpose. Thanks to @tustvold for cooking up a script to generate synthetic datasets for this purpose.

What changes are included in this PR?

Add new benchmark script parquet_filter_pushdown which will execute a series of ParquetExec plans with different filter predicates. For each predicate in the suite, we will execute the plan with all three different ParquetScanOptions configurations:

Disable filter pushdown entirely (currently the default behavior)
Enable filter pushdown and predicate reordering
Enable filter pushdown without predicate reordering.

Are there any user-facing changes?

No

thinkharderdev · 2022-10-08T14:29:13Z

@alamb @tustvold @Ted-Jiang

Ted-Jiang · 2022-10-09T03:10:02Z

benchmarks/src/bin/parquet_filter_pushdown.rs

+    ];
+
+    let filter_matrix = vec![
+        // Selective-ish filter


well-defined test case and test data! 👍

Ted-Jiang · 2022-10-09T03:34:08Z

benchmarks/src/bin/parquet_filter_pushdown.rs

+    path: PathBuf,
+
+    /// Batch size when reading Parquet files
+    #[structopt(short = "s", long = "batch-size", default_value = "8192")]


i think there two short options 's'

in fact when you run the example in debug mode it asserts on exactly this problem:

cargo run --bin parquet_filter_pushdown -- --path ./data --\|alamb@aal-dev:~/arrow-datafusion$ scale-factor 1.0 ... Running `target/debug/parquet_filter_pushdown --path ./data --scale-factor 1.0` |error[E0433]: failed to resolve: use of undeclared type `WriterProperties` thread 'main' panicked at 'Argument short must be unique | --> benchmarks/src/bin/parquet_filter_pushdown.rs:235:17 | | -s is already in use', /home/alamb/.cargo/registry/src/github.com-1ecc6299db9ec823/cla\|235 | let props = WriterProperties::builder() p-2.34.0/src/app/parser.rs:190:13 | | ^^^^^^^^^^^^^^^^ not found in this scope note: run with `RUST_BACKTRACE=1` environment variable to display a ba

Yeah, just removed one of them. I don't think batch size needs to be a cli option in this benchmark.

benchmarks/README.md

Ted-Jiang · 2022-10-09T04:42:59Z

@thinkharderdev thanks for your great bench.
I run parquet tools in local get (1.0 GB)

(venv) yangjiang@LM-SHC-15009782 data % parquet-tools column-index  ./logs.parquet                                                     
row group 0:
column index for column service:
Boudary order: UNORDERED
                      null count  min                                       max                                     
page-0                         0  backend                                   frontend                                

offset index for column service:
                          offset   compressed size       first row index
page-0                        62               117                     0

column index for column host:
Boudary order: UNORDERED
                      null count  min                                       max                                     
page-0                         0  i-1ec3ca3151468928.ec2.internal           i-1ec408f54dbd3750.ec2.internal         

offset index for column host:
                          offset   compressed size       first row index
page-0                       566               125                     0

column index for column pod:
Boudary order: UNORDERED
                      null count  min                                       max                                     
page-0                         0  aejowuublavflbbsvlfozigwpmrxldvhaollk     zxxlzhdrucrhpicpdgxtfpyuknvviimggtq     

offset index for column pod:
                          offset   compressed size       first row index
page-0                      6689               602                     0

column index for column container:
Boudary order: UNORDERED
                      null count  min                                       max                                     
page-0                         0  backend_container_0                       frontend_container_1                    

offset index for column container:
                          offset   compressed size       first row index
page-0                      7602               593                     0

There are at most two pages in one col, I think if we adjust to get more pages in one col (like reduce the page size), it will get greater performance in enable enable_page_index , we can get more opportunitys to skip whole pages without decoding! 🤔

FYI, i see impala choose to use fixed row number in one page to do benchmark for getting good performance.

alamb · 2022-10-11T14:28:50Z

Thank you @thinkharderdev -- I plan to review this PR in detail later today

alamb

This looks great -- thank you @thinkharderdev

I also verified the parquet file that was created

$ du -s -h /tmp/data/logs.parquet 
988M    /tmp/data/logs.parquet

It looks good to me (using the neat pqrs tool from @manojkarthick)

alamb@aal-dev:~/2022-10-05-slow-query-high-cardinality$ pqrs cat --csv /tmp/data/logs.parquet  | head 

############################
File: /tmp/data/logs.parquet
############################

service,host,pod,container,image,time,client_addr,request_duration_ns,request_user_agent,request_method,request_host,request_bytes,response_bytes,response_status
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000000000+00:00,127.216.178.64,-1261239112,rkxttrfiiietlsaygzphhwlqcgngnumuphliejmxfdznuurswhdcicrlprbnocibvsbukiohjjbjdygwbfhxqvurm,PUT,https://backend.mydomain.com,-312099516,1\
448834362,200
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000001024+00:00,187.49.24.179,1374800428,sdxkctvmvuqxhwigrhjaouwdzvasqlqphymcgqvfmsbjswswnzgvanmalnmvsvruakcudmqvzateabhlya,PATCH,https://backend.mydomain.com,-1363067408,1111765\
98,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000002048+00:00,14.29.229.168,-1795280692,bhlvymbbtgcqrwzujukyotusnsoidygnklhx,GET,https://backend.mydomain.com,-1323615082,-705662117,400
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000003072+00:00,180.188.29.17,-717290117,hjaynltdswdekcguqmrkucsepzqjhasklmimkibabijihitimmsglgettywifdzmraipvyvekczuwxettayslrffyz,HEAD,https://backend.mydomain.com,-1847395296,\
1206750179,200
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000004096+00:00,68.92.115.208,759902764,yupopowlaqbwskdwvtlitugpzzxoajhvnmndhca,DELETE,https://backend.mydomain.com,-50170254,-415949533,403
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000005120+00:00,230.160.203.201,-1271567754,pwbruedgdgtsavjuksxwkecxulbnjbsaltuvcjxcmblhnraawouvrunwwsmvjbq,GET,https://backend.mydomain.com,-1193079450,1281912293,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000006144+00:00,249.254.50.191,-971196614,amtuqookzibtvrtqfnyzuyesikbrafhcfnjhoaoedvmlwpkypfsedtbbwlbnzigwgjpzcwdxtwhrykcibmhlxnkckynvgli,PATCH,https://backend.mydomain.com,-2627\
74709,-1695212300,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000007168+00:00,77.183.81.164,-547300163,ogkufdxssjqzjphxwvegwvofchpsgntbyslgarcyqcawokzfoppdftoctmtlwcvikazwrujlgrzrlqueaaceibxvdicfhp,HEAD,https://backend.mydomain.com,-1349820\
595,-327759246,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000008192+00:00,63.17.88.115,-88404773,ogardohhoorttptpnkxmvyenqfzvvkjabcrfwapoywttjdunvmlgwgstmsjbefxqta,HEAD,https://backend.mydomain.com,1830978558,,200
Error: ArrowReadWriteError(CsvError("Broken pipe (os error 32)"))

benchmarks/README.md

alamb · 2022-10-11T19:33:57Z

benchmarks/src/bin/parquet_filter_pushdown.rs

+    let generator = Generator::new();
+
+    let file = File::create(&path).unwrap();
+    let mut writer = ArrowWriter::try_new(file, generator.schema.clone(), None).unwrap();


I wonder if we should make the properties used here explicit?

Like maybe explicitly set what type of statistics are created as well as potentially setting ZSTD compression

https://docs.rs/parquet/24.0.0/parquet/file/properties/struct.WriterPropertiesBuilder.html

Yeah, I'll need to revisit this again once apache/arrow-rs#2854 is released and pulled in so we cam generate the files with proper page sizes (which should make a significant difference)

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

thinkharderdev · 2022-10-11T20:00:57Z

This looks great -- thank you @thinkharderdev

I also verified the parquet file that was created

$ du -s -h /tmp/data/logs.parquet 
988M    /tmp/data/logs.parquet

It looks good to me (using the neat pqrs tool from @manojkarthick)

alamb@aal-dev:~/2022-10-05-slow-query-high-cardinality$ pqrs cat --csv /tmp/data/logs.parquet  | head 

############################
File: /tmp/data/logs.parquet
############################

service,host,pod,container,image,time,client_addr,request_duration_ns,request_user_agent,request_method,request_host,request_bytes,response_bytes,response_status
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000000000+00:00,127.216.178.64,-1261239112,rkxttrfiiietlsaygzphhwlqcgngnumuphliejmxfdznuurswhdcicrlprbnocibvsbukiohjjbjdygwbfhxqvurm,PUT,https://backend.mydomain.com,-312099516,1\
448834362,200
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000001024+00:00,187.49.24.179,1374800428,sdxkctvmvuqxhwigrhjaouwdzvasqlqphymcgqvfmsbjswswnzgvanmalnmvsvruakcudmqvzateabhlya,PATCH,https://backend.mydomain.com,-1363067408,1111765\
98,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000002048+00:00,14.29.229.168,-1795280692,bhlvymbbtgcqrwzujukyotusnsoidygnklhx,GET,https://backend.mydomain.com,-1323615082,-705662117,400
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000003072+00:00,180.188.29.17,-717290117,hjaynltdswdekcguqmrkucsepzqjhasklmimkibabijihitimmsglgettywifdzmraipvyvekczuwxettayslrffyz,HEAD,https://backend.mydomain.com,-1847395296,\
1206750179,200
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000004096+00:00,68.92.115.208,759902764,yupopowlaqbwskdwvtlitugpzzxoajhvnmndhca,DELETE,https://backend.mydomain.com,-50170254,-415949533,403
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000005120+00:00,230.160.203.201,-1271567754,pwbruedgdgtsavjuksxwkecxulbnjbsaltuvcjxcmblhnraawouvrunwwsmvjbq,GET,https://backend.mydomain.com,-1193079450,1281912293,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000006144+00:00,249.254.50.191,-971196614,amtuqookzibtvrtqfnyzuyesikbrafhcfnjhoaoedvmlwpkypfsedtbbwlbnzigwgjpzcwdxtwhrykcibmhlxnkckynvgli,PATCH,https://backend.mydomain.com,-2627\
74709,-1695212300,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000007168+00:00,77.183.81.164,-547300163,ogkufdxssjqzjphxwvegwvofchpsgntbyslgarcyqcawokzfoppdftoctmtlwcvikazwrujlgrzrlqueaaceibxvdicfhp,HEAD,https://backend.mydomain.com,-1349820\
595,-327759246,204
backend,i-1ec3ca3151468928.ec2.internal,aqcathnxqsphdhgjtgvxsfyiwbmhlmg,backend_container_0,backend_container_0@sha256:30375999bf03beec2187843017b10c9e88d8b1a91615df4eb6350fb39472edd9,1970-01\
-01T00:00:00.000008192+00:00,63.17.88.115,-88404773,ogardohhoorttptpnkxmvyenqfzvvkjabcrfwapoywttjdunvmlgwgstmsjbefxqta,HEAD,https://backend.mydomain.com,1830978558,,200
Error: ArrowReadWriteError(CsvError("Broken pipe (os error 32)"))

I stole the gen code from @tustvold so you know it works :)

alamb · 2022-10-12T10:43:51Z

CI check is unrelated : #3798

alamb · 2022-10-12T10:44:31Z

benchmarks/src/bin/parquet_filter_pushdown.rs

+        combine_filters(&[
+            col("request_method").not_eq(lit("GET")),
+            col("response_status").eq(lit(400_u16)),
+            // TODO this fails in the FilterExec with Error: Internal("The type of Dictionary(Int32, Utf8) = Utf8 of binary physical should be same")


ursabot · 2022-10-12T10:52:41Z

Benchmark runs are scheduled for baseline = ae5b23e and contender = fb39d5d. fb39d5d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

thinkharderdev added 4 commits October 8, 2022 07:57

Add benchmarks for testing row filtering

95afefa

Rework to test everything at once

66c1076

Make sure we get the same results with pushdown disabled

3ca8698

Add license header

8efb010

Ted-Jiang reviewed Oct 9, 2022

View reviewed changes

alamb approved these changes Oct 11, 2022

View reviewed changes

thinkharderdev and others added 2 commits October 11, 2022 15:44

Update benchmarks/README.md

3c843cf

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

PR comments

bb2aeac

alamb reviewed Oct 12, 2022

View reviewed changes

alamb merged commit fb39d5d into master Oct 12, 2022

alamb deleted the row-filter-benchmarks branch October 12, 2022 10:45

alamb mentioned this pull request Oct 25, 2022

Put common code used for testing code into datafusion/test_utils.rs #3960

Closed

Add benchmarks for testing row filtering #3769

Add benchmarks for testing row filtering #3769

Uh oh!

Conversation

thinkharderdev commented Oct 8, 2022

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

thinkharderdev commented Oct 8, 2022

Uh oh!

Ted-Jiang Oct 9, 2022

Choose a reason for hiding this comment

Uh oh!

Ted-Jiang Oct 9, 2022

Choose a reason for hiding this comment

Uh oh!

alamb Oct 11, 2022

Choose a reason for hiding this comment

Uh oh!

thinkharderdev Oct 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Ted-Jiang commented Oct 9, 2022

Uh oh!

alamb commented Oct 11, 2022

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb Oct 11, 2022

Choose a reason for hiding this comment

Uh oh!

thinkharderdev Oct 11, 2022

Choose a reason for hiding this comment

Uh oh!

thinkharderdev commented Oct 11, 2022

Uh oh!

alamb commented Oct 12, 2022

Uh oh!

alamb Oct 12, 2022

Choose a reason for hiding this comment

Uh oh!

ursabot commented Oct 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants