Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(query): new hash table and parallel finalize for hash join #11140

Merged
merged 30 commits into from
Apr 26, 2023
Merged

feat(query): new hash table and parallel finalize for hash join #11140

merged 30 commits into from
Apr 26, 2023

Conversation

Dousir9
Copy link
Member

@Dousir9 Dousir9 commented Apr 20, 2023 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR

Closes #10907

new hash table:

  • parallel finalize
  • KeysU8, KeysU16, KeysU32, KeysU64, KeysU128, KeysU256
  • SingleString
  • Serializer

benchmark TPCH 30G:

Query main this pr ratio(old/new)
Q1 2.777466 2.737082 102%
Q2 1.810927 1.229712 149%
Q3 1.651882 0.865494 208%
Q4 6.221335 1.221579 530%
Q5 5.398579 3.464785 157%
Q6 0.098828 0.098178 96%
Q7 8.770048 1.592533 568%
Q8 9.153106 5.887519 156%
Q9 15.310466 5.404334 293%
Q10 2.460001 1.541862 160%
Q11 0.538278 0.44402 125%
Q12 1.057991 0.948385 112%
Q13 10.809318 5.900187 184%
Q14 0.55777 0.167994 308%
Q15 0.270103 0.254141 107%
Q16 0.443502 0.336052 136%
Q17 3.876887 2.672004 149%
Q18 16.456826 8.754915 191%
Q19 1.391517 1.380548 101%
Q20 4.237823 1.641406 258%
Q21 60.075969 22.086254 271%
Q22 6.341914 0.864747 721%

@vercel
Copy link

vercel bot commented Apr 20, 2023 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
databend ⬜️ Ignored (Inspect) Visit Preview Apr 26, 2023 2:36pm

@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Apr 20, 2023
@xudong963 xudong963 self-requested a review April 20, 2023 05:15
@xudong963 xudong963 added the ci-benchmark Benchmark: run all test label Apr 20, 2023
@github-actions
Copy link
Contributor

local cloud cloud

ClickBench Report

@github-actions
Copy link
Contributor

local cloud cloud

ClickBench Report

@Dousir9 Dousir9 removed the ci-benchmark Benchmark: run all test label Apr 23, 2023
@Dousir9
refine code
bd11c24
@Dousir9
remove useless code
3d6c7e8
@Dousir9
Merge branch 'main' of github.com:datafuselabs/databend into explore_…
cd0c010 
…hash_table

 Please enter a commit message to explain why this merge is necessary,
@Dousir9 Dousir9 marked this pull request as ready for review April 24, 2023 09:57
xudong963
xudong963 reviewed Apr 25, 2023
View reviewed changes
src/common/hashtable/src/hashjoin_hashtable.rs Outdated Show resolved Hide resolved
src/common/hashtable/src/hashjoin_string_hashtable.rs Outdated Show resolved Hide resolved
src/common/hashtable/src/hashjoin_string_hashtable.rs Show resolved Hide resolved
src/common/hashtable/src/hashjoin_string_hashtable.rs Outdated Show resolved Hide resolved
src/query/service/src/pipelines/processors/transforms/hash_join/join_hash_table.rs Show resolved Hide resolved
src/query/service/src/pipelines/processors/transforms/hash_join/hash_join_state_impl.rs Outdated Show resolved Hide resolved
src/query/service/src/pipelines/processors/transforms/hash_join/hash_join_state_impl.rs Show resolved Hide resolved
src/query/service/src/pipelines/processors/transforms/hash_join/join_hash_table.rs Show resolved Hide resolved
@sundy-li sundy-li requested a review from zhang2014 April 26, 2023 03:24
@Dousir9
update
92d7bbc
@Dousir9
update
be6c756
@Dousir9
Merge branch 'main' of github.com:datafuselabs/databend into explore_…
ae416e2 
…hash_table

 Please enter a commit message to explain why this merge is necessary,
@Dousir9 Dousir9 added the ci-benchmark Benchmark: run all test label Apr 26, 2023
@datafuselabs datafuselabs deleted a comment from github-actions bot Apr 26, 2023
@Dousir9 Dousir9 removed the ci-benchmark Benchmark: run all test label Apr 26, 2023
@Dousir9 Dousir9 marked this pull request as draft April 26, 2023 07:19
@Dousir9 Dousir9 marked this pull request as ready for review April 26, 2023 07:44
@Dousir9 Dousir9 marked this pull request as draft April 26, 2023 07:52
@Dousir9 Dousir9 marked this pull request as ready for review April 26, 2023 08:26
@BohuTANG BohuTANG merged commit 08c4c68 into datafuselabs:main Apr 26, 2023
42 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhang2014 zhang2014 zhang2014 approved these changes

@sundy-li sundy-li sundy-li approved these changes

@xudong963 xudong963 xudong963 approved these changes

Assignees
No one assigned
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature: explore a new hashtable for hashjoin
5 participants
@Dousir9 @sundy-li @zhang2014 @xudong963 @BohuTANG