GH-38372: [C++] replace memo table with a swiss table like implementation #40915
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
|
this pr is work in progress now, i will add some benchmarks and test this week |
|
You can mark it as "draft" first |
mapleFU
reviewed
Mar 31, 2024
cpp/src/arrow/util/hashing.h
Outdated
| @@ -19,6 +19,8 @@ | |||
|
|
|||
| #pragma once | |||
|
|
|||
| #include <_types/_uint16_t.h> | |||
github-actions
bot
added
awaiting committer review
mapleFU
reviewed
Mar 31, 2024
cpp/src/arrow/util/bit_util.h
Outdated
| /// sub-expressions the result is the high bits set where the bytes in v were zero, since | ||
| /// the high bits set due to a value greater than 0x80 in the first sub-expression are | ||
| /// masked off by the second. | ||
| uint64_t HasZeroByte(uint64_t value) { |
There was a problem hiding this comment.
Since it's a "has" return bool?
There was a problem hiding this comment.
oh I get the point, I don't know whether putting this in bit_util a good idea, since it's like swisstable's logic
There was a problem hiding this comment.
yes, it will a little bit confusing, move this function to swiss table class and rename it to GetZeroByteMask will be better
mapleFU
reviewed
Apr 1, 2024
cpp/src/arrow/util/hashing.h
Outdated
| ++reinsert_count; | ||
| auto p = DoLookup<CompareKind::NoCompare>(old_entry->h, | ||
| [](const Payload*) { return false; }); | ||
| assert(!p.second); |
SGZW
force-pushed
the
add_swiss_table
branch
2 times, most recently
from
d045800
to
70c64f0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
replace memo table with swiss table to improve set lookup functions, vector hash functions, the count_distinct aggregate function, dictionary unification .. perfomance.
related discuss: #38372
What changes are included in this PR?
Are these changes tested?
Yes
Are there any user-facing changes?
No