Optimize murmur hash calculation #241
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mdumandag
force-pushed
the
murmur-hash
branch
2 times, most recently
from
e1dcadd to
0295ccc
Compare
We were always passing a bytes-like object to the murmur hash function but, we were copying it from the HEAP_DATA_OVERHEAD(8) offset, which can be huge. I removed that copy and tuned the offsets according to our HEAP_DATA_OVERHEAD. Another parameter of the function was the data size. For valid Data objects, this is always equal to the `len(buffer) - HEAP_DATA_OVERHEAD`. This parameter is removed and this length is calculated inside the murmur hash function now. Also, inlined the _fmix function call and removed the unnecessary method calls on hash_to_index function.
mdumandag
force-pushed
the
murmur-hash
branch
from
f0cf337 to
03e57f3
Compare
Fix the `__hash__` method of the `Data` class to convert `bytes` to `bytearray` if it executing on Python2 since we cannot use murmur hash on `str`(alias of `bytes` on Python2) objects. This extra if check is not necessary on `hash_code` method since it is only called on the Data objects to be sent by the client which will have a bytearray buffer. Also, extra test cases are added
mdumandag
force-pushed
the
murmur-hash
branch
from
03e57f3 to
1429b01
Compare
puzpuzpuz
approved these changes
Nov 11, 2020
Contributor
Author
|
Thanks for the review Andrey |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We were always passing a bytes-like object to the murmur hash function
but, we were copying it from the HEAP_DATA_OVERHEAD(8) offset, which
can be huge. I removed that copy and tuned the offsets according to our
HEAP_DATA_OVERHEAD.
Another parameter of the function was the data size. For valid Data
objects, this is always equal to the
len(buffer) - HEAP_DATA_OVERHEAD.This parameter is removed and this length is calculated inside the
murmur hash function now.
Also, inlined the _fmix function call and removed the unnecessary
method calls on hash_to_index function.