New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new hash table #24

Open
rurban opened this Issue Sep 11, 2015 · 4 comments

Comments

1 participant
@rurban
Member

rurban commented Sep 11, 2015

The old hash table uses:

  • linked lists for its collisions, with slow out-of-cache pointer chasing and data overhead.
  • unsorted flags at the end, while some flags are needed for compare.
  • has questionable security measures to slow down all cases. seed ok, randomize iter maybe, but randomize the collisions and slow hash funcs is stupid. the security should be fixed with proper implementations, not by pseudo-security theatre.

either use:

  • rurban/hash-sortbuckets: a sorted static list of collisions (up to 8, maybe start with 3, then 8) as in knuth sorted hash table,
  • khash: use open addressing as everyone else. faster, less space.
  • PERTURB_KEYS_TOP move-to-front with a linked list is the only sane strategy for simple chained bucket lists.
  • HV_FILL_RATE: try lower fill rates than 100%. 100% is pretty insane, esp. with our bad hash funcs. make it fast with builtin_ctz.
  • ✓ use __builtin_ctz for faster division in DO_HSPLIT. Allow -DHV_FILL_RATE=90 definition. (Tested to be the best with FNV1A)
  • HE_ARRAY: According to http://goanna.cs.rmit.edu.au/~jz/fulltext/spire05.pdf
    the best for chained hashing is currently a cache-friendly array:
    cache-friendly continuous buffer of HE's w/ inlined HEK (char_) + SV_ val, but no hash, no next ptr. Also for shared he's: PL_strtab.
  • array_he: inline parts of the HE into the array. array_he vs ll_he. (linked list, see also the he-array branch). array_he (HvARRAY = AHE[]) can contain { hent_he, hent_hash }. this way the hash catches 99% of all comparisons already, and we don't have to chase the external hek ptr, when the hash check fails. every HE entry will then be 2 words (128), instead of one word (64), called AHE. The linked list still contains the old HE*, with { hent_next, hent_hek, hent_val }.
  • one-word-AHE: As possible improvement on that on 64bit use 48bits for the HE ptr, and 16bits of the hash to be compared first. See https://www.semanticscholar.org/paper/Fast-Dynamically-Sized-Concurrent-Hash-Table-Barnat-Rockai/ab7dd007587f411cf99bfe056639e055eff22e0c/pdf
  • use robin-hood as this is currently the best worse-case strategy (being super defensive, but not so stupid to use SipHash, which adds no benefit). with better threading support (shared hashes) eventually use leapfrog.
  • compact ordered hash. use an array of short indices into a compacted array of hash/key/val entries as in PyPy and now python: "compact ordered dict". This saves a lot of space and only add's one indirect lookup into cache-friendly array. See methane/cpython#1 https://mail.python.org/pipermail/python-dev/2016-September/146327.html
    This also iterates over the hash in insertion order, which effectively hides any attempt to get the seed over the iterators. For attacks you need to get collision and robin-hood reordering timings.

maybe use the glibc htable (hsearch, hcreate), just add random seeds and flood detection (collision counts). (or khash, which is basically the same). coucal looks good also. https://github.com/xroche/coucal
but best is probably preshing's http://preshing.com/20160314/leapfrog-probing/, which even can be made concurrent easily.

open addressing is cache friendly and much faster, but table growth is slow.
potion uses khash, first double open, then quadratic open.

either way, first I need to abstract away the collision search in MACROS
and reduce the needed comparisons from 4 to 1. (edit: after my critic, they improved that part to one ptr cmp)
maybe we can normalize all keys to utf8 for faster checks. (probably not)
utf8 is a HEK flag. See http://cpansearch.perl.org/src/RURBAN/illguts-0.49/index.html#hek
The current implementation is just insanity and it only got worse over the years.

The last attempt was here: https://github.com/rurban/perl/commits/rurban/hash-sortbuckets
See the featurex/gh24-*hash* branches.
See also GH #102


plan

  • ✓ -Dhash_func, add and test many more funcs
  • ✓ PERTURB_KEYS_TOP
  • ✓ HV_FILL_RATE
  • ✓ d_builtin_ctz
  • ✓ seperate hv_common_magic
  • ✓ security: warn and protect against hash flood
  • ✓ fix NODEFAULT_SHAREKEYS (#201)
  • ✓ hash_loop: abstract loops and array_he
  • ✓ array_he: seperate HvARRAY from collisions entries, add hent_hash
  • 80% new_hash_table: merge HE fields for faster loop, usehekbitfield, d_little_endian.
  • one-word-AHE (hash+he on 64bit)
  • 40% open_hash, compact_ordered: various strategies with the new abstractions

@rurban rurban added the enhancement label Sep 11, 2015

@rurban rurban self-assigned this Sep 11, 2015

@rurban rurban added this to the v5.24.0 milestone Sep 13, 2015

rurban pushed a commit that referenced this issue Mar 1, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash (WIP)
calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

rurban pushed a commit that referenced this issue Mar 1, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash (WIP)
calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 1, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 1, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 2, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 8, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 12, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 19, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 21, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 23, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 23, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 24, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.
@rurban

This comment has been minimized.

Show comment
Hide comment
@rurban

rurban Mar 24, 2016

Member

Merged now very simple, traditional improvements:

  • default HV_FILL_RATE 90%,
  • PERTURB_KEYS_TOP move-to-front

no hashtable rewrite yet.

Member

rurban commented Mar 24, 2016

Merged now very simple, traditional improvements:

  • default HV_FILL_RATE 90%,
  • PERTURB_KEYS_TOP move-to-front

no hashtable rewrite yet.

rurban pushed a commit that referenced this issue Mar 24, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 26, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 26, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 27, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 28, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 28, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Mar 30, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 3, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 5, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 6, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 8, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 8, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 9, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 12, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 13, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 13, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 17, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 17, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 17, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 18, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 19, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 21, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue Apr 25, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban pushed a commit that referenced this issue May 2, 2016

Reini Urban
HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1% slower,
but we hope to get to speed with the HeARRAY patch. See below.

But the endgoal is to get rid of linked lists and store the collisions
in consecutive memory, in a HeARRAY. Measurements in
"Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 18, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 18, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 18, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 18, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 18, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 18, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 19, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 19, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 20, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 20, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 20, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 20, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 20, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 20, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 21, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 21, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 21, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 21, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 22, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 22, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 23, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 23, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 23, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 23, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 24, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 24, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 26, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 26, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 26, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 26, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Dec 27, 2017

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Dec 27, 2017

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Jan 6, 2018

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Jan 6, 2018

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Jan 9, 2018

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Jan 9, 2018

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Jan 16, 2018

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24

rurban added a commit that referenced this issue Jan 16, 2018

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Feb 4, 2018

HeArray: remove hek_hash and refcounted_he_hash
Calculate hashes on demand, but not store it in a HEK
to make HEK shorter to fill more entries into a cache line.
HEK_HASH(hek) is now invalid and gone.
Use the new HeHASH_calc(he), HEK_HASH_calc(hek), SvSHARED_HASH_calc(sv)
instead.
See http://www.ilikebigbits.com/blog/2016/8/28/designing-a-fast-hash-table
for benchmarks (HashCache).

And using 4 tests in the hot hash loop also makes not much sense,
when checking the length and the string is enough to weed out
collisions.
This strategy, recomputing the hash wehen needed, is so far 1-7% slower,
but we hope to get to speed with the HeARRAY patch. See below.

The endgoal is to get rid of linked lists and store the collisions
inlined in consecutive memory, in a HekARRAY. (len,cmp-flags,char*,other-flags,val)
Measurements in "Cache-Conscious Collision Resolution in String Hash Tables"
by Nikolas Askitis and Justin Zobel, Melbourne 2005 show that this is the
fastest strategy for Open Hashing (chained) tables.
See GH #24 and GH #102

The next idea is to use MSB varint encoding of the str length in a HEK,
because our strings are usually short, len < 63, fits into one byte.
We can then merge it with the cmp-flags, the flags only needed for comparison.
See https://techoverflow.net/blog/2013/01/25/efficiently-encoding-variable-length-integers-in-cc/
or just <63 one byte, >63 MSB: I32 len.
Note that the 1st MSB bit is already taken for UTF8.

rurban added a commit that referenced this issue Feb 4, 2018

open hash: use open addressing with linear probing (WIP)
and a layout similar to array_he, inlined 1-2 word hash array.

variants to try: simple linear, double, quadratic, robin-hood, hopscotch.
khash tried double hashing and found it slower. robin-hood with
backward deletion propagation would be an alternative, but first
we need to re-architecture and abstract the various probing implementations
and search loops scattered all over.

See GH #24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment