Smart string hashing #174
Smart string hashing #174
Conversation
funny-falcon
force-pushed
the
smart_str
branch
2 times, most recently
from
e2d8539
to
e49e91a
Compare
|
Copy benchmark numbers:
|
|
This fixes issues in my benchmark¹ as well, and I cannot see any side effects either. Without the patch and running something like 100 000 iterations it totally kills unpatched LuaJIT (while smart_str patched version runs linearly 17 secs, about 10x more than 10 000 iterations). Something like this: # Patched LuaJIT
$ luajit -e 'require "resty.template.microbenchmark".run(100000)'
Running 100 000 iterations in each test
Parsing Time: 0.393554
Compilation Time: 2.614543 (template)
Compilation Time: 0.010143 (template, cached)
Execution Time: 3.539249 (same template)
Execution Time: 0.332168 (same template, cached)
Execution Time: 4.767025 (different template)
Execution Time: 0.677392 (different template, cached)
Execution Time: 4.409578 (different template, different context)
Execution Time: 0.696215 (different template, different context, cached)
Total Time: 17.439867
# Unpatched LuaJIT
$ luajit -e 'require "resty.template.microbenchmark".run(100000)'
Running 100 000 iterations in each test
Parsing Time: 0.387828
Compilation Time: 2.541598 (template)
Compilation Time: 0.010231 (template, cached)
Execution Time: 3.388479 (same template)
Execution Time: 0.310524 (same template, cached)
Execution Time: 965.385754 (different template)
Execution Time: 566.065327 (different template, cached)
Execution Time: 1113.138669 (different template, different context)
Execution Time: 653.596720 (different template, different context, cached)
Total Time: 3304.825130¹ https://github.com/bungle/lua-resty-template/blob/master/lib/resty/template/microbenchmark.lua |
|
Only for *nix? |
|
Why "unly for *nix"? It is cause of |
|
Could you try to compile on Windows now with |
|
Sorry , I'm writing from the phone. As an experiment, try xxHash32 (xxHash64 for x64). It's really fast. |
|
Dmitry, I've played a lot with hash functions, and I know about xxhash.
xxhash32 is not more secure and not much faster than my fast hash (maybe
even slower, i don't remember exact numbers) (LUAJIT_SMART_STRING=1) . But
it is much more complex.
Secure mode has to exist for those, who wants more security. It is compile
time option and it is not default. So if you doesn't want it, you will not
pay for.
|
|
@data-man , I'm really sure, that "secure" hash could at least twice faster (by mixing 2blocks-8bytes per siphash round instead of 1block-4bytes). It still will be safe for this use-case. But I don't have much authority to people to believe such claim :-( |
|
@bungle can you rerun tests, please? Both with SMART_STRING=1 and 2 (and unpatched). I've noticed, that previous version were a bit slower for uncontendent case. So I've change heuristic a bit. I'd like to see, if there is a difference in realworld case. |
|
SMART_STRING = 1: SMART_STRING = 2: SMART_STRING = 0: I'm not sure if this is a "real world" test, but it shows a potential problem that is easy to trigger even with a lot smaller iteration counts than the above 10,000. Both SMART_STRING = 1 and SMART_STRING = 2 work about equally the same (probably somewhat in a error margin), but SMART_STRING = 0 renders exponentially worse results (SMART_STRING = 1 and SMART_STRING = 2 behave linerially when the iteration count is raised). So, I still hope these would be merged. |
|
@bungle , thank you. Now "template" and "same template" lines look like not sufferring from patch. |
|
I've made tests on my computer (i7-4770) both 64bit and 32bit version. May some one make test on arm hardware? 64bit version: And 32bit version: |
- detect when a lot of collisions generated
- if two full collisions found (ie hash value and len is equal)
- if collision chain is longer than 18
("average maximum" chain with fillfactor 1.0 is near 7)
- calculate "full" hash for strings in long collision chain
- use "bloom" filter to bookkeeping existence of strings with "full" hash
- refill "bloom" on string sweeping.
|
I've updated pull request: squashed into 3 commits. |
|
@funny-falcon
Maybe lj_siphash? |
|
@data-man well, it is not quite "siphash". It is 32bit cousine/offspring. It uses SipHash like permutation but over 32bit values (shifts are taken from Chaskey - other 32bit SipHash offspring). So it could not be named "SipHash", cause "SipHash" is defined as permutation over 64bit values. I think, comment about relation of "saphash" and "siphash" is just enough. |
|
@MikePall , I'm ready to work on it further if you describe what is wrong with this code. |
|
Sad to see this closing with wontfix, I also hope this could be sorted out somehow. |
Use full string hashing when a lot of collisions are generated.
"average maximum" chain of table with fill-factor 1.0 is near 7, so if collision chain is longer than 10, then it is bad sign.
Separately:
Obsoletes #171 and #169, closes #168