Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for a pluggable consistent hashing system #52

Closed
sontek opened this issue Jun 20, 2015 · 2 comments
Closed

Add support for a pluggable consistent hashing system #52

sontek opened this issue Jun 20, 2015 · 2 comments

Comments

@sontek
Copy link
Contributor

sontek commented Jun 20, 2015

Options:

  1. https://pypi.python.org/pypi/hash_ring
  2. https://github.com/RJ/ketama/tree/master/python_ketama
  3. https://github.com/ewdurbin/clandestined-python

Ketama is the one that pylibmc supports but since the protocols between the clients aren't 100% compatible there isn't any reason to completely follow their lead.

@sontek
Copy link
Contributor Author

sontek commented Jun 20, 2015

After doing a bunch of testing I've decided to go with clandestined. It uses a C extension for the murmur3 hashing algorithm which is pretty much necessary for performance requirements but I think its fine. So using this wont be "pure python" anymore but clandestined works with py2,py3, and pypy.

My test with 10,000 keys, this distribution looks like this:

Before Removal

crc32
ketama
murmur3

After Removal

crc32_removed
ketama_removed
murmur3_removed

You can see using HRW (Rendezvous) balancing with a Murmur3 hash we are able to prevent any hotspots and evenly distribute the keys across all our servers.

I also checked how many keys would have to be redistributed if we dropped off on server and get the following:

CRC3 Redistribution 51.04
Murmur3 Redistribution 50.3
Ketama Redistribution 59.8

Murmur3 had the most evenly distributed keys and had to rebalance the lease amount. Ketama (which is what pylibmc uses) has the worst amount of hotspots and has to redistribute the most.

This was referenced Jun 20, 2015
@duanhongyi
Copy link
Contributor

Code LGTM.
I have closed pull request #44

@cgordon cgordon closed this as completed Jul 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants