libmc is a memcached client library for Python without any other dependencies in runtime. It's mainly written in C++ and Cython. libmc can be considered as a drop in replacement for libmemcached and python-libmemcached.
libmc is developing and maintaining by Douban Inc. Currently, It is working in production environment, powering all web traffics in douban.com. Realtime benchmark result is available on travis.
Build and Installation
pip install libmc
import libmc mc = libmc.Client(['localhost:11211', 'localhost:11212']) mc.set('foo', 'bar') assert mc.get('foo') == 'bar'
Under the hood
Under the hood, libmc consists of 2 parts: an internal fully-functional
memcached client implementation in C++ and a Cython wrapper around that
implementation. Dynamic memory allocation and memory-copy are slow, so
we tried our best to avoid them. The
set_multi command is not
natively supported by the memcached
Some techniques are applied to make
set_multi command extremely fast
in libmc (compared to some other similiar libraries).
import libmc from libmc import ( MC_HASH_MD5, MC_POLL_TIMEOUT, MC_CONNECT_TIMEOUT, MC_RETRY_TIMEOUT ) mc = libmc.Client( [ 'localhost:11211', 'localhost:11212', 'remote_host', 'remote_host mc.mike', 'remote_host:11213 mc.oscar' ], do_split=True, comp_threshold=0, noreply=False, prefix=None, hash_fn=MC_HASH_MD5, failover=False ) mc.config(MC_POLL_TIMEOUT, 100) # 100 ms mc.config(MC_CONNECT_TIMEOUT, 300) # 300 ms mc.config(MC_RETRY_TIMEOUT, 5) # 5 s
servers: is a list of memcached server addresses. Each address can be in format of
aliasare optional. If
portis not given, default port
11211will be used.
aliaswill be used to compute server hash if given, otherwise server hash will be computed based on
portis not given or it is equal to
hostwill be used to compute server hash. If
portis not equal to
host:portwill be used).
do_split: Memcached server will refuse to store value if size >= 1MB, if
do_splitis enabled, large value (< 10 MB) will be splitted into several blocks. If the value is too large (>= 10 MB), it will not be stored. default:
comp_threshold: All kinds of values will be encoded into string buffer. If
buffer length > comp_threshold > 0, it will be compressed using zlib. If
comp_threshold = 0, string buffer will never be compressed using zlib. default:
noreply: Whether to enable memcached's
prefix: The key prefix. default:
hash_fn: hashing function for keys. possible values:
NOTE: fnv1_32, fnv1a_32, crc_32 implementations in libmc are per each spec, but they're not compatible with corresponding implementions in libmemcached.
failover: Whether to failover to next server when current server is not available. default:
MC_POLL_TIMEOUTTimeout parameter used during set/get procedure. (default:
MC_CONNECT_TIMEOUTTimeout parameter used when connecting to memcached server on initial phase. (default:
MC_RETRY_TIMEOUTWhen a server is not available dur to server-end error. libmc will try to establish the broken connection in every
MC_RETRY_TIMEOUTs until the connection is back to live.(default:
NOTE: The hashing algorithm for host mapping on continuum is always md5.
Contributing to libmc
Feel free to send a Pull Request. For feature requests or any questions, please open an Issue.
For SECURITY DISCLOSURE, please disclose the information responsibly by sending an email to email@example.com directly instead of creating a GitHub issue.
Does libmc support PHP?
No. But if you like, you can write a wrapper for PHP based on the C++ implementation.
Is Memcached binary protocol supported ?
No. Only Memcached ASCII protocol is supported currently.
Why reinventing the wheel?
Is libmc thread-safe ?
libmc is a single-threaded memcached client. If you initialize a libmc
client in one thread but reuse that in another thread, a Python
ThreadUnsafe will raise in Python.
Is libmc compatible with gevent?
Yes, with the help of greenify,
libmc is friendly to gevent. Read
- Thanks to @fahrenheit2539 and the llvm project for the standalone. SmallVector implementation.
- Thanks to @miloyip for the high performance i64toa implementation.
- Thanks to Ivan Novikov for the research in THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED INJECTIONS.
- Thanks to the PolarSSL project for the md5 implementation.
- Thanks to @lericson for the benchmark script in pylibmc.
- Thanks to the libmemcached project and some other projects possibly not mentioned here.
Copyright (c) 2014-2015, Douban Inc. All rights reserved.
Licensed under a BSD license: https://github.com/douban/libmc/blob/master/LICENSE.txt