DistLock
Distributed locks with Redis and Python
This library implements the DistLock algorithm introduced by @antirez, Due to the origin fork from redlock has been inactive since 2015, Then this repo was born with additional implementations:
- Bug fix
- Python3 syntax improvements
- New tweak for adaptation
There are already a few redis based lock implementations in the Python world, e.g. retools, redis-lock.
However, these libraries can only work with single-master redis server. When the Redis master goes down, your application has to face a single point of failure. We can't rely on the master-slave replication, because Redis replication is asynchronous.
This is an obvious race condition with the master-slave replication model :
- Client A acquires the lock into the master.
- The master crashes before the write to the key is transmitted to the slave.
- The slave gets promoted to master.
- Client B acquires the lock to the same resource A already holds a lock for. SAFETY VIOLATION!
To resolve this problem, the Distlock algorithm assume we have N
Redis masters. These nodes are totally independent (no replications). In order to acquire the lock, the client will try to acquire the lock in all the N instances sequentially. If and only if the client was able to acquire the lock in the majority ((N+1)/2
)of the instances, the lock is considered to be acquired.
The detailed description of the DistLock algorithm can be found in the Redis documentation: Distributed locks with Redis.
The distlock.DistLock
class shares a similar API with the threading.Lock
class in the Python Standard Library.
from distlock import DistLock
# By default, if no redis connection details are
# provided, DistLock uses redis://127.0.0.1:6379/0
lock = DistLock("distributed_lock")
lock.acquire()
do_something()
lock.release()
As with threading.Lock
, distlock.DistLock
objects are context managers thus support the With Statement. This way is more pythonic and recommended.
from distlock import DistLock
with DistLock("distributed_lock"):
do_something()
from distlock import DistLock
with DistLock("distributed_lock",
connection_details=[
{'host': 'xxx.xxx.xxx.xxx', 'port': 6379, 'db': 0},
{'host': 'xxx.xxx.xxx.xxx', 'port': 6379, 'db': 0},
{'host': 'xxx.xxx.xxx.xxx', 'port': 6379, 'db': 0},
{'host': 'xxx.xxx.xxx.xxx', 'port': 6379, 'db': 0},
]
):
do_something()
The connection_details
parameter expects a list of keyword arguments for initializing Redis clients.
Other acceptable Redis client arguments can be found on the redis-py doc.
Usually the connection details of the Redis nodes are fixed. DistLockFactory
can help reuse them, create multiple DistLocks but only initialize the clients once.
from distlock import DistLockFactory
factory = DistLockFactory(
connection_details=[
{'host': 'xxx.xxx.xxx.xxx'},
{'host': 'xxx.xxx.xxx.xxx'},
{'host': 'xxx.xxx.xxx.xxx'},
{'host': 'xxx.xxx.xxx.xxx'},
])
with factory.create_lock("distributed_lock"):
do_something()
with factory.create_lock("another_lock"):
do_something()