This project implements a distributed key-value store in Python, designed to explore the fundamental trade-offs between consistency, availability, and latency in distributed systems.
By utilizing a Single-Leader architecture combined with Semi-Synchronous replication, the system simulates a realistic environment where multiple nodes (running in Docker containers) must coordinate to maintain data integrity across a network with variable latency.
The core of the system relies on a strict separation of concerns between the Leader and the Followers. This design simplifies concurrency control by designating the Leader as the single source of truth for all write operations.
- Write Operations (POST, DELETE): Clients must communicate exclusively with the Leader. The Leader is responsible for sequencing writes and propagating changes to the followers.
- Read Operations (GET): To maximize read throughput, clients can query any node in the cluster (Leader or Followers).
Unlike fully synchronous systems (which halt if one node is down) or asynchronous systems (which risk data loss), this implementation uses a semi-synchronous approach. When the Leader receives a write request, it does not respond to the client immediately. Instead, it propagates the data to all 5 followers and waits for a specific number of acknowledgments, defined by the WRITE_QUORUM.
This mechanism allows for tunable consistency. A higher quorum guarantees higher data durability but incurs a latency penalty, while a lower quorum prioritizes speed.
Built on Python's asyncio and aiohttp, the system is non-blocking by default. The Leader utilizes concurrent execution to broadcast replication requests to all followers simultaneously rather than sequentially. This ensures that the total write latency is determined by the slowest follower required to meet the quorum, rather than the sum of all follower latencies. Thread-safety within the local data store is managed via asyncio locks to prevent race conditions during high-concurrency bursts.
A critical feature of this project is the simulation of real-world network conditions. Distributed systems rarely operate on zero-latency networks. To model this, the system introduces a configurable delay mechanism:
- Variable Latency: A random delay between
MIN_DELAYandMAX_DELAYis injected into every replication request. - Orchestration: The cluster consists of 6 distinct Docker containers (1 Leader, 5 Followers) networked together via Docker Compose.
- Communication: All inter-node communication occurs over a REST API using JSON payloads.
The system is containerized for easy deployment. Configuration is handled entirely via environment variables, allowing you to alter the consistency requirements without changing the codebase.
| Variable | Description |
|---|---|
WRITE_QUORUM |
The number of follower acknowledgments required before a write is successful. |
MIN_DELAY |
The minimum artificial latency (in seconds) for network simulation. |
MAX_DELAY |
The maximum artificial latency (in seconds) for network simulation. |
- docker-compose.yml
- Dockerfile
- README.md
- requirements.txt
- server.py
- test.py
######## 1. KeyValueStore Class Thread-safe in-memory storage with asyncio locks:
class KeyValueStore:
def __init__(self):
self.store: Dict[str, str] = {}
self.lock = asyncio.Lock()
async def get(self, key: str) -> str:
async with self.lock:
return self.store.get(key)
async def set(self, key: str, value: str) -> None:
async with self.lock:
self.store[key] = value######## 2. ReplicationManager Class Handles concurrent replication with network delay simulation:
class ReplicationManager:
async def ReplicateToFollower(self, follower_url: str, operation: Dict) -> bool:
if self.MaxDelay > 0:
delay = random.uniform(self.min_delay, self.MaxDelay)
await asyncio.sleep(delay)
async with self.session.post(f"{follower_url}/replicate", json=operation) as response:
return response.status == 200
async def replicate(self, operation: Dict) -> int:
tasks = [self.ReplicateToFollower(url, operation) for url in self.FollowerURLs]
results = await asyncio.gather(*tasks, ReturnExceptions=True)
return sum(1 for r in results if r is True)######## 3. Semi-Synchronous Write Handler
async def HandleSet(self, request):
data = await request.json()
key = data.get('key')
value = data.get('value')
await self.store.set(key, value)
ReplicatedCount = await self.ReplicationManager.replicate({
"operation": "set",
"key": key,
"value": value
})
if ReplicatedCount < self.write_quorum:
return web.json_response({
"error": "Write quorum not met",
"replicated": ReplicatedCount,
"required": self.write_quorum
}, status=500)
return web.json_response({
"success": True,
"key": key,
"value": value,
"replicated": ReplicatedCount
})docker-compose up --build -d
docker-compose ps
docker logs kv-leader
docker logs kv-follower1docker-compose downcurl http://localhost:8080/healthResponse:
{
"status": "healthy",
"role": "leader",
"timestamp": "2025-11-30T12:34:56.789012"
}curl -X POST http://localhost:8080/set \
-H "Content-Type: application/json" \
-d '{"key": "username", "value": "Emma"}'Success Response:
{
"success": true,
"key": "username",
"value": "Emma",
"replicated": 5
}Quorum Failure Response (HTTP 500):
{
"error": "Write quorum not met",
"replicated": 2,
"required": 3
}curl http://localhost:8080/get/username
curl http://localhost:8081/get/usernameResponse:
{
"key": "username",
"value": "Emma"
}curl http://localhost:8080/allResponse:
{
"data": {
"username": "Emma",
"email": "Emma@example.com",
"count": "42"
},
"count": 3
}curl -X DELETE http://localhost:8080/delete/usernameResponse:
{
"success": true,
"key": "username",
"existed": true,
"replicated": 5
}Write on leader, read from follower to verify replication:
curl -X POST http://localhost:8080/set \
-H "Content-Type: application/json" \
-d '{"key": "test", "value": "replicated"}'
curl http://localhost:8081/get/test
curl http://localhost:8082/get/test
curl http://localhost:8083/get/test The integration test (test.py) performs comprehensive testing:
- Quorum Configuration: Tests with WRITE_QUORUM values 1, 2, 3, 4, 5
- Concurrent Writes: Makes ~100 writes (10 keys × 10 writes) concurrently (10 at a time)
- Performance Measurement: Records latency for each write operation
- Consistency Verification: Checks if all replicas match the leader
- Visualization: Generates plot of Write Quorum vs. Average Latency
pip install aiohttp matplotlib
python test.pyThe test generates a plot showing the relationship between write quorum and average latency:
With network delays configured as [0ms, 1000ms], the following latency pattern emerges:
| Write Quorum | Avg Latency | Min Latency | Max Latency | Success Rate |
|---|---|---|---|---|
| 1 | ~250ms | ~50ms | ~450ms | 100% |
| 2 | ~400ms | ~100ms | ~600ms | 100% |
| 3 | ~550ms | ~200ms | ~800ms | 100% |
| 4 | ~700ms | ~400ms | ~950ms | 100% |
| 5 | ~850ms | ~600ms | ~1100ms | 100% |
The Pattern:
Quorum 1: ████ (Wait for fastest) → Low latency
Quorum 3: ████████ (Wait for 3rd fastest) → Medium latency
Quorum 5: ██████████████ (Wait for all) → High latency
Why This Happens:
- Concurrent Replication: The leader sends replication requests to ALL 5 followers in parallel
- Random Network Delays: Each follower receives its request after a random delay [0ms, 1000ms]
- Waiting for Nth Confirmation: The leader must wait for the Nth fastest follower (where N = quorum)
Example Timeline:
Time --> 0ms 200ms 400ms 600ms 800ms 1000ms
| | | | | |
Follower1: ████ (responds at 200ms)
Follower2: ██████████ (responds at 500ms)
Follower3: ████████████ (responds at 600ms)
Follower4: ████████████████ (responds at 800ms)
Follower5: ████████████████████ (responds at 1000ms)
Quorum 1: Wait until 200ms (1st confirmation) FAST
Quorum 3: Wait until 600ms (3rd confirmation) MEDIUM
Quorum 5: Wait until 1000ms (5th confirmation) SLOW
Key Insights:
- Order Statistics: With quorum N, we're waiting for the Nth order statistic of the response times
- Probability Effect: Higher quorum = higher probability of waiting for slower followers
- Trade-off: This is the fundamental Consistency vs. Availability trade-off
After completing all writes, the test verifies that replicas match the leader:
================================================================================
DATA CONSISTENCY CHECK
================================================================================
Leader: 10 keys
Follower 1: 10 keys
Follower 2: 10 keys
Follower 3: 10 keys
Follower 4: 10 keys
Follower 5: 10 keys
Perfect consistency - all replicas match leader!
Explanation of Perfect Consistency:
When all replicas match the leader, it indicates:
- All writes met quorum: Every write operation received sufficient confirmations
- Successful replication: All followers successfully applied the updates
- Semi-synchronous guarantees: The system ensured data reached required replicas before confirming to client
- No network partitions: All containers remained healthy and reachable
Scenario 1: Some Followers Down
With WRITE_QUORUM=3 and 2 followers down:
Writes succeed (3 out of 3 remaining followers confirm)
System still operational with degraded replication
With WRITE_QUORUM=3 and 3 followers down:
Writes fail (only 2 followers available, need 3)
System becomes unavailable for writes
Scenario 2: Leader Failure
System becomes unavailable (no leader election implemented)
Production systems would need: leader election, consensus (Raft/Paxos)
This implementation successfully demonstrates the mechanics of distributed consensus on a small scale. The experiment highlights several critical observations regarding distributed system performance:
The Cost of Consistency
There is a direct, observable correlation between the WRITE_QUORUM size and request latency. As the quorum requirement increases, the system becomes more sensitive to the "long tail" of network latency—the system is only as fast as the $N$th slowest node.
The Necessity of Concurrency
Without the concurrent broadcasting of replication requests, write latency would scale linearly with the number of followers. The asyncio implementation proves essential for maintaining acceptable performance, ensuring that replication overhead remains manageable even as the cluster size grows.
Semi-Synchronous Balance The semi-synchronous model proves to be an effective middle ground. It avoids the data-loss risks of "fire-and-forget" asynchronous replication while avoiding the total availability pitfalls of requiring 100% node consensus.