Investigate deployment of KeyDB

[mutambaraf]

We need to investigate how we are going to deploy KeyDB to the new cluster being created.

[mutambaraf]

---

KeyDB Three-Cluster Full Mesh Architecture

KeyDB Three-Cluster Full Mesh

Multi-Master + Active Replica — Three Clusters in Full Mesh Replication

System Overview

Three independent KeyDB clusters (PROXMOX, STFC CLOUD, ISIS VMS), each with a single KeyDB deployment, joined together in a full mesh replication topology where each cluster replicates from the other two.

┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                    3 KEYDB CLUSTERS IN FULL MESH REPLICATION                                        │
│                                                                                                     │
│     Each Cluster: Single KeyDB Instance (or local HA pair)                                          │
│     Inter-Cluster: Full Mesh Replication (multi-master + active-replica)                            │
│     Recovery: Failed cluster rejoins mesh and syncs from peers                                      │
└─────────────────────────────────────────────────────────────────────────────────────────────────────┘

                  ┌─────────────────────────────────────────────────────────────────┐
                  │         KEYDB FULL MESH REPLICATION ACROSS 3 CLUSTERS           │
                  └─────────────────────────────────────────────────────────────────┘

                                    INTER-CLUSTER FULL MESH
                                    ┌─────────────────────┐
                                    │                     │
                                    ▼                     │
                            ┌───────────────┐             │
                            │   PROXMOX     │◄────────────┤
                            │   Cluster #1  │             │
                            │               │             │
                            │  ┌─────────┐  │             │
                            │  │ KeyDB-1 │  │             │
                            │  │(master) │  │             │
                            │  └────┬────┘  │             │
                            │       │       │             │
                            └───────┼───────┘             │
                                    │                     │
                          ╔═════════╪═════════╗           │
                          ║         ▼         ║           │
                          ║   ┌───────────┐   ║           │
                          ║   │STFC CLOUD │   ║           │
                          ║   │Cluster #2 │   ║           │
                          ║   │           │   ║           │
                          ║   │┌─────────┐│   ║           │
                          ║   ││ KeyDB-2 ││   ║           │
                          ║   ││(master) ││   ║           │
                          ║   │└────┬────┘│   ║           │
                          ║   │     │     │   ║           │
                          ║   └─────┼──── ┘   ║           │
                          ║         │         ║           │
                          ║         ▼         ║           │
                          ║   ┌───────────┐   ║           │
                          ║   │ ISIS VMS  │   ║           │
                          ║   │Cluster #3 │   ║           │
                          ║   │           │   ║           │
                          ║   │┌─────────┐│   ║           │
                          ║   ││ KeyDB-3 ││   ║           │
                          ║   ││(master) ││   ║           │
                          ║   │└────┬────┘│   ║           │
                          ║   │     │     │   ║           │
                          ║   └─────┼───  ┘   ║           │
                          ║         │         ║           │
                          ║         ║         ║           │
                          ║         ║         ║           │
                          ║         ╚══════   ╝───────────┘
                          ║
                          ║  FULL MESH REPLICATION:
                          ║  • Proxmox replicates from Cloud + ISIS
                          ║  • Cloud replicates from Proxmox + ISIS
                          ║  • ISIS replicates from Proxmox + Cloud
                          ║
                          ║  Configuration per cluster:
                          ║  --multi-master yes
                          ║  --active-replica yes
                          ║  --replicaof <peer1> 6379
                          ║  --replicaof <peer2> 6379
                          ║

┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                    LOCAL APPLICATIONS                                               │
│                                                                                                     │
│  Proxmox Apps ─────► Proxmox KeyDB (local reads/writes)                                             │
│  Cloud Apps  ─────► Cloud KeyDB (local reads/writes)                                                │
│  ISIS Apps   ─────► ISIS KeyDB (local reads/writes)                                                 │
│                                                                                                     │
│  Data written to ANY cluster automatically replicates to ALL clusters                               │
└─────────────────────────────────────────────────────────────────────────────────────────────────────┘

---

Replication Topology: Inter-Cluster Full Mesh

            ┌─────────────────┐
            │    PROXMOX      │
            │   KeyDB-1       │
            └───────┬─────────┘
                    │
           ╔════════╪════════╗
           ║        │        ║
           ║        │        ║
           ▼        ▼        │
    ┌─────────────┐          │
    │ STFC CLOUD  │◄─────────┤
    │  KeyDB-2    │          │
    └──────┬──────┘          │
           │                 │
           ╲       ╱         │
            ▼     ▼          │
    ┌─────────────┐          │
    │  ISIS VMS   │◄─────────┘
    │  KeyDB-3    │
    └─────────────┘

FULL MESH CONFIGURATION:

PROXMOX KeyDB-1:
  --multi-master yes
  --active-replica yes
  --replicaof <STFC-CLOUD-IP> 6379
  --replicaof <ISIS-VMS-IP> 6379

STFC CLOUD KeyDB-2:
  --multi-master yes
  --active-replica yes
  --replicaof <PROXMOX-IP> 6379
  --replicaof <ISIS-VMS-IP> 6379

ISIS VMS KeyDB-3:
  --multi-master yes
  --active-replica yes
  --replicaof <PROXMOX-IP> 6379
  --replicaof <STFC-CLOUD-IP> 6379

---

Key Configuration

Command-Line Arguments (Required at Startup)

command:
- keydb-server
- /etc/keydb/keydb.conf
- --multi-master
- "yes"
- --active-replica
- "yes"
- --replicaof
- <PEER-CLUSTER-1-IP>
- "6379"
- --replicaof
- <PEER-CLUSTER-2-IP>
- "6379"

ConfigMap Settings

data:
  keydb.conf: |
    # Multi-Master Settings
    multi-master yes
    active-replica yes
    replica-read-only no
    
    # Memory Management
    maxmemory 1gb
    maxmemory-policy allkeys-lru
    
    # Persistence
    appendonly yes
    appendfsync everysec
    
    # Network
    bind 0.0.0.0
    port 6379
    protected-mode no

---

Data Flow Scenarios

Scenario 1: Normal Write Operation

Application in PROXMOX writes: SET key1 value1

[Proxmox App] → SET key1 value1 → [Proxmox KeyDB-1]
                                        │
                                        ├─────────────┐
                                        │             │
                                        ▼             ▼
                              [STFC CLOUD KeyDB-2]  [ISIS VMS KeyDB-3]
                                   ✓ ACK                  ✓ ACK

RESULT:
  ✓ All 3 clusters have key1=value1
  ✓ Replication latency: depends on inter-cluster network
  ✓ Read available from ANY cluster (eventual consistency)

Scenario 2: Cluster Failure and Recovery

SCENARIO: STFC CLOUD cluster goes down

T+0: STFC CLOUD KeyDB-2 becomes unreachable

T+1-5s: PROXMOX and ISIS detect KeyDB-2 failure
     └─► master_link_status: down for KeyDB-2

T+0 to T+N: PROXMOX and ISIS continue operating
     └─► Replication continues between PROXMOX ↔ ISIS
     └─► Writes to PROXMOX and ISIS replicate to each other

T+Recovery: STFC CLOUD comes back online
     └─► KeyDB-2 reconnects to PROXMOX and ISIS
     └─► Full sync from peers
     └─► Full mesh replication restored

RESULT: 
  ✓ Zero data loss (writes continued on remaining clusters)
  ✓ Automatic recovery when cluster rejoins
  ✓ Full mesh restored

Scenario 3: Network Partition

SCENARIO: ISIS VMS loses network connectivity to PROXMOX and STFC CLOUD

T+0: ISIS becomes unreachable from both other clusters

T+0 to T+Recovery:
  PROXMOX ↔ STFC CLOUD: Continue replicating (mesh intact)
  ISIS: Continues accepting local writes (isolated)

T+Recovery: Network restored

  Agent Action:
  ├─ Detect ISIS is back
  ├─ Assess consistency (sample keys)
  ├─ If mismatch: Trigger recovery sync
  └─ Resume full mesh replication

RESULT:
  ⚠ Potential data conflicts (last-write-wins)
  ✓ Mesh automatically heals

---

Known Issues and Limitations

1. Data Conflicts During Network Partitions (Critical)

• Operations like INCR, HINCRBY can cause data loss during partitions
• KeyDB uses last-write-wins semantics
• Mitigation: Only use SET/GET for critical data

2. OOM Cascading Failures (Critical)

• When one cluster crashes, remaining clusters receive all traffic
• Can cause memory pressure → OOM
• Mitigation: maxmemory + maxmemory-policy allkeys-lru configured

3. Split-Brain Scenarios

• No leader election in full mesh
• All clusters accept writes during partitions
• Mitigation: Document eventual consistency model

4. Inter-Cluster Replication Lag

• Latency depends on network between locations
• Reads immediately after writes may return stale data
• Mitigation: Document in application requirements

5. Persistent Storage

• Use PersistentVolumes for production
• Node failure with emptyDir = data loss until replication syncs

---

When to Use This Architecture

KeyDB Three-Cluster Full Mesh is suitable for:

• Multi-location deployments (Proxmox, Cloud, ISIS)
• Local read/write at each location
• Eventual consistency acceptable
• Workloads using primarily SET/GET operations
• Simple multi-master HA across locations

Consider alternatives if:

• Strong consistency is required across locations
• Heavy use of atomic operations (INCR, HINCRBY)
• Sub-100ms cross-location replication needed
• Production workloads without persistent storage

---

References

• Palark's KeyDB Multi-Master Guide
• KeyDB Documentation
• KeyDB Active Replication