# Design Proposal: Persistent Volume Support for Valkey Operator

[umar-riswan-ap]

Summary

This proposal introduces persistent volume support and automatic online volume expansion to the Valkey Operator, enabling data persistence across pod restarts and dynamic storage scaling without downtime. The implementation transitions from Deployments to StatefulSets with PersistentVolumeClaims, allowing production-ready Valkey clusters with durable storage and declarative capacity management.

Motivation

Currently, the Valkey Operator manages ephemeral clusters using Deployments with emptyDir volumes. This approach has significant limitations:

• No data persistence: All data is lost when pods restart or are rescheduled
• Production constraints: Cannot support use cases requiring durable storage (cache persistence, session storage, etc.)
• Manual recovery: Cluster restarts require manual data reloading and cache warmup
• No storage management: No way to increase storage capacity as data grows
• Resource waste: Over-provisioning required to avoid OOM issues

This limits the operator's usability for:

• Production environments requiring data durability
• Applications with expensive cache warmup times
• Clusters that need to scale storage independently of compute
• Enterprise deployments with compliance requirements

Proposed Solution

With this change, users will be able to:

1. Enable persistent storage declaratively via the ValkeyCluster CRD
2. Survive pod restarts - data persists across pod recreation, node failures, and cluster upgrades
3. Expand storage dynamically - increase volume size without downtime or manual intervention
4. Use production storage - leverage cloud provider storage (AWS EBS, GCE PD, Azure Disk) with proper provisioning
5. Maintain backward compatibility - ephemeral deployments continue to work as before

Detailed Design

API Changes

Add a new StorageSpec to the ValkeyCluster CRD:

apiVersion: valkey.io/v1alpha1
kind: ValkeyCluster
metadata:
  name: valkey-persistent
spec:
  shards: 3
  replicas: 1
  storage:
    enabled: true                    # Enable persistent storage
    storageClassName: "gp3"          # StorageClass to use (optional)
    size: "10Gi"                     # Initial volume size
    accessModes:                     # PVC access modes (optional)
      - ReadWriteOnce
  volumePermissions: true            # Fix volume ownership (optional)

New Fields:

• storage.enabled (bool): Toggle persistent storage (default: false, maintains backward compatibility)
• storage.storageClassName (string, optional): StorageClass name (uses cluster default if omitted)
• storage.size (string): Volume size in Kubernetes quantity format (e.g., "1Gi", "100Mi")
• storage.accessModes ([]string, optional): PVC access modes (default: ["ReadWriteOnce"])
• volumePermissions (bool, optional): Enable init container to fix volume permissions (default: false)

New Status Conditions:

ReasonVolumeExpanding = "VolumeExpanding"  // Expansion in progress
ReasonVolumeExpanded  = "VolumeExpanded"   // Expansion completed

Architecture

Transition: Deployment → StatefulSet

When storage.enabled=true, the operator creates StatefulSets instead of Deployments:

Component	Ephemeral (Current)	Persistent (Proposed)
Workload	Deployment	StatefulSet
Storage	EmptyDir	PersistentVolumeClaim
Pod Names	Random (valkey-xyz123)	Stable (valkey-0-0)
Network	Service discovery	Stable DNS names
Data	Ephemeral	Persistent

Benefits of StatefulSets:

• Stable, predictable pod names
• Automatic PVC lifecycle management
• Ordered scaling and updates
• Stable network identities

How It Works

1. Cluster Creation with Persistent Storage

When a user creates a ValkeyCluster with storage.enabled=true:

1. Operator generates StatefulSet per shard (instead of Deployment)
2. StatefulSet includes VolumeClaimTemplate
3. Kubernetes provisions PVCs for each pod
4. Storage provider (e.g., AWS EBS) creates actual volumes
5. Pods mount persistent volumes at /data
6. Init container fixes volume ownership if volumePermissions=true
7. Valkey starts and writes data to persistent volume

2. Volume Expansion (Zero Downtime)

When a user increases storage.size:

# Example: Expand from 10Gi to 50Gi
kubectl patch valkeycluster valkey-production --type='json' \
  -p='[{"op": "replace", "path": "/spec/storage/size", "value": "50Gi"}]'

Flow:

1. Reconciler detects size change in spec
2. Validates StorageClass has allowVolumeExpansion: true
3. Patches all PVCs with new size request
4. Kubernetes triggers volume expansion via CSI driver
5. Storage provider resizes underlying volumes (30-60s for AWS EBS)
6. Node filesystem automatically expands
7. Operator monitors PVC conditions (Resizing → Ready)
8. Updates ValkeyCluster status condition to VolumeExpanded
9. Cluster remains fully operational throughout

Timeline: ~30-60 seconds per volume (tested on AWS EBS gp3)

3. Components Involved

New Files:

• internal/controller/volume_expansion.go - Core expansion logic
  • handleVolumeExpansion() - Detects size changes and patches PVCs
  • verifyStorageClassSupportsExpansion() - Validates StorageClass
  • checkVolumeExpansionStatus() - Monitors expansion progress

Modified Files:

• api/v1alpha1/valkeycluster_types.go - Add StorageSpec and status reasons
• internal/controller/valkeycluster_controller.go - Integrate expansion checks
• config/rbac/role.yaml - Add PVC and StorageClass permissions

RBAC Permissions (New):

- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list", "watch"]

User Interaction

Creating a Persistent Cluster

apiVersion: valkey.io/v1alpha1
kind: ValkeyCluster
metadata:
  name: valkey-prod
spec:
  shards: 3
  replicas: 2
  storage:
    enabled: true
    storageClassName: "gp3"
    size: "100Gi"
  volumePermissions: true

kubectl apply -f valkey-cluster.yaml

Monitoring Storage Status

# Check PVCs
kubectl get pvc -l app.kubernetes.io/instance=valkey-prod

# Check cluster status
kubectl get valkeycluster valkey-prod -o yaml

# Watch events
kubectl get events --sort-by='.lastTimestamp' | grep VolumeExpan

Expanding Storage

# Method 1: Patch command
kubectl patch valkeycluster valkey-prod --type='json' \
  -p='[{"op": "replace", "path": "/spec/storage/size", "value": "200Gi"}]'

# Method 2: Edit and apply
kubectl edit valkeycluster valkey-prod  # Change size: "200Gi"

# Monitor expansion
kubectl get pvc -w

Storage Provider Requirements

Prerequisites:

• Kubernetes 1.25+
• StorageClass with allowVolumeExpansion: true
• CSI driver supporting online expansion

Example StorageClass (AWS EBS):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-expandable
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

API Changes

No response

User Stories

No response

Alternatives Considered

No response

Backward Compatibility

No response

Testing Strategy

No response

Open Questions

No response

References

PR raised for the same : #85

Implementation

• I'm willing to implement this design and submit a PR
• This design has been discussed with maintainers

[jdheyburn]

Thanks for raising this! I have some comments on the design.

• The automated switch from Deployments to StatefulSets when storage is enabled worries me a bit, what happens to existing clusters? Changing the underlying workload type in-place is non-trivial and could cause downtime or data loss.
  • We are supporting both Deployment and StatefulSet as we want the offering to be flexible depending on the requirements of the user
• Also worth noting that Deployments can mount PersistentVolumes too, so StatefulSets aren't strictly necessary just to get persistence. The main things StatefulSets add are stable pod identities and ordered rollouts. If those aren't needed here, a Deployment-based approach might be simpler.
• One other angle on volumes: even if we treat data as "ephemeral", there's still a good case for attaching a volume so Valkey can write an RDB snapshot on shutdown and reload it on startup. That way a restarting pod can do a partial resync rather than a full one, which can make a big difference in rollout time.

Some of these problems have already been considered in the ValkeyNode CRD, whereby setting these fields would:

• Create a Volume regardless of the management type (Deployment or StatefulSet)
• Create the Deployment/StatefulSet, attaching the volume to the workload
• For volume increases, the Operator can then adjust the volume size, ideally automatically when it reaches a % use threshold.

I'd like to hold off on this implementation until the ValkeyNode work lands first, I'm hoping to get to that next week. I'll leave volume persistence out of that PR so we have room to keep discussing the right approach in this discussion. How does that sound?