[Design] External access for ValkeyCluster (Ingress/LoadBalancer/NodePort)

[sandeepkunusoth]

External access for ValkeyCluster (Ingress/LoadBalancer/NodePort)

Summary

This is Proposal for adding external access for ValkeyCluster that exposes a cluster to topology-aware clients outside Kubernetes Cluster via one of three mechanisms — Ingress (SNI), LoadBalancer, or NodePort.

For all three types the operator creates the per-shard Services, reads back the actual allocation (NodePort number / LoadBalancer IP), advertises it to clients, and publishes every shard's external endpoint in status. The only objects a user still creates are the Gateway / TLSRoute / Ingress for the SNI case (these are ingress-implementation specific — Envoy Gateway, Istio, NGINX, OpenShift — so they stay out of the operator).

---

Why this is needed

A valkey cluster client is topology aware. It uses CLUSTER SLOTS/CLUSTER NODES to discover the shards , then connects directly to endpoint each shard advertises. Today every node runs valkey-server --cluster-announce-ip $(POD_IP), so clients discover the pod IPs of the shards - unroutable from outside the cluster.

1. Only the operator changes what a node advertises (cluster-announce-hostname + port). The cluster bus (:16379) stays on pod IPs (cluster-announce-ip $(POD_IP) untouched); the bus is never exposed externally.
2. Clients are topology-aware: seed → CLUSTER SLOTS → connect directly to each shard's advertised endpoint. So every shard must advertise an externally reachable host:port.

---

Proposed CRD spec

apiVersion: valkey.io/v1alpha1
kind: ValkeyCluster
spec:
  shards: 3

  tls:
    certificate:                      # cluster-wide; REQUIRED for type: Ingress/SNI
      secretName: my-cluster-tls

  externalAccess:
    enabled: true
    type: Ingress                    # Ingress | LoadBalancer | NodePort

    # advertisedHost — per-node hostname advertised to clients:
    #   Ingress      : REQUIRED. Per-shard SNI hostname template. Placeholders {name}{namespace}{shard}{node}
    #   LoadBalancer : OPTIONAL. If set -> announce this hostname (user DNS -> LB IP, needed for TLS SAN).
    #                  If unset -> announce the LB IP read from Service status (no DNS).
    #   NodePort     : OPTIONAL. host / DNS that fronts the nodes (node IPs).
    advertisedHost: "{name}-{shard}.example.com"

    # advertisedPort — client port for Ingress / LoadBalancer (default 6380).
    advertisedPort: 6380

    nodePort:
      # OPTIONAL preferred base. Operator requests startPort+shardIndex; if taken it falls back to
      # dynamic allocation. The ACTUAL allocated nodePort is what gets advertised + reported in status.
      startPort: 31200

    # Optional annotations applied to the operator-created Services (e.g. internal LB).
    serviceAnnotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Validations: advertisedHost required for Ingress; Ingress requires spec.tls; advertisedHost optional for LoadBalancer and NodePort.

---

What the operator creates / announces / surfaces per type

Type	Operator creates	Advertises to clients (cluster-announce-hostname or cluster-announce-ip + port)	Published in status.externalAccess.endpoints[]	User creates routes (outside operator)
Ingress (SNI)	one ClusterIP Service per shard (route backend)	cluster-announce-hostname: advertisedHost (per-shard SNI hostname) + cluster-announce-tls-port: advertisedPort (6380)	per-shard SNI hostname + port; point DNS at Ingress LoadBalancer IP	Gateway + TLSRoute (or Ingress ssl-passthrough), CNAME to ingress LB IP
LoadBalancer	one type=LoadBalancer Service per shard; waits for cloud to assign IP, then uses it	cluster-announce-ip: LB IP from Service status (default) or cluster-announce-hostname: advertisedHost if set; + cluster-announce-tls-port or cluster-announce-port: advertisedPort based on tls enabled or not	per-shard LB IP (or hostname) + port	nothing — optionally CNAME to LB IP if using advertisedHost
NodePort	one type=NodePort Service per shard (requests startPort+i; falls back to dynamic if taken); uses the actual allocated port from Service spec	(cluster-announce-ip : node IP/ cluster-announce-hostname : advertisedHost fronting nodes) + cluster-announce-tls-port or cluster-announce-port: actual allocated nodePort per shard	per-shard nodeIP/advertisedHost + actual nodePort	nothing — optional CNAME/DNS to node IP if advertisedHost is set

---

Status CRD updates

status:
  externalAccess:
    type: LoadBalancer
    ready: true
    endpoints:
      - shard: 0
        service: my-cluster-0
        host: 10.29.241.20    # LB IP (LoadBalancer) | SNI host (Ingress) | advertisedHost/VIP (NodePort)
        port: 6380             # advertisedPort (Ingress/LB) | actual nodePort (NodePort)
        address: "10.29.241.20:6380"
      - shard: 1
        service: my-cluster-1
        host: 10.29.241.21
        port: 6380
        address: "10.29.241.21:6380"
      - shard: 2
        service: my-cluster-2
        host: 10.29.241.22
        port: 6380
        address: "10.29.241.22:6380"

Per type, endpoints[] reports respectively: Ingress → the per-shard SNI hostnames (clients/DNS point these at the Gateway IP); LoadBalancer → the per-shard LB IPs; NodePort → the advertisedHost or nodeIP + the actual nodePort per shard.

---

Operator changes when external access is enabled

1. Reconcile Services for the chosen type (ClusterIP / LoadBalancer / NodePort), one per shard, selecting valkey.io/shard-index: "<i>".
2. Read back allocations: NodePort → spec.ports[].nodePort; LoadBalancer → poll status.loadBalancer.ingress[] until an IP/hostname appears (requeue meanwhile); Ingress → resolve advertisedHost template (no wait).
3. Resolve per-node advertisement and set it on each ValkeyNode → config valkey-server with (--cluster-announce-hostname or --cluster-announce-ip) + --cluster-announce-(tls-)port. cluster-preferred-endpoint-type hostname in config based on service type.
4. Publish status.externalAccess with every shard's resolved endpoints/IPs/ports based on service type.

within operator we need to create services based on the type user selects in spec externalAccess.type

A- Ingress

apiVersion: v1
kind: Service                         # one ClusterIP per shard 0,1,2... for route backend
metadata: { name: my-cluster-0, namespace: test }
spec:
  selector: { valkey.io/cluster: my-cluster, valkey.io/shard-index: "0" }
  ports: [{ name: tcp-valkey, port: 6380, targetPort: 6379 }]
# ...repeat for shard 1, 2.
---
# extra service to point to all shards so that client can initially connect with any shard
apiVersion: v1
kind: Service         
metadata: { name: my-cluster, namespace: test }
spec:
  selector: { valkey.io/cluster: my-cluster}
  ports: [{ name: tcp-valkey, port: 6380, targetPort: 6379 }]

B — LoadBalancer (one LB per shard)

apiVersion: v1
kind: Service
metadata:
  name: my-cluster-0  # one LB per shard 0,1,2...
  namespace: test
  # annotations: { service.beta.kubernetes.io/azure-load-balancer-internal: "true" }
spec:
  type: LoadBalancer
  selector: { valkey.io/cluster: my-cluster, valkey.io/shard-index: "0" }
  ports: [{ name: tcp-valkey, port: 6379, targetPort: 6379 }]
# ...repeat for shard 1, 2. DNS: my-cluster-{shard}.example.com -> that shard's LB IP.

C — NodePort (port-per-shard)

apiVersion: v1
kind: Service
metadata:
  name: my-cluster-0  # one nodeport per shard 0,1,2...
  namespace: test
  labels: { valkey.io/cluster: my-cluster, valkey.io/external-access: shard }
spec:
  type: NodePort
  selector: { valkey.io/cluster: my-cluster, valkey.io/shard-index: "0" }
  ports: [{ name: tcp-valkey, port: 6379, targetPort: 6379, nodePort: 31200 }]  # startPort + 0
# shard 1 -> nodePort 31201, shard 2 -> nodePort 31202.

---

The three approaches

#	Type	External IPs	resources created by operator	DNS/CNAME	Port	TLS (spec.tls)
A	Ingress (SNI)	1 shared	N+1 services (one per shard, one for all shards)	per-shard host → 1 IP	same port	required
B	LoadBalancer	N (per shard)	N+1 services (one per shard, one for all shards)	per-host → LB IP	same port	optional
C	NodePort	0 new (node IPs)	N+1 services (one per shard, one for all shards)	single host (or node IP)	per-shard startPort+i	optional

---

Client-side Routes example (only needed for Ingress/SNI)

For LoadBalancer and NodePort the operator creates the Services — users create nothing (optionally DNS). For Ingress the user provides the SNI routes and points DNS at it, as these are implementation-specific (Envoy Gateway vs Isitio vs NGINX vs Openshift Routes etc.)

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: valkey-gw, namespace: test }
spec:
  gatewayClassName: envoy             # or istio
  listeners:
    - name: tls-passthrough
      protocol: TLS
      port: 6380
      tls: { mode: Passthrough }
      allowedRoutes: { kinds: [{ kind: TLSRoute }] }
---
# one TLSRoute per shard; hostnames come from status.externalAccess.endpoints[].host
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata: { name: my-cluster-0, namespace: test }
spec:
  parentRefs: [{ name: valkey-gw }]
  hostnames: ["my-cluster-0.example.com"]    # SNI
  rules:
    - backendRefs: [{ name: my-cluster-0, port: 6379 }]  # operator created ClusterIP svc for shard 0

DNS/CNAME: for each shard create a CNAME record my-cluster-{shard}.example.com -> Gateway LB IP. NGINX alternative: an Ingress per shard with nginx.ingress.kubernetes.io/ssl-passthrough: "true". extra CNAME my-cluster.example.com -> Gateway LB IP. This allows clients to connect to my-cluster.example.com:6380, not per shard.

---

References

• Strimzi operator: per-broker Services (operator-created), advertised addresses populated from Service status; Ingress is TLS-passthrough (SNI), always TLS; NodePort uses port offsets with host overrides.
• Redis Enterprise Operator : exposes DB endpoints via ingress with methods nginx, istio, openshiftRoute.
• [issues](Valkey cluster in "cluster mode" will not work in Red Hat OpenShift with routes #223, (feat) External access for ValkeyCluster (externalAccess.type: None | LoadBalancer | Ingress; maybe Proxy in future) #58 )
• [Idea] External access for Valkey #77

Contribution

• I'm willing to implement enahncements needed for supporting ingress.

[scrothers]

I took this design forward and implemented it under #276, following the direction sketched here: per-shard Services, reading back allocated ports, and publishing endpoints in status, plus a Gateway API path.

The core idea that made it safe is that Valkey 9.0 separates the node-to-node announce settings (cluster-announce-ip/-port/-bus-port) from the client-facing ones (cluster-announce-hostname, cluster-announce-client-port/-client-tls-port, and the cluster-wide cluster-preferred-endpoint-type), so the cluster bus stays on internal pod IPs and the operator's reconciler is untouched while clients are handed an external endpoint.

It ships as a stack of small, backward-compatible PRs behind an opt-in externalAccess block:

• #277 adds the API and announces human-readable node names.
• #278 creates one Service per shard with each node pinned to its own port (NodePort allocated by Kubernetes and read back into status.externalEndpoints, or LoadBalancer).
• #279 announces per-shard hostnames (<prefix>-<shardIndex>.<domain>, also handy for TLS SNI).
• #280 flips preferredEndpointType: hostname so MOVED/ASK redirects resolve cross-shard to a routable endpoint.
• #281 adds the optional Gateway API path (one TCPRoute per node attached to a user-owned Gateway, inert when the Gateway API CRDs aren't installed).

The SNI/TLSRoute routing you raised isn't in this first cut, so it would be a natural follow-up. Feedback welcome on #276.