Kubernetes quickstart: replicas:2 creates non-clustering pods (misleading)

[maxt1907]

Description:
Official kubernetes/keycloak.yaml deploys 2 replicas that never form a cluster (https://www.keycloak.org/getting-started/getting-started-kube)

Issue:

1. YAML: replicas: 2
2. Docs: "two replicas to allow for rolling updates"
3. Result: 2x Singletons, Cluster Health DOWN, NoRouteToHostException spam

keycloak-0-12955: failed sending message to 10.42.0.32:7800: java.net.NoRouteToHostException: No route to host

I tried specific network policies which could account for the problem. Also I tried a modified version of the keycloak-discovery service to point directly to the affected ports for multi replica use.

I think it maybe a configuration issue from my side but I tried troubleshooting it without success.

Expected: Working 2-node cluster oder replicas: 1 im YAML
Actual: Misleading "cluster-ready" appearance

Besides that, I configured my own Keycloak statefulset which does not work correctly either.

apiVersion: v1
kind: Service
metadata:
name: keycloak
labels:
app: keycloak
spec:
ports:
- protocol: TCP
port: 8443
targetPort: https
name: https
selector:
app: keycloak
type: ClusterIP

apiVersion: v1
kind: Service
metadata:
labels:
app: keycloak
name: keycloak-discovery
spec:
selector:
app: keycloak
clusterIP: None
type: ClusterIP

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: keycloak
labels:
app: keycloak
spec:
serviceName: keycloak-discovery
replicas: 2
selector:
matchLabels:
app: keycloak
template:
metadata:
labels:
app: keycloak
spec:
containers:
- name: keycloak
image: quay.io/keycloak/keycloak:26.4.7
volumeMounts:
- name: keycloak-tls
mountPath: /etc/keycloak/tls
readOnly: true
args: ["start"]
env:
- name: KC_BOOTSTRAP_ADMIN_USERNAME
valueFrom:
secretKeyRef:
name: keycloak-admin-account
key: admin-username
- name: KC_BOOTSTRAP_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-admin-account
key: admin-password
- name: 'KC_PROXY_HEADERS'
value: 'xforwarded'
- name: 'KC_HTTP_ENABLED'
value: "true"
- name: 'KC_HOSTNAME'
value: 'https://example.com'
- name: 'KC_HOSTNAME_ADMIN'
value: 'https://example.com'
- name: 'KC_HTTPS_PORT'
value: '8443'
- name: 'KC_HTTPS_CERTIFICATE_FILE'
value: '/etc/keycloak/tls/tls.crt'
- name: 'KC_HTTPS_CERTIFICATE_KEY_FILE'
value: '/etc/keycloak/tls/tls.key'
- name: 'KC_HOSTNAME_STRICT'
value: 'false'
- name: 'KC_HEALTH_ENABLED'
value: 'true'
- name: 'KC_CACHE'
value: 'ispn'
- name: 'KC_CACHE_STACK'
value: 'kubernetes'
- name: 'POD_IP'
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: KC_CACHE_EMBEDDED_NETWORK_BIND_ADDRESS
value: '$(POD_IP)'
- name: 'KC_DB_URL'
value: 'jdbc:postgresql://keycloak-db.homelab.svc.cluster.local:5432/keycloak'
- name: 'KC_DB'
value: 'postgres'
- name: 'KC_HTTP_MANAGEMENT_SCHEME'
value: 'http'
- name: 'KC_DB_PASSWORD'
valueFrom:
secretKeyRef:
name: keycloak-postgres-credentials
key: postgres-password
- name: 'KC_DB_USERNAME'
valueFrom:
secretKeyRef:
name: keycloak-postgres-credentials
key: postgres-user
ports:
- name: http
containerPort: 8080
- name: https
containerPort: 8443
- name: jgroups
containerPort: 7800
- name: jgroups-fd
containerPort: 57800
startupProbe:
httpGet:
path: /health/started
port: 9000
initialDelaySeconds: 80
periodSeconds: 10
failureThreshold: 600
readinessProbe:
httpGet:
path: /health/ready
port: 9000
initialDelaySeconds: 80
periodSeconds: 10
failureThreshold: 600
livenessProbe:
httpGet:
path: /health/live
port: 9000
initialDelaySeconds: 80
periodSeconds: 10
failureThreshold: 600
resources:
limits:
cpu: 2000m
memory: 2000Mi
requests:
cpu: 500m
memory: 1700Mi
volumes:
- name: keycloak-tls
secret:
secretName: keycloak-tls

I appreciate your help.

[shawkins]

I could not reproduce what you are seeing on minikube. Can you provide exactly what flavor and version of Kubernetes you are running on?

In your statefulset note that the kubernetes cache stack is deprecated.

[maxt1907]

Hi, thank you for your quick answer.

I am currently running k3s version v.133.6+k3s1 (b5847677) on AlmaLinux (kernel 5.14.0-611.13.1.el9_7.x86_64).

I solved the issue. It was caused by the firewalld daemon, which blocked the network traffic in the pod network.

The official k3s guide states that you need specific firewall rules within firewalld.

I applied the following rules according to the guide (https://docs.k3s.io/installation/requirements?os=rhel)

firewall-cmd --permanent --add-port=6443/tcp #apiserver
firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 #pods
firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 #services
firewall-cmd --reload