Skip to content

Commit

Permalink
Use standard liveness and readiness settings as KC 22.0.5 is released
Browse files Browse the repository at this point in the history
  • Loading branch information
ahus1 committed Oct 30, 2023
1 parent b56a4fc commit 2e2648e
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 19 deletions.
11 changes: 3 additions & 8 deletions doc/kubernetes/modules/ROOT/pages/running/concepts/threads.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,9 @@ Future version of Keycloak will have better means to handle that: https://github
[#probes]
=== Probes

All health probes, including liveness and readiness probes, are handled in the Quarkus executor worker pool.
With all requests being queued, also the liveness probe is queued, and is therefore slow.
During Keycloak Infinispan view updates for members leaving and rebalancing, there is an increased latency for all requests, observed with up to 10 seconds.
In a high-load or even overload scenario, the probes will be queued in the executor thread pool, and won't return in time.
When requests queue up in Keycloak, also Readiness and Liveness probes are delayed, which might trigger failure detection in Kubernetes and will lead to Pod restarts in overload or load-shedding situations.
With load shedding activated, when requests are rejected from the executor thread pool, failing readiness probes will lead to Pods not receiving any load for a period or time, and with failing liveness probes the Pods will eventually be restarted.
This is tracked in https://github.com/keycloak/keycloak/issues/22109[keycloak#22109].
For the time being, consider a longer timeout for the probes to survive spikes in the delay, or disabling the liveness probe to avoid Pod restarts.
All health probes, including liveness and readiness probes, are handled in the Quarkus executor worker pool by default.
Starting with Keyloak 22.0.5, there is a workaround in place to have the liveness probe non-blocking (see: https://github.com/keycloak/keycloak/issues/22109[keycloak#22109]).
Future version of Keycloak and Quarkus plan to have other probes also being non-blocking.

=== OS Resources

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,12 @@ See xref:./concepts/threads.adoc[] for details.
<4> The JVM options set additional parameters:
* `jgroups.thread_dumps_threshold` ensures that a log message "`thread pool is full`" appears once the JGroup thread pool is full for the first time. See xref:./concepts/threads.adoc[] for details.
* Adjust the memory settings for the heap.
<5> Probes will be delayed when requests queue up under load, and fail when a load is shed.
So the best way to run Keycloak in Kubernetes would be to disable those probes, for now.
See xref:./concepts/threads.adoc#probes[Probes] for details.

[NOTE]
====
Previous versions of this documentation recommended to disable the liveness and readiness probes.
With https://github.com/keycloak/keycloak/issues/22109[keycloak#22109] being available in Keycloak 22.0.5, this is no longer recommended.
====

== Optional: Load shedding

Expand Down
16 changes: 8 additions & 8 deletions provision/minikube/keycloak/templates/keycloak.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -257,15 +257,15 @@ spec:
limits:
{{ if .Values.cpuLimits }}cpu: "{{ .Values.cpuLimits }}"{{end}}
{{ if .Values.memoryLimitsMB }}memory: "{{ .Values.memoryLimitsMB }}M"{{end}}
readinessProbe: # <5>
exec:
command:
- 'true'
livenessProbe: # <5>
exec:
command:
- 'true'
# end::keycloak[]
# readinessProbe:
# exec:
# command:
# - 'true'
# livenessProbe:
# exec:
# command:
# - 'true'
# tag::keycloak-ispn[]
volumeMounts:
{{ if .Values.infinispan.customConfig }}
Expand Down

0 comments on commit 2e2648e

Please sign in to comment.