Creating KeycloakClient while Keycloak server is down

[zaikinlv]

We are creating Keycloak clients with keycloak-controller for existing Keycloak
server (7.0.0). For that to happen, first we're creating Keycloak object
pointing to the Keycloak server(url points to internal kubernetes service here,
which works great by the way):

apiVersion: k8s.kiwigrid.com/v1beta1
kind: Keycloak
metadata:
  name: fully-configured-keycloak
  namespace: keycloak
spec:
  url: http://keycloak-http.keycloak.svc.cluster.local/auth
  realm: master
  clientId: admin-cli
  username: keycloak
  passwordSecretNamespace: keycloak
  passwordSecretName: keycloak-http
  passwordSecretKey: password

Output shows it is connected:

INFO  com.kiwigrid.keycloak.controller.keycloak.KeycloakController Connected to fully-configured-keycloak in version 7.0.0.

While keycloak is availabe/ready, clients are created succefully and
available via keycloak UI. Example of KeycloakClient object:

apiVersion: k8s.kiwigrid.com/v1beta1
kind: KeycloakClient
metadata:
  name: client-example
  namespace: keycloak
spec:
  keycloak: fully-configured-keycloak
  realm: myrealm
  clientId: client-example
  clientType: confidential
  directAccessGrantsEnabled: false
  standardFlowEnabled: true
  implicitFlowEnabled: false
  secretNamespace: keycloak
  secretName: clinet-example-client-secret
  secretKey: secret
  mapper:
  - name: example-service-audience
    protocolMapper: oidc-audience-mapper
    config:
      claim.name: audience
      access.token.claim: "true"
      id.token.claim: "true"
      included.custom.audience: my-service

Error occurs when we are creating KeycloakClient, while keycloak is not
available. There may be several reasons for that e.g. keycloak pod restart
due to upgrade procedures either of keycloak itself or other pieces of
infrastructure.

The error itself is obvious - keycloak is not avilable during KeycloakClient
object creation:

09:00:35.689 16.0.1/... ERROR     com.kiwigrid.keycloak.controller.client.ClientController Failed to ADDED resource keycloak/client-example.
javax.ws.rs.ProcessingException: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:599)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:436)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:148)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
	at com.sun.proxy.$Proxy59.toRepresentation(Unknown Source)
	at com.kiwigrid.keycloak.controller.client.ClientController.lambda$realm$2(ClientController.java:159)
	at java.base/java.util.Optional.filter(Optional.java:223)
	at com.kiwigrid.keycloak.controller.client.ClientController.realm(ClientController.java:157)
	at com.kiwigrid.keycloak.controller.client.ClientController.apply(ClientController.java:46)
	at com.kiwigrid.keycloak.controller.client.ClientController.apply(ClientController.java:26)
	at com.kiwigrid.keycloak.controller.KubernetesController.eventReceived(KubernetesController.java:67)
	at com.kiwigrid.keycloak.controller.KubernetesController.eventReceived(KubernetesController.java:14)
	at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:232)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:231)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:191)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.extractors.BodyEntityExtractor.extractEntity(BodyEntityExtractor.java:60)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:150)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
	at com.sun.proxy.$Proxy47.refreshToken(Unknown Source)
	at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:106)
	at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:71)
	at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:64)
	at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:586)
	... 24 common frames omitted

What we are missing here is a sort of resilience - while Keycloak is down,
KeycloakClient object is created by controller, but no actual client is created
in the Keycloak even when it is up again. The only fix we've found is either
to restart keycloak-controller or to delete/apply KeycloakClient again once
Keycloak server is ready.

Would be nice to have a feature when keycloak-controller tries to recover
connection to keycloak on attempt to create KeycloakClient, if keycloak is not
available at that moment. Same approach is used when controller tries to connect
to keycloak on start and logging WARN each 60s:

WARN  com.kiwigrid.keycloak.controller.keycloak.KeycloakController Connecting to fully-configured-keycloak failed: Keycloak returned 503 with: no healthy upstream

What do you think about this / how you're solving this?

[axdotl]

Hi @zaikinlv and sorry for the delayed response.
You're right, the controller is not resilient in this case. But actually we were not aware of this, as we never faced an unavailable keycloak (we're running HA setups and performing RollingUpdates).

We would be glad to receive a PR for this!