External Infinispan Deployments

[pruivo]

Description

In the context of multi-site high availability deployment, this issue suggests the removal of the embedded clustered caches and using the required external Infinispan server in cross-site mode only.

This is a requirement for Active-Active deployments.

Motivation

Keeping the embedded clustered caches in sync with the external Infinispan is not simple. The current solution relies on events streaming to asynchronously update the embedded caches in both sites (sessions mostly) which is costly. Parts of the code also require all sessions to be available in the embedded caches and not fetch any updated value from external Infinispan (search for SKIP_CACHE_LOAD flag).

To simplify the architecture and Keycloak source code, the proposal is to remove those embedded cache.

The drawback focuses mainly on performance:

• Reads had a num_owners / number of Keycloak servers of finding the data locally and avoiding any network call. With the external Infinispan, a remote call is required (except if near-caching is enabled)
• Search and iteration are slower since all dataset is streamed to the Keycloak node. This will be improved in the future by using Infinispan Protostream marshaling and Queries.

There are benefits from this approach too. Besides the simplified source code, write operations would be faster since only the external Infinispan replicates data. Keycloak will be stateless, improving the startup times and, if possible by Keycloak logic, allowing different Keycloak versions to participate. Finally, upgrading Keycloak does not lose the sessions stored in the external Infinispan.

Tasks

Beta Give feedback

1. External Infinispan as cache - Part 1 #28140
   kind/enhancement +1
   
2. External Infinispan as cache - Part 2 #28648
   kind/enhancement +1
   
3. CLI options to disable encryption and authentication to external Infinispan #28750
   area/dist/quarkus kind/feature release/25.0.0 team/cross-dc +4
   
4. Remove usages of EnvironmentDependentProviderFactory.isSupported #28751
   kind/task release/25.0.0 +2
   
5. Refactor and remove deprecated Infinispan methods from DefaultInfinispanConnectionProviderFactory #28752
   kind/task release/25.0.0 +2
   
6. Remove WildFly Clustering Marshaller #29135
   kind/task release/25.0.0 +2
   
7. External Infinispan as cache - Part 3 / login failures cache #28754
   kind/enhancement team/cross-dc +2
   
8. External Infinispan as cache - Part 4 / user + client sessions online + offline #28755
   kind/enhancement team/cross-dc +2
   
9. Remove unused code from model/infinispan module #29137
   kind/task release/25.0.0 +2
   
10. Infinispan Protostream #29394
    kind/enhancement status/triage +2
    

[sventorben]

Quick question: Does this imply that also single-site deployments with multiple instances would need to set up an external infinispan instance/cluster to be able to operate or will all data be shared via the database in such a scenario?

[pruivo]

Quick question: Does this imply that also single-site deployments with multiple instances would need to set up an external infinispan instance/cluster to be able to operate or will all data be shared via the database in such a scenario?

@sventorben It doesn't.
The single-site deployment will use the database but, the users may choose to use an external Infinispan if they want to make the Keycloak nodes stateless (for example, for auto-scaling deployments).

[u2bo]

Do you support the keycloak which 3-node cluster Pods to two k8s clusters and uses fixed IP addresses for communication?

[u2bo]

Does the keycloak operator support Infinispan high availability deployment only? How can I change the support if I am using k8s sts?

[pruivo]

Do you support the keycloak which 3-node cluster Pods to two k8s clusters and uses fixed IP addresses for communication?

Yes. It is mostly transparent for the Keycloak side but, the external Infinispan needs to be configured with cross-site enabled. A single external Infinispan shared between the Keycloak cluster also works, but it will be a single point of failure.

Does the keycloak operator support Infinispan high availability deployment only? How can I change the support if I am using k8s sts?

You need to configure the trust store because Keycloak uses TLS to communicate with the external Infinispan. Then, configure the credentials for access, as described in the documentation, converting the Keycloak Operator option into the stateful set environment variables.

For the external Infinispan, you need to install the Infinispan operator as documented here

[u2bo]

Do you support the keycloak which 3-node cluster Pods to two k8s clusters and uses fixed IP addresses for communication?

Yes. It is mostly transparent for the Keycloak side but, the external Infinispan needs to be configured with cross-site enabled. A single external Infinispan shared between the Keycloak cluster also works, but it will be a single point of failure.

Does the keycloak operator support Infinispan high availability deployment only? How can I change the support if I am using k8s sts?

You need to configure the trust store because Keycloak uses TLS to communicate with the external Infinispan. Then, configure the credentials for access, as described in the documentation, converting the Keycloak Operator option into the stateful set environment variables.

For the external Infinispan, you need to install the Infinispan operator as documented here

i used it and config but not ready! lost some config?
the keycloak log

2024-05-29 06:23:55,759 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) example-kc-0-62364: no members discovered after 2002 ms: creating cluster as coordinator
2024-05-29 06:23:55,768 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [example-kc-0-62364|0] (1) [example-kc-0-62364]
2024-05-29 06:23:55,821 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `example-kc-0-62364`, physical addresses are `[100.122.31.237:7800]`
2024-05-29 06:23:55,830 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2024-05-29 06:24:10,351 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: example-kc-0-62364, Site name: null
2024-05-29 06:24:10,470 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener

[pruivo]

@u2bo do you want to create a GitHub discussion and share your configuration and logs?

[u2bo]

@u2bo do you want to create a GitHub discussion and share your configuration and logs?

#29960