Cannot display 'Authentication Flows' screen when a realm contains more than ~4000 clients

[reda-alaoui]

Before reporting an issue

• I have searched existing issues
• I have reproduced the issue with the latest release

Area

storage

Describe the bug

One of our realms contains 8754 clients.

On this realm, we are unable to display the Authentication/Flows screen.

Version

21.1.1

Expected behavior

The Authentication/Flows screen should be usable even when the database contains many clients.

Actual behavior

When accessing Authentication/Flows, the http request on https://identity-test.cosium.com/auth/admin/realms/general/ui-ext/authentication-management/flows hangs for 5 minutes then fails with a 500 error.

On the server side, we see that the request has been killed by the TransactionReaper.

How to Reproduce?

Create a realm holding ~4000 clients, then try to load the authentication flows screen.

Anything else?

Here is the stacktrace that always comes up from thread dumps made while the request is hanging:

"executor-thread-3" - Thread t@154
   java.lang.Thread.State: RUNNABLE
        at org.hibernate.engine.spi.CollectionEntry.preFlush(CollectionEntry.java:172)
        at org.hibernate.event.internal.AbstractFlushingEventListener.lambda$prepareCollectionFlushes$0(AbstractFlushingEventListener.java:195)
        at org.hibernate.event.internal.AbstractFlushingEventListener$$Lambda$1619/0x0000000800e02440.accept(Unknown Source)
        at org.hibernate.engine.internal.StatefulPersistenceContext.forEachCollectionEntry(StatefulPersistenceContext.java:1140)
        at org.hibernate.event.internal.AbstractFlushingEventListener.prepareCollectionFlushes(AbstractFlushingEventListener.java:194)
        at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:86)
        at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:50)
        at org.hibernate.internal.SessionImpl$$Lambda$1560/0x0000000800d89840.accept(Unknown Source)
        at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:107)
        at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1372)
        at org.hibernate.internal.SessionImpl.scroll(SessionImpl.java:1597)
        at org.hibernate.query.internal.AbstractProducedQuery.doScroll(AbstractProducedQuery.java:1588)
        at org.hibernate.query.internal.AbstractProducedQuery.scroll(AbstractProducedQuery.java:1574)
        at org.hibernate.query.internal.AbstractProducedQuery.stream(AbstractProducedQuery.java:1598)
        at org.hibernate.query.Query.getResultStream(Query.java:1140)
        at org.keycloak.models.jpa.JpaRealmProvider.getClientScopes(JpaRealmProvider.java:986)
        at org.keycloak.storage.ClientStorageManager.getClientScopes(ClientStorageManager.java:212)
        at org.keycloak.models.cache.infinispan.RealmCacheSession.getClientScopes(RealmCacheSession.java:1355)
        at org.keycloak.models.jpa.ClientAdapter.getClientScopes(ClientAdapter.java:366)
        at org.keycloak.models.cache.infinispan.entities.CachedClient.<init>(CachedClient.java:112)
        at org.keycloak.models.cache.infinispan.RealmCacheSession.cacheClient(RealmCacheSession.java:1152)
        at org.keycloak.models.cache.infinispan.RealmCacheSession.getClientById(RealmCacheSession.java:1114)
        at org.keycloak.models.delegate.ClientModelLazyDelegate$WithId.lambda$new$0(ClientModelLazyDelegate.java:53)
        at org.keycloak.models.delegate.ClientModelLazyDelegate$WithId$$Lambda$2226/0x00000008011db840.get(Unknown Source)
        at org.keycloak.models.delegate.ClientModelLazyDelegate.getDelegate(ClientModelLazyDelegate.java:83)
        at org.keycloak.models.delegate.ClientModelLazyDelegate.getAuthenticationFlowBindingOverrides(ClientModelLazyDelegate.java:324)
        at org.keycloak.admin.ui.rest.model.AuthenticationMapper.lambda$convertToModel$1(AuthenticationMapper.java:32)
        at org.keycloak.admin.ui.rest.model.AuthenticationMapper$$Lambda$2224/0x00000008011db040.test(Unknown Source)
        at java.base@11.0.18/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
        at java.base@11.0.18/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
        at java.base@11.0.18/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
        at java.base@11.0.18/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
        at java.base@11.0.18/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
        at java.base@11.0.18/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
        at java.base@11.0.18/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base@11.0.18/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base@11.0.18/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base@11.0.18/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
        at org.hibernate.query.spi.StreamDecorator.collect(StreamDecorator.java:209)
        at org.keycloak.utils.ClosingStream.collect(ClosingStream.java:182)
        at org.keycloak.admin.ui.rest.model.AuthenticationMapper.convertToModel(AuthenticationMapper.java:35)
        at org.keycloak.admin.ui.rest.AuthenticationManagementResource.lambda$listIdentityProviders$1(AuthenticationManagementResource.java:66)
        at org.keycloak.admin.ui.rest.AuthenticationManagementResource$$Lambda$2217/0x00000008011d9440.apply(Unknown Source)
        at java.base@11.0.18/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
        at java.base@11.0.18/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
        at java.base@11.0.18/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
        at java.base@11.0.18/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base@11.0.18/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base@11.0.18/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base@11.0.18/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base@11.0.18/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
        at org.keycloak.admin.ui.rest.AuthenticationManagementResource.listIdentityProviders(AuthenticationManagementResource.java:67)
        at java.base@11.0.18/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base@11.0.18/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base@11.0.18/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base@11.0.18/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
        at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524)
        at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474

According to my investigation, the request process spends a lot of time in https://github.com/keycloak/keycloak/blob/21.1.1/rest/admin-ui-ext/src/main/java/org/keycloak/admin/ui/rest/model/AuthenticationMapper.java#L35 . The MAX_USED_BY==9 limit is useless, because none of our clients overrides the authentication flow.

Keycloak iterates on the full list of clients. Consequently, the full list of clients loads up in Hibernate 1st level cache leading to big performance degradation of the Hibernate autoflush process.

[reda-alaoui]

https://github.com/Cosium/keycloak-issue-21010 shows how to reproduce the issue

[jsorah]

Noting that it also appears if you were to have a high number of identity providers configured you would hit a similar issue, though that may be a more rare case for folks, it is still going to lead to a scaling issue.

[annasarte]

Hi @mposolda . I have verified this is fixed on Keycloak 23.0.4. We are actually using Red Hat Build for Keycloak 22.0.6. is this fix already back-ported to Keycloak 22.0.9 (per the added label yesterday?)