Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode #22904

Closed
ghost opened this issue Sep 2, 2023 · 14 comments · Fixed by #25492
Closed

Flaky test: org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode #22904

ghost opened this issue Sep 2, 2023 · 14 comments · Fixed by #25492
Assignees
@ahus1
Labels
area/ci Indicates an issue on the CI area/infinispan flaky-test kind/bug Categorizes a PR related to a bug release/24.0.0 team/continuous-testing

Comments

@ghost
Copy link

ghost commented Sep 2, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@4bd0b697 waiting for 132169 ms blocked for 1336 ms
	at java.base@17.0.8/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...
@ghost ghost added area/ci Indicates an issue on the CI flaky-test kind/bug Categorizes a PR related to a bug team/continuous-testing labels Sep 2, 2023
@ahus1 ahus1 assigned tondrusk and ahus1 and unassigned tondrusk Sep 5, 2023
@ghost
Copy link
Author

ghost commented Sep 6, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #23006

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@44c9a80f waiting for 120861 ms blocked for 37 ms
	at java.base@17.0.8/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

1 similar comment
@ghost
Copy link
Author

ghost commented Sep 6, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #23006

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@44c9a80f waiting for 120861 ms blocked for 37 ms
	at java.base@17.0.8/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@ghost
Copy link
Author

ghost commented Sep 15, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #23284

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@3e0f6fad waiting for 128401 ms blocked for 40 ms
	at java.base@17.0.8.1/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8.1/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@ghost
Copy link
Author

ghost commented Oct 4, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #23708

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@43a29c9a waiting for 149037 ms blocked for 76 ms
	at java.base@17.0.8.1/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8.1/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@ghost
Copy link
Author

ghost commented Oct 5, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #22700

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@477d73af waiting for 145706 ms blocked for 3274 ms
	at java.base@17.0.8.1/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8.1/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-25(site-id=site-1), see cause for remote stack trace
	at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:25)
	at org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:47)
	at org.infinispan.remoting.transport.impl.VoidResponseCollector.addException(VoidResponseCollector.java:19)
	at org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
...

@ghost
Copy link
Author

ghost commented Oct 10, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@7605efa4 waiting for 143286 ms blocked for 389 ms
	at java.base@17.0.8.1/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8.1/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@ahus1 ahus1 removed their assignment Oct 10, 2023
@vramik vramik mentioned this issue Oct 12, 2023
Merged
@ghost
Copy link
Author

ghost commented Nov 1, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #24086

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@660e2d36 waiting for 173879 ms blocked for 11071 ms
	at java.base@17.0.8.1/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.8.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.8.1/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@ghost
Copy link
Author

ghost commented Nov 8, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@17b7591b waiting for 134614 ms blocked for 37 ms
	at java.base@17.0.9/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.9/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@martin-kanis
Copy link
Contributor

@ahus1 We assigned this to you as it seems similar/same as #24085

@ghost
Copy link
Author

ghost commented Nov 20, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #24438

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@9b8c816 waiting for 129119 ms blocked for 59 ms
	at java.base@17.0.9/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.9/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@ghost
Copy link
Author

ghost commented Nov 20, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #24843

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@5df8de8b waiting for 131904 ms blocked for 56 ms
	at java.base@17.0.9/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.9/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@ghost
Copy link
Author

ghost commented Nov 27, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #24881

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@7df826fa waiting for 132888 ms blocked for 60 ms
	at java.base@17.0.9/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.9/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

1 similar comment
@ghost
Copy link
Author

ghost commented Nov 27, 2023

org.keycloak.testsuite.model.session.OfflineSessionPersistenceTest#testPersistenceMultipleNodesClientSessionAtSameNode

Keycloak CI - Store Model Tests / Pull Request #24881

java.lang.AssertionError: 
threads didn't terminate in time: [main (TIMED_WAITING): locked on java.util.concurrent.FutureTask@7df826fa waiting for 132888 ms blocked for 60 ms
	at java.base@17.0.9/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@17.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
	at java.base@17.0.9/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:444)
...

@lhanusov
Copy link
Contributor

lhanusov commented Dec 11, 2023 •
edited

#21665 this issue checked by @martin-kanis has a same failure as a recent wf failure: https://github.com/keycloak/keycloak/actions/runs/7132662872/job/19423989873

2023-12-07T19:21:10.4648785Z 19:21:10,356 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread-node-22-p129-t1) ISPN000136: Error executing command PutMapCommand on Cache 'offlineClientSessions', writing keys [5ddd5090-4de0-4d50-b6e0-f136d44fec27, 16e1e931-b5a2-4fcb-9ece-6f363a2efcaa, 294fec52-c6ef-45ff-9fd1-8aec1104cef3, ab22f432-e9b3-4e34-81a2-5d84c7c2c87b, 2ca93588-8f33-4521-ae5b-00e1518b8609, 4d0fdbbe-8d57-44fe-9a46-c941b1e32631, 7fba826a-f0fd-4609-9bd1-1d02a29b3722, 1ac3b7c8-810f-4f7a-9e62-e740dbbdc5ee...<30 other elements>]
2023-12-07T19:21:10.4655138Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-22(site-id=site-2), node-24(site-id=site-2)]). Id=97513
2023-12-07T19:21:10.4657699Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4659827Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4662062Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4663727Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4665740Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4667873Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4669666Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4671020Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:10.4676672Z 19:21:10,360 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread-node-22-p129-t1) ISPN000136: Error executing command PutMapCommand on Cache 'offlineClientSessions', writing keys [549d5316-7b8e-4039-804c-3ad9831b1072, 05bb0e37-fd7b-44d3-8440-58521e4435f1, 80901b24-7945-489c-84a6-80c1d8946c1d, 8b2cedd0-e632-4057-ba40-b4a9ef1b34a8, cb3c1335-7b73-4c40-a71d-ef71af41b312, c7a29ad5-482e-4de7-a532-3849dde284a1, 9a38f1e4-2250-4398-9b7e-86e257c68186, 3fbb6ebe-6263-4bb0-81b5-c7ada9ba58bd...<26 other elements>]
2023-12-07T19:21:10.4683395Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97514
2023-12-07T19:21:10.4685648Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4687520Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4689283Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4690738Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4692375Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4694410Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4695979Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4697186Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:10.4702761Z 19:21:10,360 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread-node-24-p134-t1) ISPN000136: Error executing command PutMapCommand on Cache 'offlineClientSessions', writing keys [b75785ef-2c81-46cf-81bf-99c4701b6f57, f93c8139-b445-490e-89c0-21eb18efd58f, 00c3e22d-a211-4044-a8aa-cfba74b4e919, 651eb91a-a8f2-43ca-a314-4b090163d7ca, 475087dd-859a-4c6e-92d7-5d4120121ff1, f3d34c13-6447-425e-844e-1ac15604e41e, d7c92946-d132-4b2e-98bf-a5832362d740, 0d8b37a4-d7ff-484d-8266-ebf8d7cce001...<29 other elements>]
2023-12-07T19:21:10.4708233Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97515
2023-12-07T19:21:10.4710299Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4712111Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4713832Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4715244Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4716851Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4718667Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4720196Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4721517Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:10.4723918Z 19:21:10,360 WARN [org.infinispan.CLUSTER] (jgroups-7,node-24(site-id=site-2)) ISPN000071: Caught exception when handling command org.infinispan.manager.impl.ReplicableManagerFunctionCommand@6a91c2c0
2023-12-07T19:21:10.4726878Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97515
2023-12-07T19:21:10.4728902Z at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:258)
2023-12-07T19:21:10.4730519Z at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:323)
2023-12-07T19:21:10.4731884Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:111)
2023-12-07T19:21:10.4733208Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:93)
2023-12-07T19:21:10.4734428Z at org.infinispan.cache.impl.CacheImpl.putAll(CacheImpl.java:1361)
2023-12-07T19:21:10.4735618Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:562)
2023-12-07T19:21:10.4736861Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:567)
2023-12-07T19:21:10.4738082Z at org.infinispan.cache.impl.EncoderCache.putAll(EncoderCache.java:792)
2023-12-07T19:21:10.4739898Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:144)
2023-12-07T19:21:10.4742315Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:47)
2023-12-07T19:21:10.4744577Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker$1.run(SessionInitializerWorker.java:71)
2023-12-07T19:21:10.4746531Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:10.4748379Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:10.4750321Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:10.4752248Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:67)
2023-12-07T19:21:10.4754456Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:34)
2023-12-07T19:21:10.4756722Z at org.infinispan.manager.impl.ReplicableManagerFunctionCommand.invokeAsync(ReplicableManagerFunctionCommand.java:45)
2023-12-07T19:21:10.4759051Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:150)
2023-12-07T19:21:10.4761655Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:124)
2023-12-07T19:21:10.4764357Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:78)
2023-12-07T19:21:10.4766534Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1538)
2023-12-07T19:21:10.4768421Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1465)
2023-12-07T19:21:10.4770370Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$1(JGroupsTransport.java:1656)
2023-12-07T19:21:10.4771835Z at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2023-12-07T19:21:10.4773208Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1648)
2023-12-07T19:21:10.4774497Z at org.jgroups.JChannel.up(JChannel.java:749)
2023-12-07T19:21:10.4775372Z at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:939)
2023-12-07T19:21:10.4776280Z at org.jgroups.protocols.FRAG2.up(FRAG2.java:161)
2023-12-07T19:21:10.4777312Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
2023-12-07T19:21:10.4778308Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
2023-12-07T19:21:10.4779237Z at org.jgroups.protocols.pbcast.GMS.up(GMS.java:859)
2023-12-07T19:21:10.4780156Z at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
2023-12-07T19:21:10.4781308Z at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1108)
2023-12-07T19:21:10.4782474Z at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:882)
2023-12-07T19:21:10.4783541Z at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:549)
2023-12-07T19:21:10.4784481Z at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:722)
2023-12-07T19:21:10.4785514Z at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:146)
2023-12-07T19:21:10.4786643Z at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)
2023-12-07T19:21:10.4787648Z at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:202)
2023-12-07T19:21:10.4788499Z at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
2023-12-07T19:21:10.4789374Z at org.jgroups.protocols.Discovery.up(Discovery.java:314)
2023-12-07T19:21:10.4790211Z at org.jgroups.protocols.RED.up(RED.java:119)
2023-12-07T19:21:10.4791024Z at org.jgroups.protocols.TP.passBatchUp(TP.java:1210)
2023-12-07T19:21:10.4792220Z at org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:161)
2023-12-07T19:21:10.4793658Z at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:157)
2023-12-07T19:21:10.4795105Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4796630Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4797761Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:10.4799542Z Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97515
2023-12-07T19:21:10.4801799Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4803771Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4805500Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4806910Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4808633Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4809961Z ... 3 more
2023-12-07T19:21:10.4810608Z Suppressed: org.infinispan.commons.util.logging.TraceException
2023-12-07T19:21:10.4812055Z at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:39)
2023-12-07T19:21:10.4813900Z at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:249)
2023-12-07T19:21:10.4815530Z at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:323)
2023-12-07T19:21:10.4816896Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:111)
2023-12-07T19:21:10.4818235Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:93)
2023-12-07T19:21:10.4819467Z at org.infinispan.cache.impl.CacheImpl.putAll(CacheImpl.java:1361)
2023-12-07T19:21:10.4820658Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:562)
2023-12-07T19:21:10.4821923Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:567)
2023-12-07T19:21:10.4823151Z at org.infinispan.cache.impl.EncoderCache.putAll(EncoderCache.java:792)
2023-12-07T19:21:10.4824955Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:144)
2023-12-07T19:21:10.4827371Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:47)
2023-12-07T19:21:10.4829763Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker$1.run(SessionInitializerWorker.java:71)
2023-12-07T19:21:10.4831819Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:10.4833670Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:10.4835490Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:10.4837425Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:67)
2023-12-07T19:21:10.4839614Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:34)
2023-12-07T19:21:10.4841777Z at org.infinispan.manager.impl.ReplicableManagerFunctionCommand.invokeAsync(ReplicableManagerFunctionCommand.java:45)
2023-12-07T19:21:10.4844114Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:150)
2023-12-07T19:21:10.4845575Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:124)
2023-12-07T19:21:10.4846994Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:78)
2023-12-07T19:21:10.4848229Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1538)
2023-12-07T19:21:10.4849304Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1465)
2023-12-07T19:21:10.4850419Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$1(JGroupsTransport.java:1656)
2023-12-07T19:21:10.4851413Z at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2023-12-07T19:21:10.4852240Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1648)
2023-12-07T19:21:10.4853008Z at org.jgroups.JChannel.up(JChannel.java:749)
2023-12-07T19:21:10.4853551Z at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:939)
2023-12-07T19:21:10.4854099Z at org.jgroups.protocols.FRAG2.up(FRAG2.java:161)
2023-12-07T19:21:10.4854652Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
2023-12-07T19:21:10.4855323Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
2023-12-07T19:21:10.4855897Z at org.jgroups.protocols.pbcast.GMS.up(GMS.java:859)
2023-12-07T19:21:10.4856462Z at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
2023-12-07T19:21:10.4857089Z at org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1108)
2023-12-07T19:21:10.4857781Z at org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:882)
2023-12-07T19:21:10.4858420Z at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:549)
2023-12-07T19:21:10.4858987Z at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:722)
2023-12-07T19:21:10.4859653Z at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:146)
2023-12-07T19:21:10.4860331Z at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)
2023-12-07T19:21:10.4860949Z at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:202)
2023-12-07T19:21:10.4861465Z at org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
2023-12-07T19:21:10.4862001Z at org.jgroups.protocols.Discovery.up(Discovery.java:314)
2023-12-07T19:21:10.4862520Z at org.jgroups.protocols.RED.up(RED.java:119)
2023-12-07T19:21:10.4863015Z at org.jgroups.protocols.TP.passBatchUp(TP.java:1210)
2023-12-07T19:21:10.4863724Z at org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:161)
2023-12-07T19:21:10.4864565Z at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:157)
2023-12-07T19:21:10.4865231Z ... 3 more
2023-12-07T19:21:10.4868172Z 19:21:10,363 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread-node-22-p129-t1) ISPN000136: Error executing command PutMapCommand on Cache 'offlineClientSessions', writing keys [fe130e35-ea6d-41b5-96c3-8c7a382e2c68, 7b346ef3-0222-48e8-82ce-b7b5c58eb07a, 1c6c3397-f75d-4fd7-8b6c-108f28d9e7c8, e82350c8-b64c-4f01-a72e-4ca2c0067ce9, 86abf913-28df-4db5-8312-62fd4d6542dd, 444810ab-216e-444e-938e-42d4dd4e2411, 3fd33ad9-e7d1-4551-9766-94bfad17c50c, 8ad94f0c-c005-43d4-8179-39b5ce40b041...<23 other elements>]
2023-12-07T19:21:10.4871229Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97516
2023-12-07T19:21:10.4872406Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4873455Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4874442Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4875260Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4876209Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4877256Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4878157Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4878828Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:10.4881772Z 19:21:10,366 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread-node-24-p134-t1) ISPN000136: Error executing command PutMapCommand on Cache 'offlineClientSessions', writing keys [61756ae7-f652-4bcb-b557-6c14dbf9ffb1, a764205f-2202-49f3-bc4d-3b4525419502, f4f8d6a7-313f-4b84-99e0-91870eb35db1, 1c701070-0a88-4bee-87fd-d94b341fa88c, d5cc99ff-2ebc-484d-b80d-1fe625c5fd69, f82c2707-2d4b-4e61-9b4c-07d22985729f, ec99a89b-d185-47b7-b9e8-285a2a7fd1f5, c51b06ba-e66f-422c-b6ea-566db754d838...<39 other elements>]
2023-12-07T19:21:10.4885052Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97518
2023-12-07T19:21:10.4886319Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4887362Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4888343Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4889160Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4890107Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4891153Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4892043Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4892730Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:10.4893996Z 19:21:10,367 WARN [org.infinispan.CLUSTER] (jgroups-12,node-24(site-id=site-2)) ISPN000071: Caught exception when handling command org.infinispan.manager.impl.ReplicableManagerFunctionCommand@7dc28b8c
2023-12-07T19:21:10.4895592Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97518
2023-12-07T19:21:10.4896737Z at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:258)
2023-12-07T19:21:10.4897672Z at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:323)
2023-12-07T19:21:10.4898564Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:111)
2023-12-07T19:21:10.4899345Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:93)
2023-12-07T19:21:10.4900131Z at org.infinispan.cache.impl.CacheImpl.putAll(CacheImpl.java:1361)
2023-12-07T19:21:10.4900841Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:562)
2023-12-07T19:21:10.4901587Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:567)
2023-12-07T19:21:10.4902317Z at org.infinispan.cache.impl.EncoderCache.putAll(EncoderCache.java:792)
2023-12-07T19:21:10.4903355Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:144)
2023-12-07T19:21:10.4904708Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:47)
2023-12-07T19:21:10.4905992Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker$1.run(SessionInitializerWorker.java:71)
2023-12-07T19:21:10.4907108Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:10.4908174Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:10.4909211Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:10.4910326Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:67)
2023-12-07T19:21:10.4911566Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:34)
2023-12-07T19:21:10.4912784Z at org.infinispan.manager.impl.ReplicableManagerFunctionCommand.invokeAsync(ReplicableManagerFunctionCommand.java:45)
2023-12-07T19:21:10.4914170Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:150)
2023-12-07T19:21:10.4915591Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:124)
2023-12-07T19:21:10.4917007Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:78)
2023-12-07T19:21:10.4918296Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1538)
2023-12-07T19:21:10.4919371Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1465)
2023-12-07T19:21:10.4920441Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1642)
2023-12-07T19:21:10.4921207Z at org.jgroups.JChannel.up(JChannel.java:733)
2023-12-07T19:21:10.4921738Z at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:936)
2023-12-07T19:21:10.4922297Z at org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
2023-12-07T19:21:10.4922838Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:253)
2023-12-07T19:21:10.4923570Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:261)
2023-12-07T19:21:10.4924133Z at org.jgroups.protocols.pbcast.GMS.up(GMS.java:845)
2023-12-07T19:21:10.4924690Z at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:226)
2023-12-07T19:21:10.4925326Z at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1083)
2023-12-07T19:21:10.4925983Z at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:822)
2023-12-07T19:21:10.4926653Z at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:804)
2023-12-07T19:21:10.4927285Z at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:453)
2023-12-07T19:21:10.4927853Z at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:680)
2023-12-07T19:21:10.4928481Z at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
2023-12-07T19:21:10.4929237Z at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
2023-12-07T19:21:10.4929853Z at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:188)
2023-12-07T19:21:10.4930371Z at org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
2023-12-07T19:21:10.4930907Z at org.jgroups.protocols.Discovery.up(Discovery.java:294)
2023-12-07T19:21:10.4931736Z at org.jgroups.stack.Protocol.up(Protocol.java:340)
2023-12-07T19:21:10.4932280Z at org.jgroups.protocols.TP.passMessageUp(TP.java:1184)
2023-12-07T19:21:10.4932999Z at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:107)
2023-12-07T19:21:10.4933896Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4934782Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4935470Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:10.4936497Z Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97518
2023-12-07T19:21:10.4937713Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4938756Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4939733Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4940565Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4941504Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4942268Z ... 3 more
2023-12-07T19:21:10.4942683Z Suppressed: org.infinispan.commons.util.logging.TraceException
2023-12-07T19:21:10.4943533Z at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:39)
2023-12-07T19:21:10.4944683Z at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:249)
2023-12-07T19:21:10.4945621Z at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:323)
2023-12-07T19:21:10.4946423Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:111)
2023-12-07T19:21:10.4947198Z at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:93)
2023-12-07T19:21:10.4947991Z at org.infinispan.cache.impl.CacheImpl.putAll(CacheImpl.java:1361)
2023-12-07T19:21:10.4948702Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:562)
2023-12-07T19:21:10.4949445Z at org.infinispan.cache.impl.DecoratedCache.putAll(DecoratedCache.java:567)
2023-12-07T19:21:10.4950178Z at org.infinispan.cache.impl.EncoderCache.putAll(EncoderCache.java:792)
2023-12-07T19:21:10.4951219Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:144)
2023-12-07T19:21:10.4952586Z at org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionsLoader.loadSessions(RemoteCacheSessionsLoader.java:47)
2023-12-07T19:21:10.4953868Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker$1.run(SessionInitializerWorker.java:71)
2023-12-07T19:21:10.4954985Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:10.4956052Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:10.4957099Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:10.4958207Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:67)
2023-12-07T19:21:10.4959479Z at org.keycloak.models.sessions.infinispan.initializer.SessionInitializerWorker.apply(SessionInitializerWorker.java:34)
2023-12-07T19:21:10.4960793Z at org.infinispan.manager.impl.ReplicableManagerFunctionCommand.invokeAsync(ReplicableManagerFunctionCommand.java:45)
2023-12-07T19:21:10.4962158Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:150)
2023-12-07T19:21:10.4963739Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:124)
2023-12-07T19:21:10.4965158Z at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:78)
2023-12-07T19:21:10.4966386Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1538)
2023-12-07T19:21:10.4967457Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1465)
2023-12-07T19:21:10.4968535Z at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1642)
2023-12-07T19:21:10.4969310Z at org.jgroups.JChannel.up(JChannel.java:733)
2023-12-07T19:21:10.4969842Z at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:936)
2023-12-07T19:21:10.4970390Z at org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
2023-12-07T19:21:10.4970941Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:253)
2023-12-07T19:21:10.4971542Z at org.jgroups.protocols.FlowControl.up(FlowControl.java:261)
2023-12-07T19:21:10.4972102Z at org.jgroups.protocols.pbcast.GMS.up(GMS.java:845)
2023-12-07T19:21:10.4972665Z at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:226)
2023-12-07T19:21:10.4973301Z at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1083)
2023-12-07T19:21:10.4973957Z at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:822)
2023-12-07T19:21:10.4974635Z at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:804)
2023-12-07T19:21:10.4975384Z at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:453)
2023-12-07T19:21:10.4975963Z at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:680)
2023-12-07T19:21:10.4976585Z at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
2023-12-07T19:21:10.4977257Z at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
2023-12-07T19:21:10.4977884Z at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:188)
2023-12-07T19:21:10.4978399Z at org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
2023-12-07T19:21:10.4979006Z at org.jgroups.protocols.Discovery.up(Discovery.java:294)
2023-12-07T19:21:10.4979542Z at org.jgroups.stack.Protocol.up(Protocol.java:340)
2023-12-07T19:21:10.4980074Z at org.jgroups.protocols.TP.passMessageUp(TP.java:1184)
2023-12-07T19:21:10.4980795Z at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:107)
2023-12-07T19:21:10.4981409Z ... 3 more
2023-12-07T19:21:10.4984202Z 19:21:10,371 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread-node-22-p129-t1) ISPN000136: Error executing command PutMapCommand on Cache 'offlineClientSessions', writing keys [26453732-6f60-41f1-a98f-339e2b291656, 541388da-20a4-4469-ae30-27de31e74886, 0cb26e62-6d60-4118-918a-4e20d148cf47, 4c6ca7f3-3918-4fd1-97d2-b51cc5de5308, 9d8e1e35-52e1-46ab-9226-27d4c873f0e2, 3616cc4c-a86f-4298-a8e7-99bbd1e8290b, cdb75046-b35a-4961-af15-8f4075568f9d, 65de5b91-4882-4171-a62c-af85510dd78e...<29 other elements>]
2023-12-07T19:21:10.4987202Z org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks ([node-24(site-id=site-2)]). Id=97519
2023-12-07T19:21:10.4988385Z at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:228)
2023-12-07T19:21:10.4989437Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:281)
2023-12-07T19:21:10.4990415Z at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:261)
2023-12-07T19:21:10.4991516Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:10.4992600Z at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-12-07T19:21:10.4993761Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:10.4994664Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:10.4995351Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:19.5864353Z 19:21:19,493 ERROR [org.keycloak.models.sessions.infinispan.initializer.CacheInitializer] (pool-41-thread-3) Interrupted
2023-12-07T19:21:19.5865946Z java.lang.InterruptedException: sleep interrupted
2023-12-07T19:21:19.5866790Z at java.base/java.lang.Thread.sleep(Native Method)
2023-12-07T19:21:19.5867625Z at java.base/java.lang.Thread.sleep(Thread.java:344)
2023-12-07T19:21:19.5868628Z at java.base/java.util.concurrent.TimeUnit.sleep(TimeUnit.java:446)
2023-12-07T19:21:19.5870225Z at org.keycloak.models.sessions.infinispan.initializer.CacheInitializer.loadSessions(CacheInitializer.java:41)
2023-12-07T19:21:19.5872489Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$6.run(InfinispanUserSessionProviderFactory.java:340)
2023-12-07T19:21:19.5874529Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:19.5876371Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:19.5878144Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:19.5880392Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCache(InfinispanUserSessionProviderFactory.java:326)
2023-12-07T19:21:19.5883442Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCaches(InfinispanUserSessionProviderFactory.java:318)
2023-12-07T19:21:19.5886334Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.access$800(InfinispanUserSessionProviderFactory.java:69)
2023-12-07T19:21:19.5888891Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$1.lambda$onEvent$0(InfinispanUserSessionProviderFactory.java:126)
2023-12-07T19:21:19.5891246Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:19.5893065Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:19.5894799Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:19.5896589Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithTimeout(KeycloakModelUtils.java:401)
2023-12-07T19:21:19.5898866Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$1.onEvent(InfinispanUserSessionProviderFactory.java:120)
2023-12-07T19:21:19.5901024Z at org.keycloak.services.DefaultKeycloakSessionFactory.publish(DefaultKeycloakSessionFactory.java:91)
2023-12-07T19:21:19.5902928Z at org.keycloak.testsuite.model.KeycloakModelTest.createKeycloakSessionFactory(KeycloakModelTest.java:341)
2023-12-07T19:21:19.5904791Z at org.keycloak.testsuite.model.KeycloakModelTest.inIndependentFactory(KeycloakModelTest.java:488)
2023-12-07T19:21:19.5906623Z at org.keycloak.testsuite.model.KeycloakModelTest.lambda$inIndependentFactories$6(KeycloakModelTest.java:393)
2023-12-07T19:21:19.5908152Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:19.5909539Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:19.5911122Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:19.5912556Z at java.base/java.lang.Thread.run(Thread.java:840)
2023-12-07T19:21:19.5914727Z 19:21:19,493 ERROR [org.keycloak.models.sessions.infinispan.initializer.InfinispanCacheInitializer] (pool-41-thread-1) Interruped exception when computed future. Errors: 1
2023-12-07T19:21:19.5916641Z java.lang.InterruptedException
2023-12-07T19:21:19.5917683Z at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386)
2023-12-07T19:21:19.5919147Z at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2023-12-07T19:21:19.5921134Z at org.keycloak.models.sessions.infinispan.initializer.InfinispanCacheInitializer.startLoadingImpl(InfinispanCacheInitializer.java:183)
2023-12-07T19:21:19.5923734Z at org.keycloak.models.sessions.infinispan.initializer.InfinispanCacheInitializer.startLoading(InfinispanCacheInitializer.java:113)
2023-12-07T19:21:19.5925945Z at org.keycloak.models.sessions.infinispan.initializer.CacheInitializer.loadSessions(CacheInitializer.java:66)
2023-12-07T19:21:19.5928205Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$6.run(InfinispanUserSessionProviderFactory.java:340)
2023-12-07T19:21:19.5930292Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:19.5932141Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:19.5933937Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:19.5936405Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCache(InfinispanUserSessionProviderFactory.java:326)
2023-12-07T19:21:19.5939619Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.loadSessionsFromRemoteCaches(InfinispanUserSessionProviderFactory.java:318)
2023-12-07T19:21:19.5942445Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory.access$800(InfinispanUserSessionProviderFactory.java:69)
2023-12-07T19:21:19.5945203Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$1.lambda$onEvent$0(InfinispanUserSessionProviderFactory.java:126)
2023-12-07T19:21:19.5948423Z at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:260)
2023-12-07T19:21:19.5950415Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:382)
2023-12-07T19:21:19.5952178Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:259)
2023-12-07T19:21:19.5953945Z at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithTimeout(KeycloakModelUtils.java:401)
2023-12-07T19:21:19.5956040Z at org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory$1.onEvent(InfinispanUserSessionProviderFactory.java:120)
2023-12-07T19:21:19.5958068Z at org.keycloak.services.DefaultKeycloakSessionFactory.publish(DefaultKeycloakSessionFactory.java:91)
2023-12-07T19:21:19.5959911Z at org.keycloak.testsuite.model.KeycloakModelTest.createKeycloakSessionFactory(KeycloakModelTest.java:341)
2023-12-07T19:21:19.5961765Z at org.keycloak.testsuite.model.KeycloakModelTest.inIndependentFactory(KeycloakModelTest.java:488)
2023-12-07T19:21:19.5963756Z at org.keycloak.testsuite.model.KeycloakModelTest.lambda$inIndependentFactories$6(KeycloakModelTest.java:393)
2023-12-07T19:21:19.5965300Z at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-12-07T19:21:19.5966664Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-12-07T19:21:19.5968198Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-12-07T19:21:19.5969357Z at java.base/java.lang.Thread.run(Thread.java:840)

@ahus1 ahus1 mentioned this issue Dec 12, 2023
Merged
@ghost ghost removed the status/triage label Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ahus1 ahus1

Labels
area/ci Indicates an issue on the CI area/infinispan flaky-test kind/bug Categorizes a PR related to a bug release/24.0.0 team/continuous-testing
Projects
Storage
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Avoid shutdown of Infinispan when using cache ahus1/keycloak
4 participants
@martin-kanis @ahus1 @lhanusov @tondrusk