Performance Degradation from Keycloak Version 22.0.3 to Keycloak 23.0.X

[avillella59]

Before reporting an issue

• I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

core

Describe the bug

Hello !

We are writing to seek your assistance regarding performance issues encountered after upgrading our Java Spring Boot microservices architecture application from Keycloak version 22.0.3 to 23.0.X.

Before each production deployment, we are launching performance testing using Jmeter. Previously, during our performance testing, we were able to generate approximately 360,000 Keycloak tokens within one hour. However, since migrating to Keycloak 23.0.X, we have observed a significant decrease in performance, with only around 180,000 tokens generated within the same time frame.

Further analysis through performance profiling has revealed that the latency on the Keycloak side has notably increased since the upgrade to Keycloak 23.0.X.

Figure 1 : % of time spent during our performance testing

Figure 2 : Latency during another performance testing

To gain further insights into the performance degradation, we conducted additional testing without our architectural layer. Using this script (https://github.com/opfab/keycloak-perf/blob/main/test.sh) on our local machines with Docker, we installed Keycloak 22.0.3 and Keycloak 23.0.X separately. The results of these tests closely mirror our real-world observations : under Keycloak 22.0.3, we were able to generate 10,000 tokens in approximately 14 minutes, whereas under Keycloak 23.0.X, it took around 25 minutes to achieve the same result.

We ultimately decided to install Keycloak 24.0.1 on our local machines, hopeful that it would resolve our issue. Considering the significant changes in password hashing intervals introduced in Keycloak 24, we adjusted our password hashing policy like this :

After that, we launched the same test as before and observed that the performance remained consistent with the tests conducted on Keycloak 23.0.X.

These findings provide compelling evidence that the performance issues we are encountering are directly associated with an upgrade introduced in Keycloak 23.0.X. Unfortunately, there's no errors in our server log, just latency.

Given the critical role of Keycloak in our authentication and authorization processes within our microservices architecture, resolving these performance issues promptly is crucial to maintaining our system's reliability and scalability.

We kindly request your expertise and guidance in identifying the root cause of these performance discrepancies and implementing any necessary optimizations or configurations to restore our system's performance to previous levels.

Version

23.0.7

Regression

• The issue is a regression

Expected behavior

Approximately the same performance / number of access token generated with Keycloak 23.0.X than Keycloak 22.0.3.

Actual behavior

Since migrating to Keycloak 23.0.X (we tried Keycloak 23.0.0, then 23.0.1, then 23.0.2 and finally 23.0.7), there's a significant decrease in performance. (high latency and low number of access token generated)

How to Reproduce?

Generate a large number of access token (>100k) with the following script on a Keycloak 22.0.3 then on a Keycloak version >= 23.0.0
https://github.com/opfab/keycloak-perf

Anything else?

No response

[pedroigor]

I'm wondering how tokens look like in terms of claims (e.g.: mapping groups, roles, user attributes, etc) so that we can have an idea of what else is involved when issuing tokens, in addition to user/client credential verification.

Also, is the issue reproducible using a raw distribution and a realm with a single client and user?

[ahus1]

Thank you for providing the reproducer. I ran it locally, and with 10000 tokens, both the 22.x and the 23.x container took ~320 seconds to create those tokens on my machine, which is about 5 minutes.

So I didn't see a difference.

I slighly modified the reproducer to be more realistic, see opfab/keycloak-perf#1

Still I saw the same results with both start and start-dev.

Please review the reproducer, and compare it to your production environment.

Once more information is available, we can re-triage the issue.

[keycloak-github-bot]

Thanks for reporting this issue, but there is insufficient information or lack of steps to reproduce.

Please provide additional details, otherwise this issue will be automatically closed within 14 days.

[keycloak-github-bot]

Due to lack of updates in the last 14 days this issue will be automatically closed.