-
Notifications
You must be signed in to change notification settings - Fork 6.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Argon2 password hashing leads to increased Major GC's in Keycloak's JVM during load tests #29033
Comments
@kami619 As a side note: we have been using G1C in production since we started using Keycloak without problems. |
Steven created an issue in the bouncy castle upstream project to optimize the memory usage of Argon2. We don't know if and when this will be implemented. Therefore, we will need to continue changing the GC settings. |
Update the default GC from ParallelGC to G1GC Signed-off-by: Kamesh Akella <kamesh.asp@gmail.com>
Closes keycloak#29033 Signed-off-by: Kamesh Akella <kamesh.asp@gmail.com> Co-authored-by: Alexander Schwartz <aschwart@redhat.com>
Closes #29033 Signed-off-by: Kamesh Akella <kamesh.asp@gmail.com> Co-authored-by: Alexander Schwartz <aschwart@redhat.com>
Re-opened as we need release notes and an updated sizing guide (the sizing guide doesn't reflect Argon2 yet) |
Closes keycloak#29033 Signed-off-by: Alexander Schwartz <aschwart@redhat.com>
Closes #29033 Signed-off-by: Kamesh Akella <kamesh.asp@gmail.com> Signed-off-by: Alexander Schwartz <aschwart@redhat.com> Co-authored-by: Alexander Schwartz <alexander.schwartz@gmx.net> Co-authored-by: Václav Muzikář <vaclav@muzikari.cz> Co-authored-by: Alexander Schwartz <aschwart@redhat.com>
Before reporting an issue
Area
dist/quarkus
Describe the bug
Argon2 password hashing leads to increased Major GC's in Keycloak's JVM during load tests.
With Argon2 occupying larger amounts of heap memory and with the existing GC(ParallelGC) results in continuous Major GC's leading for higher CPU contention and lower performance across the board.
We looked into the tuning options and did several series of tests. And the findings are below.
But we understand there is an intervention needed due to this password hashing algorithm change and there are couple of recommendations suggested in this ticket based on our tests.
Version
nightly
Regression
Expected behavior
We would want the Keycloak's JVM heap to behave consistently in terms of both throughput and endpoint performance.
Actual behavior
With the Argon2 hashing change, we find the application to behave abnormally during medium to high load situations with increased JVM GC overhead and CPU utilization.
With the Keycloak current JVM defaults:
High Major GC counts during a load test run and higher JVM GC Overhead.
How to Reproduce?
Using the keycloak-benchmark and the default Keycloak JVM heap settings, run the below command
Anything else?
We then disabled the
UseAdaptiveSizePolicy
and that stabilized the load test runs. Further more we observed the Adaptive policy is aggressively looking to clear the heap resulting in frequent heap resizing events, but with disabling the adaptive policy, we were able to control that better. But disabling a key attribute is not a longer term solution, now we started to look into the G1GC which suits better to our use case and ran few experiments. Below is the result from those experiments.Based on this data, it seems like, G1GC even with current Keycloak JVM defaults works better than ParallelGC and it offers some more tuning bandwidth by adjusting GCTimeRatio and AdaptiveSizePolicyWeight. We would like to hear how the Cloud Native team interprets this data and take appropriate next steps.
The text was updated successfully, but these errors were encountered: