Persistent user sessions (preview)

[mhajas]

One of the biggest pain points of Keycloak is currently losing sessions on restarts. There were countless complaints on this, including a lot of workarounds that are usually unsupported and come with some trade-offs.

The current approach for storing sessions has become a blocker for active/active multi-site support that we are currently pursuing and therefore we needed to rethink this approach. While brainstorming a new approach we came up with a conclusion that we should also include durable sessions as that feature was already on our radar.

This is now part of Keycloak 25, see https://www.keycloak.org/docs/latest/upgrading/index.html#persistent-user-sessions for more details on how to migrate. Please provide your feedback here (where it works, and were it should be enhanced).

Idea

The current behavior of Keycloak is that online sessions are stored only in Infinispan while offline sessions are also stored in the database. Therefore, the new feature's main idea is to store online sessions in the same table structure.

Plans

To follow this work proceed to the following GitHub epic: #28265

Highlights of the Keycloak 25 release

• Store user online sessions and client online sessions in the database across restarts and upgrades (Persist online sessions to the database #27976)
• Don't use Keycloak's embedded distributed caches to store user session information (Persist online sessions to the database #27976)
• Apply the same code to offline user and offline client sessions, allowing to not use any JVM memory (Use the same new code for persistent sessions for offline sessions #28318)
• Limit the JVM memory used to cache online and offline sessions to a number of entries (by default 10_000) (Use the same new code for persistent sessions for offline sessions #28318)
• Batch updates needed for refreshing tokens (Batch updates to the database to avoid using too many IOPS #28501)
• Batch database updates needed for logins/logouts (Improve persistent sessions DB throughput for logins/logouts by batching #28862)

How to test

• Download Keycloak 25 or later

• Run Keycloak with the persistent-user-sessions feature. Proceed to the following guide on how to enable/disable Keycloak features.

  Example command in development mode:

  bin/kc.[sh|bat] start-dev --features="persistent-user-sessions"

• To confirm that online sessions are stored in the database check the database table offline_user_session contains rows with offline_flag column equal to 0.

[ahus1]

The first set of features are now available in the nightly build. I've updated the description.

[mhajas]

@afgarcia86 No. Tables that should contain the data are offline_client_session and offline_user_session. We are using the same table structure as offline sessions. You can check whether you have some sessions with offline_flag column equal to 0, if there are some it means the persistent online sessions work.

However, thank you for pointing out the table user_session I am not sure why it is there. It seems it is some relict from Keycloak 1.x 🤣. We should have a look at it.

[afgarcia86]

@mhajas thank you for the clarification! I see the data now :)

You may also want to look at client_session as its another empty table that I spent some time on haha.

Do you all have an eta for KC25?

[ahus1]

Keycloak 25 has been released, see https://www.keycloak.org/docs/latest/upgrading/index.html#persistent-user-sessions for more details on how to migrate. Please provide your feedback here (where it works, and were it should be enhanced).

[daviddelannoy]

Hi @ahus1,

Sadly we won't be able to test it in the coming weeks in a production-like environment due to other activities, but we will provide a feedback on how we handled the migration, even if it is a bit late.

By reading the migration guide, I think that it is implicit but it can be forgotten easily : between step 4 and step 5, after the XML ISPN configuration has been updated to remove the JDBC/external ISPN persistence, a new build is required before restarting, isn't it ? :
$KEYCLOAK_HOME/bin/kc.sh build --cache-config-file=ispn.xml ...

Thanks again for all enhancements on this subject

[ahus1]

It is new in KC25 that the cache options are runtime options and don't require a rebuild 🎉

It is now possible to specify the cache, cache-stack, and cache-config-file options during runtime. This eliminates the need to execute the build phase and rebuild your image due to them.

[ahus1]

Is Keycloak with persistent-user-sessions-no-cache fast enough for you?

It will use the database cache, so we hope it would run reasonable fast. If this is case, it would simplify the code of Keycloak a lot, and the data would always be consistently stored in Keycloak's database, and you need to worry less about Keycloak's JVM memory usage.

Please comment here with your findings - possibly with your logins per second and number of concurrent user sessions.

Thanks!

[Tristan971]

We'd be happy to try this and give some feedback on this as we will care quite a bit if it's an issue in the future. Our stats are on average:

• 1-2 login/s
• 150 oauth token refresh/s
• 550k total realm sessions, with ~25k active within a 15min sliding window

It will use the database cache [...] and you need to worry less about Keycloak's JVM memory usage

Do you mean the Hibernate entity cache, or hoping our DB has more RAM than ~ its data size? I imagine the latter, but for what it's worth Keycloak has never seemed that demanding to us on the RAM end of things.

Sure it's has a relatively high base cost for small use-cases, but it seems to scale very well as the number of sessions increases. I assume we are into the fairly large category, and it seems to not even really want to use a mere 3GB of heap size (as per this comment), so if I had to choose between saving memory on the JVM or saving iops on Postgres, I'll choose the latter every day. Especially as IO capacity is usually a much scarcer/pricey resource than memory.

Either way, your message notes:

You can't upgrade from a nightly build to a release build of Keycloak [...] always start with an empty database or a backup from a regular release.

And here I'm not sure how we could reasonably do that? Do you know how others try these?

I mean we can run synthetic tests, potentially, but I assume you can do that already and are looking for real-world reports instead? And I'm not sure how to do that would massive impact on our users if it means more than 1 mass-dc event.

[ahus1]

Do you mean the Hibernate entity cache, or hoping our DB has more RAM than ~ its data size? I imagine the latter, but for what it's worth Keycloak has never seemed that demanding to us on the RAM end of things.

I was hoping that the RAM of the database would be able to cache the pages for the active user sessions and the relevant indexes. We don't plan for Hibernate caching. The next iteration will have a bounded cache for sessions in Infinispan though, see #28318.

There are some people with 50m+ sessions, and they struggle with the JVM memory this uses, so using the DB is an alternative.

I agree IO is pricey, but the trade-off is different depending on the scenario. The current implementation will write to the database on each refresh of a token. A future version might batch those updates (which comes with tradeoffs) - I'll add this to the epic as another task.

As you are asking on how to reasonably test this: Synthetic tests in test environments are the thing we're currently looking for, with the load tests that people created to match their production load. There is no one-size-fits-all load test, and we would need to know when we meet people's trade-offs between in-memory-only-and-gone-after-restart and store-in-database-which-is-more-expensive-but-ok-i-need-it. This will depend on business needs, available infrastructure and infrastructure costs.

The discussion we're having now is also valuable, as it helps to steer our efforts into the right direction.

While feedback during discussions and on nightly releases is usually fast, we have new major Keycloak releases about every three months which is comparatively slow.

Once there is a release of Keycloak 25, this is then something you can properly upgrade to, and switch on-and-off in a production environment and we'll receive feedback. Any fixes to the code will need to wait for another major release except when they are small and they could then end up in a patch release.

[daviddelannoy]

Some production statistics :

• 0.5 LOGIN/s (I mean people entering their password or using social provider)
• 25 LOGIN/s including all LOGIN events triggered by silent check-sso or just normal SSO
• 15 REFRESH_TOKEN/s

count from ISPN database persistence (right now with the hacky -Dkeycloak.infinispan.ignoreSkipCacheStore=true) :

• 2 millions user sessions (long life session max, one month idle)
• 2 millions user client sessions

8 vCPU / 32 Go RAM, huge JVM around 20 Go (way too big, it has to be reviewed following recent guide and benchmark results) with most of users and user sessions in cache.
Database : 2 vCPU / 8 Go RAM

CPU usage is low on both instances (KC & database), they are oversized for current trafic.

NOTE: delay to get a refresh token has increased a lot between KC 20 and KC 22, from 40 ms to 220 ms with the same infrastructure, a little bit more sessions but not too much. We did not take time to investigate more on this.

[ahus1]

Thank you for sharing this numbers, The nightly build should support this, still we need to do some proper testing. Once #28501 is in, refreshes will use a lot less DB IO.

I've heard about some increased delay in previous versions, but it got better in KC24. It that remains, please reach out via an issue (but be ready to do some profiling in your environment).

[daviddelannoy]

Sure, I can't wait to upgrade to v24 to check it.

[Sjoerd82]

I've been tracking this discussion and its tendrils across github, trying to figure out what problem "persistent user sessions" solves to begin with.... Or to be honest, if it adressess a specific use case of mine:

Will this feature allow users to remain offline for a "long" time (say 30 days), and be able to silently log back in again? (the goal being to make it as easy as possible for the user to continue using the application). From what I understand, this can be mimicked by misuing an offline token, but I also have been told that this is not what offline tokens are meant for (which is that they are meant for executing authorized tasks when the user is not online) and may introduce unneccessary security risks.

Thinking about it, it seems that there is always additional risk involved by allowing this (since a token or some sort of password must always be stored on the, potentially vulnerable, client, but it seems to be acceptable for big apps like Gmail, and even secure apps like Protonmail.

Thanks for any clarification!

[Sjoerd82]

Great! The additional IO makes sense, but IMHO is a small price for the convenience it brings the user. Just so we are clear, this does mean that the client device holds on to a token of sorts? Will be keeping my eyes out for the next major release! :-D

[mhajas]

One addition to what @ahus1 said. In general, what you mentioned @Sjoerd82 can be solved also without this feature by increasing the session timeout to more than 30 days.

However, the problem this is solving is, that today when you set timeout to more than 30 days and there is a Keycloak restart (all nodes fail or there is a version upgrade) in the meantime, all online sessions are lost because they live only in memory.

When this feature is enabled, the sessions are stored also in the database, therefore the name Persistent sessions, and for this reason, the sessions are not lost on restarts. Naturally, if you set session max lifespan to 5 days it won't be available after 30 days.

The exciting part about this is this change is opening a few interesting doors for us because the new approach does not assume all sessions are present in memory and therefore we can have lower memory consumption and better scalability.

[ahus1]

Just so we are clear, this does mean that the client device holds on to a token of sorts?

The common case for these kind of long running user sessions is some web (browser) based application where the user closes the browser, and then opens it up the next day or week and wants to be logged in. For this to work, only the cookie Keycloak sets needs to be available. This will connect to user with their browser to an existing user session. The the tokens for the client(s) are then re-created from that.

Depending on the client/application and possibly a specific interaction in the client, the client can ask the user to still re-authenticate if they haven't done so for the last X seconds.

[daviddelannoy]

The exciting part about this is this change is opening a few interesting doors for us because the new approach does not assume all sessions are present in memory and therefore we can have lower memory consumption and better scalability.

I totally agree @mhajas that the 2 improvements "keep sessions after shutdown & restart" and "lower memory consumption" (even with some trade-offs on DB IOs) will provide both a greater user experience and open interesting technical doors.

Currently, with infinispan persistence, sometimes we have to truncate the session table because of unmarshalling problems due to infinispan upgrades, between majors version upgrades.
Will it still be the case with new persistence in keycloak database (no guarantee of compatibility when upgrading) or is it something that you are also working on ?

[ahus1]

The current nightly version already support upgrading. During an upgrade all entries in Infinispan will be cleared, an all entries will be reloaded on-demand from the database as needed. With this feature, the plan is to make this work in KC25, so upgrades to future minor and major versions should not loose online sessions.

This is the same behavior you have today with offline sessions.

[xgp]

@mhajas @ahus1 Really happy to see the progress on this. Per #27976 (comment) I'm curious if this will also (eventually) encompass User Sessions, Authentication Sessions, User Login Failures, and Single Use Objects. We have a similar experiment for all the "cached" objects using JPA via the DatastoreProvider SPI (https://github.com/xgp/keycloak-jpa-cache). I'm curious if there will be an option for full-blown Infinispan "replacement", and if we can contribute any work/code there.

[ahus1]

This focuses on user sessions (online and offline) only, as there is the greatest pain, and this discussion should focus on these. Once this is GA, we'll consider the next step on this evolutionary journey.

[xgp]

Thanks @ahus1 I'll open (or find) a separate discussion for the future issues.

[ahus1]

Instead of opening new issues now, I'd rather like you to hold back on new issues as there is already the one about seamless upgrades in #24655. Once this is closed, we can detail the next steps.

For now, it would be great if you could provide feedback on the nightly builds or the referenced PRs above to get this ready for KC25, and to avoid too many scattered activities.

[ahus1]

I would be curious for which scenarios you know this would work, and for which other scenarios it wouldn't work.

[matar3s]

Hi All,

We use Keycloak, and currently, version 23.0.7 is live. Our issue is that the session breaks when we modify a theme during a new deployment (unfortunately, every update for us involves theme modification), causing sessions to be cleared and users to be logged out, which is very inconvenient.

We believe that #28271 would solve our problem. Therefore, our question is, when can we expect the deployment of this ticket?

We tested the nightly, and it looks fine prefer not to deploy nightly builds into production

Is there any plan that when you will deliver this development? :)

Thank you in advance for your assistance and information :)

[dasniko]

Why don't you just test the nightly build in your test environment and see, if it solves your problem, instead of asking directly for production support? After your tests you can contribute and give valuable feedback to the team and all the other users here.

[matar3s]

We tested it and looks fine in our test environment, just we don't want to use nightly at prod.

That's why i'm asked when we can expect KC25?

Maybe my my wording was inaccurate, sorry

[dasniko]

Thanks for clarifying, sounds good! :)

See the milestone https://github.com/keycloak/keycloak/milestone/47 for planned date. Don't (never) rely on it, see this as planned and hopefully to be available around this time. Things can change every time.

[ahus1]

Our issue is that the session breaks when we modify a theme during a new deployment (unfortunately, every update for us involves theme modification), causing sessions to be cleared and users to be logged out, which is very inconvenient.

When you re-deploy Keycloak with a new version of a theme, you should use rolling updates, where each instance of Keycloak is restarted one after the other. This is already supported in Keycloak for many versions.

The one thing that Keycloak doesn't support is running different versions concurrently (like 24.0.1 and 24.0.2 at the same time, or 23.0.0 and 24.0.0).

[darius-m]

Will this change also allow (or at least make it easier) to display recently expired sessions for users in the account console? I am assuming that since Infinispan is a cache, it's a bit harder to query for all user sessions, including expired ones, and clearing the information from the cache as soon as possible is best. It would be a pretty good boon from a security standpoint to be able to review recent user sessions, similar to how Microsoft displays such information.

[ahus1]

@darius-m - no, this is not part of this effort. A current functionality is to enable events and store in Keycloak's database. I know a previous version of the account console was able to show some of the events, but the current doesn't allow that AFAIK.

@ssilvert - I think I remember a Keycloak issue to get this functionality back, but I didn't find it. Can you please post an update here with the current plans / a link to the issue?

[tu-pm]

+1.

I'd be great if there is a REST API to retrieve user sessions to implement several important security features requested by our customers on the account console, such as showing the latest sessions, and if possible the user's location of each login session.

[dasniko]

@tu-pm In the account API thhere is already an endpoint to retrieve the current user's sessions, it's called devices (why ever): /realms/demo/account/sessions/devices, it's also displayed in the account console and the user is able to sign out of other sessions the current one:

However, expired sessions are not displayed.

[ssilvert]

@ssilvert - I think I remember a Keycloak issue to get this functionality back, but I didn't find it. Can you please post an update here with the current plans / a link to the issue?

@ahus1
I guess we never created an issue for it. We don't have any current plans to restore this functionality and I don't remember why (if?) we decided against it. All I could find is this PR where I removed the flag for enabling it.
https://github.com/keycloak/keycloak/pull/21039/files

If the account REST API is still there for this then I guess it wouldn't hurt to add this function back in. If the REST API is missing then that probably means someone had a very good reason for getting rid of it. Maybe open a discussion on the dev list?

[minusdavid]

I don't have much to add other than that this would be well received by our users.