Zero downtime upgrade of keycloak

[SAppleton]

Description

As a Keycloak admin I want to be able to upgrade Keycloak without downtime

Discussion

No response

Motivation

It was a request from a Red Hat customer and is was captured in https://issues.redhat.com/browse/KEYCLOAK-7301 but seem not to have made it across to the GitHub Issue tracker

Details

see https://issues.redhat.com/browse/KEYCLOAK-7301

[sstojak1]

Hi team, Do we have any updates regarding this feature?

[PSanetra]

@sstojak1 I am not part of the team, but as far as I know the last attempt to support zero downtime upgrades was canceled. This was announced in this blog post: https://www.keycloak.org/2023/10/map-store-removal

[jonkoops]

cc @ahus1

[ahus1]

Thank you for creating this issue.

At the moment there are no plans to work on this in the near future.

A possible idea for a first attempt: Keep locks due to database schema migrations on upgrades on specific database vendor (to be named) with a yet-to-be-defined number of rows in an existing database table below a given threshold of X seconds.

If someone is interested in investigating which database is best suited as it allow schema upgrades without much downtime, and would be defining a blueprint to fill such a database (possibly based on the efforts in https://github.com/keycloak/keycloak-benchmark, then we'd by happy to join this discussion.

For now, I'm marking this "help wanted".

[Corinari]

As of now, is there a possibility of HA zero downtime Upgrades, even if you have two instances of Keycloak running, with either one database or a shared database (mysql master-master)?

[y-tabata]

@ahus1

If someone is interested in investigating which database is best suited as it allow schema upgrades without much downtime, and would be defining a blueprint to fill such a database (possibly based on the efforts in https://github.com/keycloak/keycloak-benchmark, then we'd by happy to join this discussion.

How about using CockroachDB's online schema changes?
I'd also like to know the current status of CockroachDB support.

[ahus1]

Thank you @y-tabata for raising this. The "small print" in the docs of CockroachDB is:

• "Your application's queries can run normally" - which implies that writes don't work, which is expected, but implies that there is still some downtime.
• "[CockroachDB supports] one schema change operation per transaction" / "CockroachDB only supports DDL changes within implicit transactions"

PostgreSQL on the other hand can do that for schema changes within a database. PostgreSQL also supports concurrent index builds, something we don't leverage in Keycloak as of today.

When we tested CockroachDB for the map storage, it was one of the slowest databases and memory/CPU hungry I have ever seen (especially compared to a regular PostgreSQL). Some PostgreSQL features didn't work (for example some kind of indexes), the serializable transaction level lead to deadlocks and rolled back transactions in concurrency situations, and the locking of rows was giving us nightmares even if there were using the primary keys.

I would happy to listen if someone provides the pros and cons for alternative databases, and shows the additional benefits and costs of such a database. Still, at the end it would not only be me to convince here, but some additional maintainers as well.