Keycloak 17 installation broken #11316

EinarArnason · 2022-04-14T12:52:27Z

Describe the bug

After upgrading on Arch Linux from 16.1.0 to 17.x I can't get past this error:

kc.sh --verbose build
Updating the configuration and installing your custom providers, if any. Please wait.
2022-04-14 12:48:03,819 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 10814ms
ERROR: Failed to run 'build' command.
Error details:
picocli.CommandLine$ExecutionException: Failed to update server configuration.
	at org.keycloak.quarkus.runtime.Messages.cliExecutionError(Messages.java:51)
	at org.keycloak.quarkus.runtime.cli.command.AbstractCommand.executionError(AbstractCommand.java:36)
	at org.keycloak.quarkus.runtime.cli.command.Build.run(Build.java:81)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:84)
	at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:77)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:60)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:31)
Caused by: java.lang.ExceptionInInitializerError
	at org.keycloak.quarkus.runtime.configuration.Configuration.getRawPersistedProperty(Configuration.java:73)
	at org.keycloak.quarkus.runtime.configuration.Configuration.getBuildTimeProperty(Configuration.java:53)
	at org.keycloak.quarkus.runtime.Environment.isDevMode(Environment.java:140)
	at org.keycloak.quarkus.runtime.cli.command.Build.run(Build.java:76)
	... 16 more
Caused by: java.lang.RuntimeException: Failed to load persisted properties from /usr/share/java/keycloak/bin/../lib/quarkus/generated-bytecode.jar
	at org.keycloak.quarkus.runtime.configuration.PersistedConfigSource.loadPersistedConfig(PersistedConfigSource.java:120)
	at org.keycloak.quarkus.runtime.configuration.PersistedConfigSource.readProperties(PersistedConfigSource.java:72)
	at org.keycloak.quarkus.runtime.configuration.PersistedConfigSource.<init>(PersistedConfigSource.java:47)
	at org.keycloak.quarkus.runtime.configuration.PersistedConfigSource.<clinit>(PersistedConfigSource.java:44)
	... 20 more
Caused by: java.util.zip.ZipException: invalid entry CRC (expected 0xa72a76eb but got 0xf9c4b42)
	at java.base/java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:410)
	at java.base/java.util.zip.ZipInputStream.read(ZipInputStream.java:199)
	at java.base/java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:143)
	at java.base/java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:121)
	at org.keycloak.quarkus.runtime.configuration.PersistedConfigSource.loadPersistedConfig(PersistedConfigSource.java:114)
	... 23 more

Version

17.0.1

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Anything else?

No response

EinarArnason · 2022-04-14T16:15:10Z

Downgrading zlib (1:1.2.12-1 => 1:1.2.11-5) worked to get around the issue.

EinarArnason · 2022-04-14T18:52:54Z

Now it strands at this point

kc.sh --verbose -cf /etc/keycloak/keycloak.conf start
2022-04-14 18:42:43,969 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: login.systemat.is, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: 8,087, Proxied: true
2022-04-14 18:42:45,333 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-04-14 18:42:45,409 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-04-14 18:42:45,439 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-04-14 18:42:45,707 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.6.Final
2022-04-14 18:42:45,822 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2022-04-14 18:42:45,823 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2022-04-14 18:42:45,918 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-04-14 18:42:45,919 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20.00MB, but the OS only allocated 212.99KB
2022-04-14 18:42:45,920 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2022-04-14 18:42:45,921 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25.00MB, but the OS only allocated 212.99KB
2022-04-14 18:42:47,933 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) gamla-50976: no members discovered after 2002 ms: creating cluster as coordinator
2022-04-14 18:42:47,945 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [gamla-50976|0] (1) [gamla-50976]
2022-04-14 18:42:47,951 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `gamla-50976`, physical addresses are `[192.168.3.100:46092]`
2022-04-14 18:42:48,572 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: gamla-50976, Site name: null
2022-04-14 18:42:50,540 INFO  [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel `ISPN`
2022-04-14 18:42:50,618 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2022-04-14 18:42:50,618 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) Error details:: java.lang.NullPointerException
	at org.keycloak.models.cache.infinispan.entities.CachedRealm.<init>(CachedRealm.java:266)
	at org.keycloak.models.cache.infinispan.RealmCacheSession.getRealm(RealmCacheSession.java:414)
	at org.keycloak.services.managers.ApplianceBootstrap.isNewInstall(ApplianceBootstrap.java:46)
	at org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory.createMasterRealm(QuarkusJpaConnectionProviderFactory.java:232)
	at org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory.initSchema(QuarkusJpaConnectionProviderFactory.java:199)
	at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:239)
	at org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory.postInit(QuarkusJpaConnectionProviderFactory.java:136)
	at org.keycloak.quarkus.runtime.integration.QuarkusKeycloakSessionFactory.init(QuarkusKeycloakSessionFactory.java:96)
	at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.initializeKeycloakSessionFactory(QuarkusKeycloakApplication.java:68)
	at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.startup(QuarkusKeycloakApplication.java:49)
	at org.keycloak.quarkus.runtime.integration.QuarkusLifecycleObserver.onStartupEvent(QuarkusLifecycleObserver.java:37)
	at org.keycloak.quarkus.runtime.integration.QuarkusLifecycleObserver_Observer_onStartupEvent_b0e82415b143738dc1f986a5fa4668e83d0a5dea.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:320)
	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:302)
	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:73)
	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:128)
	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:97)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:103)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
	at org.keycloak.quarkus.runtime.KeycloakMain.start(KeycloakMain.java:86)
	at org.keycloak.quarkus.runtime.cli.command.AbstractStartCommand.run(AbstractStartCommand.java:34)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:84)
	at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:77)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:60)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:31)

EinarArnason · 2022-04-15T11:01:52Z

I couldn't directly use the existing database. I had to let keycloak 17 create a new database and then swap them after that. Then it started up.

pedroigor · 2022-04-18T14:13:01Z

@EinarArnason Could you please describe the steps you did during the upgrade?

EinarArnason · 2022-04-20T20:13:59Z

I upgraded the package from 16.1.0 to 17.0.1
Migrated configuration from standalone.xml to keycloak.conf with PostgreSQL connection
Started the service

RaitoBezarius · 2022-04-27T08:13:32Z

Just ran into this issue: NixOS/nixpkgs#170539 — we have a zlib ≥ 1.2.12 on NixOS too, I performed the same operations as @EinarArnason

Anyone understand how Quarkus works?

pedroigor · 2022-04-27T20:30:41Z

@RaitoBezarius The code failing is not really related to Quarkus but to Keycloak itself.

The file failing the CRC check is indeed generated by Quarkus (lib/quarkus/generated-bytecode.jar) during re-augmentation but we are reading it in Keycloak to load the properties set when running the build command.

I'm not sure why CRC is failing for that file but if changing zlib version solves the problem, perhaps this can be a bug in the JVM version? Or even with zlib?

RaitoBezarius · 2022-04-27T21:37:34Z

@pedroigor it is a bit of a shame to have to downgrade zlib, how can we debug further the issue?

pedroigor · 2022-04-28T11:39:17Z

@RaitoBezarius Not really in favor of the downgrade. I'm kinda intrigued about this issue and why the JAR is corrupted.

I would like to reproduce it locally but I did manage to do it. Do you have the steps I can follow to reproduce it?

RaitoBezarius · 2022-04-28T12:00:12Z

@pedroigor If you have Nix in the hand: nix-build thisfile.nix -A postgres with this file: https://clbin.com/bMYYg ; it will try to build Keycloak with a specific configuration file and when kc.sh build is run, it will fail.

It seems like @EinarArnason succeeded to get this issue on Archlinux with a recent enough zlib, I assume it requires also a configuration file.

In the original NixOS automatic test, we tested with providing a snakeoil SSL certificate, and it passed, if I remove these, it fails.
(As most deployments are using a reverse proxy + LE, Keycloak is unusable at the moment on our latest channels.)

To be precise, here's our build phase: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/servers/keycloak/default.nix#L25-L46 — the configuration we install is https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/servers/keycloak/default.nix#L28 ; then we install plugins if any and run the build. We crash there: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/servers/keycloak/default.nix#L43

So to summary:

A non-trivial configuration file in conf/keycloak.conf
kc.sh build
zlib ≥ 1.2.12

I mailed the zlib authors to ask if they had any idea on what could be going on.

RaitoBezarius · 2022-06-09T16:07:48Z

As per NixOS/nixpkgs#177053

We have found out the issue is related to a change in how zlib verifies checksums, see madler/zlib#618 and a patch madler/zlib@ec3df00 seems to fix the issue.

It is not specific to Keycloak except that Keycloak uses Quarkus augmentation feature, which seems to be reliant on zlib or some CRC implementations that I am not aware of.

If Keycloak people are fine with it, I'm okay with closing the issue here, @pedroigor — at some point, upstream will release a new version and rolling release distributions will catch up + backport and other distributions will skip this version.

pedroigor · 2022-06-09T16:29:02Z

@RaitoBezarius Thanks, I appreciate your help and feedback on this one. I'll close the issue then.

Looks like other projects, like Flyway, are also having a similar issue as per this link madler/zlib#618. So I guess this "bad behavior" now being reverted is JDK-related and not specific to a project.

If you think it makes sense I can reach out Quarkus team (as there are more smart people than me there to look at this).

RaitoBezarius · 2022-06-09T17:09:38Z

@pedroigor If you have an easier way to give a heads-up to the Quarkus team, that would be great, I think. Of course, I think we tried hard enough to link all different threads to let everyone who run into this figure out what to do, so they might stumble on this by themselves I imagine.

Thank you for your time and quick answers!

Starting zlib 1.2.12, CRC validation has became stricter. This broke Keycloak ≥ 17 in certain situations, for details, see: - keycloak/keycloak#11316 ; - NixOS#170539 This patch makes the CRC validation comprehensive with respect to older or already existing checksums out there.

Starting zlib 1.2.12, CRC validation has became stricter. This broke Keycloak ≥ 17 in certain situations, for details, see: - keycloak/keycloak#11316 ; - #170539 This patch makes the CRC validation comprehensive with respect to older or already existing checksums out there. (cherry picked from commit 8335c46)

pedroigor · 2022-06-10T13:02:25Z

@RaitoBezarius @EinarArnason Tky u. Created quarkusio/quarkus#26046.

Starting zlib 1.2.12, CRC validation has became stricter. This broke Keycloak ≥ 17 in certain situations, for details, see: - keycloak/keycloak#11316 ; - NixOS#170539 This patch makes the CRC validation comprehensive with respect to older or already existing checksums out there.

EinarArnason added kind/bug Categorizes a PR related to a bug status/triage labels Apr 14, 2022

stianst added the area/dist/quarkus label Apr 21, 2022

RaitoBezarius mentioned this issue Jun 9, 2022

zlib: backport upstream fix on CRC validation NixOS/nixpkgs#177053

Merged

13 tasks

pedroigor closed this as completed Jun 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keycloak 17 installation broken #11316

Keycloak 17 installation broken #11316

EinarArnason commented Apr 14, 2022

EinarArnason commented Apr 14, 2022

EinarArnason commented Apr 14, 2022

EinarArnason commented Apr 15, 2022

pedroigor commented Apr 18, 2022

EinarArnason commented Apr 20, 2022

RaitoBezarius commented Apr 27, 2022

pedroigor commented Apr 27, 2022

RaitoBezarius commented Apr 27, 2022

pedroigor commented Apr 28, 2022

RaitoBezarius commented Apr 28, 2022 •

edited

Loading

RaitoBezarius commented Jun 9, 2022

pedroigor commented Jun 9, 2022

RaitoBezarius commented Jun 9, 2022

pedroigor commented Jun 10, 2022

Keycloak 17 installation broken #11316

Keycloak 17 installation broken #11316

Comments

EinarArnason commented Apr 14, 2022

Describe the bug

Version

Expected behavior

Actual behavior

How to Reproduce?

Anything else?

EinarArnason commented Apr 14, 2022

EinarArnason commented Apr 14, 2022

EinarArnason commented Apr 15, 2022

pedroigor commented Apr 18, 2022

EinarArnason commented Apr 20, 2022

RaitoBezarius commented Apr 27, 2022

pedroigor commented Apr 27, 2022

RaitoBezarius commented Apr 27, 2022

pedroigor commented Apr 28, 2022

RaitoBezarius commented Apr 28, 2022 • edited Loading

RaitoBezarius commented Jun 9, 2022

pedroigor commented Jun 9, 2022

RaitoBezarius commented Jun 9, 2022

pedroigor commented Jun 10, 2022

RaitoBezarius commented Apr 28, 2022 •

edited

Loading