[bitnami/keycloak] Admin URL no longer works for chart versions higher than 21.0.2

[CaptainKrby]

Name and Version

bitnami/keycloak:21.2.1

What architecture are you using?

None

What steps will reproduce the bug?

Here is my values.yaml :

keycloak:
  replicaCount: 1
  production: true
  proxy: edge
  httpRelativePath: "/auth/"
  podAntiAffinityPreset: hard
  clusterDomain: k8s-valid.local

  auth:
    adminUser: root
    adminPassword: "xxxx"

  resources:
    requests:
      cpu: 2
      memory: 512Mi
    limits:
      cpu: 3
      memory: 1024Mi

  ingress:
    enabled: true
    ingressClassName: "nginx"
    hostname: keycloak-dr.company.net
    path: "{{ .Values.httpRelativePath }}"
    pathType: ImplementationSpecific
    servicePort: http
    annotations:
      nginx.ingress.kubernetes.io/app-root: /auth/realms/realm-internal/account/#
      nginx.ingress.kubernetes.io/affinity: cookie
    extraTls:
      - hosts:
          - keycloak-dr.company.net

  adminIngress:
    enabled: true
    ingressClassName: "nginx"
    hostname: keycloak-dr-admin.company.net
    path: "{{ .Values.httpRelativePath }}"
    pathType: ImplementationSpecific
    servicePort: http
    annotations:
      nginx.ingress.kubernetes.io/app-root: "{{ .Values.httpRelativePath }}"
      nginx.ingress.kubernetes.io/affinity: cookie
    extraTls:
      - hosts:
          - keycloak-dr-admin.company.net

  networkPolicy:
    enabled: false

  cache:
    enabled: true
    stackName: kubernetes
    stackFile: "/opt/bitnami/keycloak/conf/cache-ispn.xml"

  postgresql:
    enabled: true
    architecture: replication
    auth:
      username: keycloak
      password: "xxxxxx"
      database: keycloakdb
      postgresPassword: "yyyy"
      replicationPassword: "zzzz"
    primary:
      persistence:
        existingClaim: keycloak-dr-primary-pvc
    readReplicas:
      replicaCount: 1
      podAntiAffinityPreset: hard
      persistence:
        existingClaim: keycloak-dr-read-pvc

What do you see instead?

https://keycloak-dr-admin.company.net/admin/ returns :

404: Not Found

Additional information

I tried to upgrade from bitnami/keycloak:21.0.2 to bitnami/keycloak:21.2.1.

This added a new discovery port as well as a change on the tls part of the admin ingress, however I do not notice any typo.

Here are the pod startup logs :

keycloak keycloak 08:25:06.70 INFO  ==> 
keycloak keycloak 08:25:06.71 INFO  ==> Welcome to the Bitnami keycloak container
keycloak keycloak 08:25:06.71 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
keycloak keycloak 08:25:06.71 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
keycloak keycloak 08:25:06.71 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
keycloak keycloak 08:25:06.71 INFO  ==> 
keycloak keycloak 08:25:06.71 INFO  ==> ** Starting keycloak setup **
keycloak keycloak 08:25:06.73 INFO  ==> Validating settings in KEYCLOAK_* env vars...
keycloak keycloak 08:25:06.74 INFO  ==> Trying to connect to PostgreSQL server keycloak-dr-postgresql-primary...
keycloak keycloak 08:25:06.75 INFO  ==> Found PostgreSQL server listening at keycloak-dr-postgresql-primary:5432
keycloak keycloak 08:25:06.76 INFO  ==> Configuring database settings
keycloak keycloak 08:25:06.78 INFO  ==> Enabling statistics
keycloak keycloak 08:25:06.79 INFO  ==> Enabling health endpoints
keycloak keycloak 08:25:06.80 INFO  ==> Configuring http settings
keycloak keycloak 08:25:06.82 INFO  ==> Configuring hostname settings
keycloak keycloak 08:25:06.82 INFO  ==> Configuring cache count
keycloak keycloak 08:25:06.84 INFO  ==> Configuring log level
keycloak keycloak 08:25:06.85 INFO  ==> Configuring proxy
keycloak 
keycloak keycloak 08:25:06.86 INFO  ==> ** keycloak setup finished! **
keycloak keycloak 08:25:06.87 INFO  ==> ** Starting keycloak **
keycloak Appending additional Java properties to JAVA_OPTS: -Djgroups.dns.query=keycloak-dr-headless.keycloak-dr.svc.k8s-valid.local
keycloak Changes detected in configuration. Updating the server image.
keycloak Updating the configuration and installing your custom providers, if any. Please wait.
Stream closed EOF for keycloak-dr/keycloak-dr-0 (init-quarkus-directory)
keycloak 2024-05-17 08:25:08,028 WARN  [org.key.qua.run.cli.Picocli] (main) The following used options or option values are DEPRECATED and will be removed in a future release:
keycloak     - proxy: Use proxy-headers.
keycloak Consult the Release Notes for details.
keycloak 2024-05-17 08:25:12,087 WARN  [org.key.services] (build-46) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
keycloak 2024-05-17 08:25:12,693 WARN  [org.key.services] (build-46) KC-SERVICES0047: metrics-listener (org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory) is implementing the internal SPI eventsListener. This SPI is internal and may change without notice
keycloak 2024-05-17 08:25:16,103 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index org.apache.tools.ant.Task: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,183 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index org.springframework.core.io.DefaultResourceLoader: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,184 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index org.springframework.core.io.ResourceLoader: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,187 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index org.springframework.core.io.Resource: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,309 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index jakarta.jms.Connection: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,315 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index jakarta.jms.XAConnection: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,316 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index jakarta.jms.XASession: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,316 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index jakarta.jms.XAConnectionFactory: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,334 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index org.apache.activemq.artemis.core.journal.RecordInfo: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,335 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index org.apache.activemq.artemis.core.journal.Journal: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:16,337 WARN  [io.qua.dep.ind.IndexWrapper] (build-11) Failed to index io.mashona.logwriting.ArrayStore: Class does not exist in ClassLoader QuarkusClassLoader:Deployment Class Loader: PROD for keycloak@632aa1a3
keycloak 2024-05-17 08:25:24,637 INFO  [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 15138ms
keycloak Server configuration updated and persisted. Run the following command to review the configuration:
keycloak 
keycloak     kc.sh show-config
keycloak 
keycloak Next time you run the server, just run:
keycloak 
keycloak     kc.sh -cf=/opt/bitnami/keycloak/conf/keycloak.conf start --optimized
keycloak 
keycloak 2024-05-17 08:25:25,716 WARN  [org.keycloak.quarkus.runtime.cli.Picocli] (main) The following used options or option values are DEPRECATED and will be removed in a future release:
keycloak     - proxy: Use proxy-headers.
keycloak Consult the Release Notes for details.
keycloak 2024-05-17 08:25:26,798 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: https://keycloak-dr-admin.company.net, Admin: keycloak-dr-admin.company.net, Port: -1, Proxied: true
keycloak 2024-05-17 08:25:27,218 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak 2024-05-17 08:25:27,718 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN` with stack `kubernetes`
keycloak 2024-05-17 08:25:27,725 INFO  [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 17ec31c6-e5fc-4b88-b11d-67bdefd3f72e, name: keycloak-dr-0-27105
keycloak 2024-05-17 08:25:27,756 INFO  [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.57800
keycloak 2024-05-17 08:25:29,668 WARN  [io.quarkus.agroal.runtime.DataSources] (JPA Startup Thread) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak 2024-05-17 08:25:29,763 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-dr-0-27105: no members discovered after 2003 ms: creating cluster as coordinator
keycloak 2024-05-17 08:25:29,775 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-dr-0-27105|0] (1) [keycloak-dr-0-27105]
keycloak 2024-05-17 08:25:29,901 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak-dr-0-27105`, physical addresses are `[192.168.50.145:7800]`
keycloak 2024-05-17 08:25:29,924 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak 2024-05-17 08:25:30,936 WARN  [io.quarkus.vertx.http.runtime.VertxHttpRecorder] (main) The X-Forwarded-* and Forwarded headers will be considered when determining the proxy address. This configuration can cause a security issue as clients can forge requests and send a forwarded header that is not overwritten by the proxy. Please consider use one of these headers just to forward the proxy address in requests.
keycloak 2024-05-17 08:25:30,944 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
keycloak 2024-05-17 08:25:30,978 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-dr-0-27105, Site name: null
keycloak 2024-05-17 08:25:32,564 INFO  [io.quarkus] (main) Keycloak 24.0.4 on JVM (powered by Quarkus 3.8.4) started in 7.717s. Listening on: http://0.0.0.0:8080
keycloak 2024-05-17 08:25:32,564 INFO  [io.quarkus] (main) Profile prod activated. 
keycloak 2024-05-17 08:25:32,564 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, logging-gelf, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, vertx]

Thanks for your help.

[bjsee]

Hi,
we have observed a similar issue. With Helm Chart version 21.0.4, everything was functioning as expected. However, after updating to 21.1.0, the Admin WebApp can no longer be accessed because the "auth" part is being removed. When I try to access the URL https://mdomain.de/auth/, it redirects to https://mdomain.de/admin/ instead of https://mdomain.de/auth/admin/.

Here are the chart values used:

httpRelativePath: /auth/
ingress:
    annotations:
        kubernetes.io/ingress.class: nginx
        nginx.org/location-snippets: |
          proxy_set_header X-Forwarded-Proto https;
          proxy_set_header X-Forwarded-Port 443;
        nginx.org/proxy-buffer-size: 128k 
        nginx.org/proxy-buffers: 4 256k
        nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
        cert-manager.io/issuer: cert-issuer
    enabled: true
    hostname: …
    path: /
    pathType: Prefix
    tls:
        - hosts:
            - ...
          secretName: cert-secret
adminIngress:
    annotations:
        kubernetes.io/ingress.class: nginx
    enabled: true
    hostname: …
    path: /auth/admin
    pathType: Prefix
    tls:
        - hosts:
            - …
          secretName: cert-secret

Thanks for your help.

[bjsee]

I just reviewed the changes from 21.0.4 to 21.1.0 and saw that since then the env variables KC_HOSTNAME_URL and KC_HOSTNAME_ADMIN_URL are used instead of KC_HOSTNAME_ADMIN in

charts/bitnami/keycloak/templates/statefulset.yaml

Line 216 in e10625f

- name: KC_HOSTNAME_ADMIN_URL

If I'm not mistaken, it seems that the path defined in the ingress or in httpRelativePath is not used in this context. Is this an oversight, or am I misunderstanding the configuration of the environment variables?

[CaptainKrby]

Any update ?

[alemorcuq]

Thanks for the investigation you did, @bjsee. I can see this in Keycloak's documentation:

hostname-admin-url

Set the base URL for accessing the administration console, including scheme, host, port and path

CLI: --hostname-admin-url
Env: KC_HOSTNAME_ADMIN_URL

Have you tried adding your httpRelativePath to the KC_HOSTNAME_ADMIN_URL environment variable? Perhaps its missing there.

[CaptainKrby]

Hi @alemorcuq, I've been trying all morning to adjust the values but nothing works, even adjusting httpRelativePath to KC_HOSTNAME_ADMIN_URL...
I reiterate that everything was working on chart 21.0.2 and then only 404 errors.

Can you reproduce my environment?

[lerminou]

Hi, it is introduced by the commit: #25386. I'm impacted too.
My use case is to serve the admin ingress on the same host but with the adminIngress as a subPath to allow ip restriction on this path only.

[leunamnauj]

No updates?

[CaptainKrby]

Waiting too. In the meantime, Keycloak updates are blocked...