[bitnami/keycloak] Guidance on how to automate realm backup for Keycloak

[scubbo]

Name and Version

bitnami/keycloak:19.3.4

What is the problem this feature will solve?

Exporting realms provides resiliency against persistence failures, and allows an application to be "cold-started" from configuration rather than having to be manually initialized

What is the feature you are proposing to solve the problem?

Documentation on the recommended process to carry out realm exports automatically

What alternatives have you considered?

• Using cron to run kc.sh export from the Keycloak container. cron is not installed on the custom image (and installing new software is not possible), so this is not feasible
• Scheduling a CronJob to run the backup. Other pods do not seem to be fully configured to be able to backup the application's data - the following Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: keycloak-backup
  namespace: keycloak
spec:
  template:
    spec:
      containers:
        - args:
            - /opt/bitnami/keycloak/bin/kc.sh
            - export
            - --file
            - /tmp/realm-export.json
            - --realm
            - new-realm
          env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: BITNAMI_DEBUG
              value: "false"
            - name: KEYCLOAK_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: admin-password
                  name: keycloak
            - name: KEYCLOAK_DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: keycloak-postgresql
            - name: KEYCLOAK_HTTP_RELATIVE_PATH
              value: /
            - name: KEYCLOAK_CACHE_TYPE
              value: local
          envFrom:
            - configMapRef:
                name: keycloak-env-vars
          image: docker.io/bitnami/keycloak:24.0.2
          imagePullPolicy: IfNotPresent
          name: backup-container
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
            - containerPort: 7800
              name: infinispan
              protocol: TCP
          volumeMounts:
            - mountPath: /tmp
              name: empty-dir
              subPath: tmp-dir
            - mountPath: /opt/bitnami/keycloak/conf
              name: empty-dir
              subPath: app-conf-dir
            - mountPath: /opt/bitnami/keycloak/lib/quarkus
              name: empty-dir
              subPath: app-quarkus-dir
      restartPolicy: Never
      volumes:
        - emptyDir: {}
          name: empty-dir

resulted in the following logs:

keycloak 22:21:45.23 INFO  ==>
keycloak 22:21:45.24 INFO  ==> Welcome to the Bitnami keycloak container
keycloak 22:21:45.24 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
keycloak 22:21:45.24 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
keycloak 22:21:45.24 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
keycloak 22:21:45.24 INFO  ==>

Appending additional Java properties to JAVA_OPTS: -Djgroups.dns.query=keycloak-headless.keycloak.svc.cluster.local
Exception in thread "main" java.nio.file.NoSuchFileException: /opt/bitnami/keycloak/lib/quarkus/quarkus-application.dat
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
	at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
	at java.base/java.nio.file.Files.newInputStream(Files.java:160)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:53)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)

[scubbo]

By adding an initContainer, I was able to get the external Job process to run, but the realm could not be found:

Yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: keycloak-backup
  namespace: keycloak
spec:
  template:
    spec:
      containers:
        - args:
            - /opt/bitnami/keycloak/bin/kc.sh
            - export
            - --file
            - /tmp/realm-export.json
            - --realm
            - new-realm
          env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: BITNAMI_DEBUG
              value: "false"
            - name: KEYCLOAK_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: admin-password
                  name: keycloak
            - name: KEYCLOAK_DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: keycloak-postgresql
            - name: KEYCLOAK_HTTP_RELATIVE_PATH
              value: /
            - name: KEYCLOAK_CACHE_TYPE
              value: local
          envFrom:
            - configMapRef:
                name: keycloak-env-vars
          image: docker.io/bitnami/keycloak:24.0.2
          imagePullPolicy: IfNotPresent
          name: backup-container
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
            - containerPort: 7800
              name: infinispan
              protocol: TCP
          volumeMounts:
            - mountPath: /tmp
              name: empty-dir
              subPath: tmp-dir
            - mountPath: /opt/bitnami/keycloak/conf
              name: empty-dir
              subPath: app-conf-dir
            - mountPath: /opt/bitnami/keycloak/lib/quarkus
              name: empty-dir
              subPath: app-quarkus-dir
      initContainers:
        - args:
            - -ec
            - |
              #!/bin/bash
              cp -r /opt/bitnami/keycloak/lib/quarkus/* /quarkus
          command:
            - /bin/bash
          image: docker.io/bitnami/keycloak:24.0.2
          imagePullPolicy: IfNotPresent
          name: init-quarkus-directories
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
               - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsGroup: 0
            runAsNonRoot: true
            runAsUser: 1001
            seccompProfile:
              type: RuntimeDefault
          volumeMounts:
          - mountPath: /tmp
            name: empty-dir
            subPath: tmp-dir
          - mountPath: /quarkus
            name: empty-dir
            subPath: app-quarkus-dir
      restartPolicy: Never
      volumes:
        - emptyDir: {}
          name: empty-dir

Logs:

�[38;5;6mkeycloak �[38;5;5m22:42:25.01 �[0m�[38;5;2mINFO �[0m ==> 
�[38;5;6mkeycloak �[38;5;5m22:42:25.02 �[0m�[38;5;2mINFO �[0m ==> �[1mWelcome to the Bitnami keycloak container�[0m
�[38;5;6mkeycloak �[38;5;5m22:42:25.02 �[0m�[38;5;2mINFO �[0m ==> Subscribe to project updates by watching �[1mhttps://github.com/bitnami/containers�[0m
�[38;5;6mkeycloak �[38;5;5m22:42:25.02 �[0m�[38;5;2mINFO �[0m ==> Submit issues and feature requests at �[1mhttps://github.com/bitnami/containers/issues�[0m
�[38;5;6mkeycloak �[38;5;5m22:42:25.02 �[0m�[38;5;2mINFO �[0m ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit �[1mhttps://bitnami.com/enterprise�[0m
�[38;5;6mkeycloak �[38;5;5m22:42:25.02 �[0m�[38;5;2mINFO �[0m ==> 

Appending additional Java properties to JAVA_OPTS: -Djgroups.dns.query=keycloak-headless.keycloak.svc.cluster.local
Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
2024-04-06 22:42:29,155 WARN  [org.key.services] (build-11) KC-SERVICES0047: metrics (org.jboss.aerogear.keycloak.metrics.MetricsEndpointFactory) is implementing the internal SPI realm-restapi-extension. This SPI is internal and may change without notice
2024-04-06 22:42:29,583 WARN  [org.key.services] (build-11) KC-SERVICES0047: metrics-listener (org.jboss.aerogear.keycloak.metrics.MetricsEventListenerFactory) is implementing the internal SPI eventsListener. This SPI is internal and may change without notice
2024-04-06 22:42:34,127 INFO  [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 6446ms
Server configuration updated and persisted. Run the following command to review the configuration:

	kc.sh show-config

Next time you run the server, just run:

	kc.sh export --file=/tmp/realm-export.json --realm=new-realm --optimized

2024-04-06 22:42:35,859 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
2024-04-06 22:42:36,353 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2024-04-06 22:42:36,624 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2024-04-06 22:42:36,793 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2024-04-06 22:42:36,800 INFO  [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 815f0b4b-a7cb-40c1-8eaf-1003e47c0d92, name: keycloak-backup-pqcrd-12944
2024-04-06 22:42:36,811 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2024-04-06 22:42:36,811 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB
2024-04-06 22:42:36,811 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2024-04-06 22:42:36,811 WARN  [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB
2024-04-06 22:42:36,827 INFO  [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.24863
2024-04-06 22:42:38,279 WARN  [io.quarkus.agroal.runtime.DataSources] (JPA Startup Thread) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2024-04-06 22:42:38,835 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-backup-pqcrd-12944: no members discovered after 2002 ms: creating cluster as coordinator
2024-04-06 22:42:38,883 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-backup-pqcrd-12944|0] (1) [keycloak-backup-pqcrd-12944]
2024-04-06 22:42:38,921 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak-backup-pqcrd-12944`, physical addresses are `[10.42.0.49:40399]`
2024-04-06 22:42:38,972 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2024-04-06 22:42:41,246 INFO  [org.keycloak.quarkus.runtime.storage.legacy.liquibase.QuarkusJpaUpdaterProvider] (main) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml

UPDATE SUMMARY
Run:                        124
Previously run:               0
Filtered out:                 0
-------------------------------
Total change sets:          124

2024-04-06 22:42:43,948 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-backup-pqcrd-12944, Site name: null
2024-04-06 22:42:44,094 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2024-04-06 22:42:44,187 INFO  [org.keycloak.services] (main) KC-SERVICES0050: Initializing master realm
2024-04-06 22:42:46,171 INFO  [org.keycloak.services] (main) KC-SERVICES0034: Export of realm 'new-realm' requested.
2024-04-06 22:42:46,172 INFO  [org.keycloak.exportimport.singlefile.SingleFileExportProvider] (main) Exporting realm 'new-realm' into file /tmp/realm-export.json
2024-04-06 22:42:46,413 INFO  [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel `ISPN`
2024-04-06 22:42:46,446 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (import_export) mode
2024-04-06 22:42:46,447 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: realm not found by realm name 'new-realm'
2024-04-06 22:42:46,447 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.

Demonstration that realm exists:

$ kubectl exec -it keycloak-0 -- bash
Defaulted container "keycloak" out of: keycloak, init-quarkus-directory (init), add-backup-to-crontab (init)
keycloak@keycloak-0:/$ /opt/bitnami/keycloak/bin/kc.sh export --file /tmp/realm-export.json --realm new-realm
Appending additional Java properties to JAVA_OPTS: -Djgroups.dns.query=keycloak-headless.keycloak.svc.cluster.local
2024-04-06 22:47:06,232 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2024-04-06 22:47:06,303 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2024-04-06 22:47:06,521 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: true
2024-04-06 22:47:08,344 WARN  [io.quarkus.agroal.runtime.DataSources] (JPA Startup Thread) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2024-04-06 22:47:09,870 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2024-04-06 22:47:09,893 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: node_308461, Site name: null
2024-04-06 22:47:10,613 INFO  [org.keycloak.services] (main) KC-SERVICES0034: Export of realm 'new-realm' requested.
2024-04-06 22:47:10,613 INFO  [org.keycloak.exportimport.singlefile.SingleFileExportProvider] (main) Exporting realm 'new-realm' into file /tmp/realm-export.json
2024-04-06 22:47:11,584 INFO  [org.keycloak.services] (main) KC-SERVICES0035: Export finished successfully
2024-04-06 22:47:11,616 INFO  [io.quarkus] (main) Keycloak 24.0.2 on JVM (powered by Quarkus 3.8.3) started in 7.483s. Listening on:
2024-04-06 22:47:11,616 INFO  [io.quarkus] (main) Profile import_export activated.
2024-04-06 22:47:11,616 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, logging-gelf, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, vertx]
2024-04-06 22:47:11,675 INFO  [io.quarkus] (main) Keycloak stopped in 0.055s

I suspect this indicates that the separate Pod is not connecting to the same Keycloak datasource - but I don't know how to configure it to do so.

[scubbo]

I was able to get a separate Pod to export Keycloak realms by adding a --db option to the kc.sh export command. I have a separate issue open with the Keycloak team asking if that is the recommended way to do a scheduled import. Whatever their response is should probably be included in documentation for this chart (or, better yet, added as a value-controlled set of resources)

[javsalgar]

Hi!

Thank you so much for sharing your tests! The community will benefit from this. Let's see what the upstream devs say so we can improve the README.md file

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.