OIDC with Zitadel SaaS stops working after some time (signature could not be validated using the provided keys)

[baua1310]

Describe the Bug

When Zitadel SaaS is used for authentication with OIDC in Bookstack, it will stop working after some time, at the latest after 24 hours, and the following error message is shown: ID token validation failed with error: Token signature could not be validated using the provided keys.

Workaround: Deleting the bookstack docker container and recreating it fixes the error for some hours.

Steps to Reproduce

1. set up OIDC with Zitadel SaaS as described in OIDC using Zitadel does not work #4682 by @megastary
2. test successful sign in with SSO
3. wait 24 hours
4. retry sign in with SSO
5. see error: ID token validation failed with error: Token signature could not be validated using the provided keys

Expected Behaviour

When set up correctly, authentication with OIDC in bookstack works also after 24 hours.

Screenshots or Additional Context

Browser Details

Brave (1.66.118 Chromium: 125.0.6422.147 (Official Build) (64-bit)) on Windows 11 Version 23H2 (Build 22631.3593)

Exact BookStack Version

v24.05.1

[ssddanbrown]

As mentioned in #4682, BookStack does cache discovered details but only for 15 minutes.

First, it would be good to test/rule-out instance cache issues.
Can you try setting the cache to be database based.
This is done by setting CACHE_DRIVER=database in your existing .env file, or by setting CACHE_DRIVER=database to the environment for your BookStack app container. Remember to re-create the container if altering container environment options.

[baua1310]

@ssddanbrown I have set the suggested environment variable:

user@SRV001:/opt/bookstack$ docker compose ps
NAME            IMAGE                                  COMMAND                  SERVICE         CREATED       STATUS       PORTS
bookstack_app   lscr.io/linuxserver/bookstack:latest   "/init"                  bookstack_app   7 hours ago   Up 4 hours   80/tcp, 443/tcp
bookstack_db    mariadb:11                             "docker-entrypoint.s…"   bookstack_db    7 hours ago   Up 4 hours   3306/tcp
user@SRV001:/opt/bookstack$ docker exec bookstack_app printenv
PATH=/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=581ed5423b8b
APP_THEME=custom
OIDC_ISSUER_DISCOVER=true
CACHE_DRIVER=database
OIDC_NAME=ZITADEL
OIDC_DISPLAY_NAME_CLAIMS=name
MAIL_ENCRYPTION=tls
MAIL_PASSWORD=somepassword
PUID=1000
OIDC_GROUPS_CLAIM=custom:roles
DB_PASS=somepassword
OIDC_END_SESSION_ENDPOINT=false
MAIL_DRIVER=smtp
DB_USER=bookstack
OIDC_REMOVE_FROM_GROUPS=true
OIDC_CLIENT_ID=someid@wiki
DB_HOST=bookstack_db
APP_DEBUG=true
AUTH_AUTO_INITIATE=true
MAIL_HOST=smtp-relay.brevo.com
MAIL_FROM=wiki@some.domain
PGID=1000
APP_URL=https://wiki.some.domain
OIDC_CLIENT_SECRET=somesecret
DB_DATABASE=bookstack
AUTH_METHOD=oidc
OIDC_ADDITIONAL_SCOPES=urn:zitadel:iam:org:projects:roles
MAIL_FROM_NAME=Wiki
OIDC_USER_TO_GROUPS=true
OIDC_ISSUER=https://some-instance.zitadel.cloud
DB_PORT=3306
MAIL_PORT=587
MAIL_USERNAME=mail@some.domain
PS1=$(whoami)@$(hostname):$(pwd)\$
HOME=/root
TERM=xterm
S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
S6_VERBOSITY=1
S6_STAGE2_HOOK=/init-hook
VIRTUAL_ENV=/lsiopy
LSIO_FIRST_PARTY=true

Unfortunately, the error pattern persists. Even recreating the container does not solve the problem. This means that signing in with OIDC is not possible at all.

[ssddanbrown]

@baua1310 Does it start working again after removing the CACHE_DRIVER option again, and a container recreate?

[baua1310]

@ssddanbrown Yes after removing CACHE_DRIVER and recreating the container sign in with OIDC started working again.

[PandieTech]

Just wanted to +1, I have the exact same problem after implementing a fresh Bookstack instance using Zitadel as my IDP, albeit mine is self hosted not SaaS Zitadel. Followed the same path from the other issue #4682, and now after some time, not necessarily 24 hours, I get the ID token validation failed with error: Token signature could not be validated using the provided keys error.

Edit: I have not restarted / rebuilt my Bookstack container, and after waiting a while after experiencing the error, it did start working again, probably when that cache expired and it renegotiated.

To add some more possibly worthwhile info, certain OIDC token settings can be modified in Zitadel (at least in self hosted, can't speak for SaaS). @ssddanbrown would you normally expect other OIDC IDPs to have a longer access, or in this case probably more relevant ID token expiry?

And apologies for my novice showing, but initially I did not include the Refresh Token grant type, will adding that have any bearing on this ID token issue?

Ultimately I suppose it would be nice, selfishly for this scenario, for this error to then prompt Bookstack to clear that cache and retrieve it fresh, but I'm sure it's easier said than done.

[ssddanbrown]

@pandieme

would you normally expect other OIDC IDPs to have a longer access, or in this case probably more relevant ID token expiry?
[...] I did not include the Refresh Token grant type, will adding that have any bearing on this ID token issue?

Those settings don't really matter in the context of BookStack's use. We don't hold on to tokens for refresh, I believe we just go through the original process direct with the auth provided at login time.

Ultimately I suppose it would be nice, selfishly for this scenario, for this error to then prompt Bookstack to clear that cache and retrieve it fresh

The BookStack cache should only remain for 15 minutes, that's what I can't understand and need to dive deeper into. The above tests indicate it's not being cleared (not expiring), but I feel this would have show up in other auth providers/scenarios if our cache system was fundementally not listening to cache lifetime.

Out of interest, can you describe your environment too (BookStack hosting environment& install method) just so I can guage any potential patterns?

[PandieTech]

@ssddanbrown

Out of interest, can you describe your environment too (BookStack hosting environment& install method) just so I can guage any potential patterns?

Ours is running on a Debian 12 VM, hosted on an XCP-ng pool, using Docker compose and behind a Caddy reverse proxy also running in a container on a shared Docker network.

Docker versions

Docker Compose version v2.28.1
Docker version 27.0.3, build 7d4bcd8

Bookstack

Container info

../bookstack# docker compose ps
NAME               IMAGE                           COMMAND   SERVICE        CREATED        STATUS        PORTS
bookstack      lscr.io/linuxserver/bookstack   "/init"   bookstack      41 hours ago   Up 41 hours   80/tcp, 443/tcp
bookstack_db   lscr.io/linuxserver/mariadb     "/init"   bookstack_db   41 hours ago   Up 41 hours   3306/tcp

../bookstack# docker exec bookstack printenv
PATH=/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=b1317d85c73f
DB_DATABASE=bookstackapp
PGID=1000
OIC_DUMP_USER_DETAILS=false
MAIL_FROM=our@email.address
AUTH_AUTO_INITIATE=false
OIDC_ISSUER_DISCOVER=true
OIDC_GROUPS_CLAIM=groups
OIDC_CLIENT_SECRET=ourclientsecret
OIDC_ISSUER=https://idp.fqdn
MAIL_DRIVER=smtp
DB_PASS=ourpass
AUTH_METHOD=oidc
DB_USER=bookstack
MAIL_USERNAME=ouruser
TZ=Europe/London
MAIL_PORT=587
OIDC_USER_TO_GROUPS=true
OIDC_DISPLAY_NAME_CLAIMS=name
MAIL_PASSWORD=ourpass
OIDC_REMOVE_FROM_GROUPS=true
MAIL_FROM_NAME=Intranet
APP_THEME=custom
DB_PORT=3306
MAIL_ENCRYPTION=tls
APP_URL=https://our.fqdn
MAIL_HOST=smtp.fqdn
DB_HOST=bookstack_db
OIDC_NAME=SSO
OIDC_END_SESSION_ENDPOINT=false
OIDC_CLIENT_ID=ourclientid
DB_ROOT_PASS=ourpass
PUID=1000
PS1=$(whoami)@$(hostname):$(pwd)\$
HOME=/root
TERM=xterm
S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
S6_VERBOSITY=1
S6_STAGE2_HOOK=/init-hook
VIRTUAL_ENV=/lsiopy
LSIO_FIRST_PARTY=true

Docker Compose

---
services:
  bookstack:
    image: lscr.io/linuxserver/bookstack
    container_name: bookstack
    environment:
      PUID: 1000
      PGID: 1000
      TZ: Europe/London
      APP_URL: https://our.fqdn
      DB_HOST: bookstack_db
      DB_PORT: 3306
      DB_USER: bookstack
      DB_PASS: ${DB_PASS}
      DB_DATABASE: bookstackapp
    volumes:
      - app_data:/config
    restart: unless-stopped
    depends_on:
      - bookstack_db
    networks:
      - default
      - caddy
    env_file:
      - .env

  bookstack_db:
    image: lscr.io/linuxserver/mariadb
    container_name: bookstack_db
    environment:
      PUID: 1000
      PGID: 1000
      TZ: Europe/London
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASS}
      MYSQL_DATABASE: bookstackapp
      MYSQL_USER: bookstack
      MYSQL_PASSWORD: ${DB_PASS}
    volumes:
      - db_data:/config
    restart: unless-stopped
    env_file:
      - .env

volumes:
  app_data:
    name: bookstack_app_data
  db_data:
    name: bookstack_db_data

networks:
  caddy:
    name: caddy
    external: true

.env

DB_ROOT_PASS=ourpass
DB_PASS=ourpass

AUTH_METHOD=oidc
AUTH_AUTO_INITIATE=false

OIDC_NAME=SSO
OIDC_DISPLAY_NAME_CLAIMS=name
OIDC_CLIENT_ID=ourclientid
OIDC_CLIENT_SECRET=ourclientsecret
OIDC_ISSUER=https://idp.fqdn
OIDC_END_SESSION_ENDPOINT=false
OIDC_ISSUER_DISCOVER=true

APP_THEME=custom

OIDC_DUMP_USER_DETAILS=false

OIDC_USER_TO_GROUPS=true
OIDC_GROUPS_CLAIM=groups
OIDC_REMOVE_FROM_GROUPS=true

MAIL_DRIVER=smtp
MAIL_HOST=smtp.fdqn
MAIL_PORT=587
MAIL_ENCRYPTION=tls
MAIL_USERNAME=ouruser
MAIL_PASSWORD=ourpass
MAIL_FROM=our@email.fqdn
MAIL_FROM_NAME=Intranet

Custom theme function

As described in this issue #4682 we added the following file to our bookstack containers volume to Replace multiple aud values with single azp value to cater for Zitadel's returned array.

# /var/lib/docker/volumes/bookstack_app_data/_data/www/themes/custom

<?php

use BookStack\Facades\Theme;
use BookStack\Theming\ThemeEvents;

Theme::listen(ThemeEvents::OIDC_ID_TOKEN_PRE_VALIDATE, function (array $idTokenData, array $accessTokenData) {
    if (is_array($idTokenData['aud']) && in_array($idTokenData['azp'], $idTokenData['aud'])) {
        return array_merge($idTokenData, [
            'aud' => [$idTokenData['azp']]
        ]);
    }
});

Caddy (Reverse Proxy)

Caddyfile

our.fqdn {
        reverse_proxy http://bookstack
}

Docker Compose

services:
  caddy:
    container_name: caddy
    image: caddy:2.8.4
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - data:/data:rw
      - config:/config:rw
    networks:
      - caddy

volumes:
  data:
  config:

networks:
  caddy:
    name: caddy

[Rob787]

+1 for this issue also with self-hosted Zitadel

[ssddanbrown]

@Rob787 Are you also using the linuxserver docker image?

[Rob787]

@ssddanbrown For Bookstack? Yes, the linuxserver Docker image indeed.

[cybrwshl]

@ssddanbrown have you had time to take a closer look at the problem?

[baua1310]

Hi, just a quick update from my side: As the problem with OIDC and Zitadel had not been solved, I switched to SAML 2.0.
SAML 2.0 and Zitadel is working fine since a few months. Here is my configuration:

AUTH_METHOD=saml2
AUTH_AUTO_INITIATE=true
SAML2_NAME=ZITADEL
SAML2_EMAIL_ATTRIBUTE=Email
SAML2_EXTERNAL_ID_ATTRIBUTE=UserID
SAML2_DISPLAY_NAME_ATTRIBUTES=FullName
SAML2_IDP_ENTITYID=https://<domain>.zitadel.cloud/saml/v2/metadata
SAML2_AUTOLOAD_METADATA=true
SAML2_USER_TO_GROUPS=true
SAML2_GROUP_ATTRIBUTE=roles
SAML2_REMOVE_FROM_GROUPS=true

To map Zitadel roles to bookstack groups I added added a custom script in Zitadel called "samlRoles" inside Actions and added the script in the flow "Complement SAMLResponse" to the trigger "Pre SAMLResponse creation".

/**
 * Add an custom attribute to the SAMLResponse, if it's not already present.
 * Adds the roles as an additional attribute to the SAMLResponse, if it's not already present.
 *
 * Flow: Complement SAMLResponse, Triggers: Pre SAMLResponse creation
 *
 * @param ctx
 * @param api
 */
function samlRoles(ctx, api) {
  if (ctx.v1.user.grants == undefined || ctx.v1.user.grants.count == 0) {
    return;
  }

  let roles = [];
  ctx.v1.user.grants.grants.forEach(grant => {
    grant.roles.forEach(role => {
        roles.push(role)  
    })
  })

  api.v1.attributes.setCustomAttribute('roles', '', ...roles)
}

[cybrwshl]

Thanks @baua1310, works like a charm! Except for the logout process, but that's no problem.