3.7.0 broke auth_token

[Red-M]

What happened:
An unrelated change pushed the version of grafana-image-renderer in my environment to 3.7.0 which is ok but suddenly it started to receive 401 unauth responses from grafana. A rollback to 3.6.4 fixed this.

I did try hardcoding the token to test on grafana and grafana-image-renderer and it was still getting the same thing.

What you expected to happen:
3.7.0 worked with existing functioning configuration.

How to reproduce it (as minimally and precisely as possible):
Upgrade to 3.7.0 as far as I can see. Happy to provide more detail if required.

Anything else we need to know?:
Running on kubernetes using the grafana provided chart which defaults to latest.

Environment:

• Grafana Image Renderer version: 3.6.4/3.7.0
• Grafana version: v9.4.3 (cf0a135595)
• Installed plugin or remote renderer service: remote
• OS Grafana Image Renderer is installed on: N/A, using docker image in kubernetes
• User OS & Browser: N/A, between services.
• Others: None.

[AgnesToulet]

Hello! Could you share your Kubernetes config files? We tested several scenarios before releasing 3.7.0 and did not notice any issues.

[Red-M]

This is the grafana.ini in the Kube CM:

[alerting]
enabled = false

[analytics]
check_for_updates = true

[auth.azuread]
allow_assign_grafana_admin = true
allow_sign_up = false
auth_url = https://login.microsoftonline.com/tenantID/oauth2/v2.0/authorize
auto_login = false
enabled = true
name = AzureAD
oauth_skip_org_role_update_sync = true
role_attribute_strict = false
scopes = openid email profile
skip_org_role_sync = true
token_url = https://login.microsoftonline.com/tenantID/oauth2/v2.0/token

[auth.generic_oauth]
allow_signup = false
api_url = https://Oauth-provider/userinfo
auth_url = https://Oauth-provider/authorize
enabled = true
name = OAuth
scopes = openid profile email
token_url = https://Oauth-provider/oauth/token

[feature_toggles]
enable = tempoSearch tempoBackendSearch traceqlEditor panelTitleSearch
publicDashboards autoMigrateGraphPanels

[grafana_net]
url = https://grafana.net

[live]
max_connections = 0

[log]
level = warn
mode = console

[panels]
enable_alpha = true

[paths]
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning

[plugins]
enable_alpha = true

[rendering]
callback_url = http://grafana:80/
server_url = http://grafana-image-renderer:8081/render

[server]
domain = grafana.uat
root_url = https://grafana.uat/

[unified_alerting]
enabled = true
ha_peers = grafana-headless:9094

[unified_alerting.screenshots]
capture = true
capture_timeout = 30s

Here is the env variables that the grafana pod has:

env:
  - name: GF_AUTH_GENERIC_OAUTH_CLIENT_ID
    valueFrom:
      secretKeyRef:
        key: client-id
        name: grafana-auth
  - name: GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        key: client-secret
        name: grafana-auth
  - name: GF_AUTH_AZUREAD_CLIENT_ID
    valueFrom:
      secretKeyRef:
        key: aad-client-id
        name: grafana-auth
  - name: GF_AUTH_AZUREAD_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        key: aad-client-secret
        name: grafana-auth
  - name: GF_DATABASE_USER
    valueFrom:
      secretKeyRef:
        key: username
        name: grafana-db
  - name: GF_DATABASE_NAME
    valueFrom:
      secretKeyRef:
        key: database
        name: grafana-db
  - name: GF_DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        key: password
        name: grafana-db
  - name: POD_IP
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: status.podIP
  - name: GF_SECURITY_ADMIN_USER
    valueFrom:
      secretKeyRef:
        key: admin-user
        name: grafana-admin
  - name: GF_SECURITY_ADMIN_PASSWORD
    valueFrom:
      secretKeyRef:
        key: admin-password
        name: grafana-admin
  - name: GF_SMTP_USER
    valueFrom:
      secretKeyRef:
        key: user
        name: grafana-smtp
  - name: GF_SMTP_PASSWORD
    valueFrom:
      secretKeyRef:
        key: password
        name: grafana-smtp
  - name: GF_RENDERING_SERVER_URL
    value: 'http://grafana-image-renderer.monitoring:8081/render'
  - name: GF_RENDERING_CALLBACK_URL
    value: 'http://grafana.monitoring:80/'
  - name: GF_PATHS_DATA
    value: /var/lib/grafana/
  - name: GF_PATHS_LOGS
    value: /var/log/grafana
  - name: GF_PATHS_PLUGINS
    value: /var/lib/grafana/plugins
  - name: GF_PATHS_PROVISIONING
    value: /etc/grafana/provisioning
  - name: GF_AUTH_BASIC_ENABLED
    value: 'false'
  - name: GF_AUTH_DISABLE_LOGIN_FORM
    value: 'true'
  - name: GF_AUTH_PROXY_AUTO_SIGN_UP
    value: 'false'
  - name: GF_AUTH_PROXY_ENABLED
    value: 'true'
  - name: GF_AUTH_PROXY_HEADERS
    value: 'Name:X-WEBAUTH-NAME Email:X-WEBAUTH-EMAIL'
  - name: GF_AUTH_PROXY_HEADER_NAME
    value: X-WEBAUTH-USERNAME
  - name: GF_AUTH_PROXY_HEADER_PROPERTY
    value: username
  - name: GF_AUTH_PROXY_WHITELIST
    value: 'CENSORED'
  - name: GF_DATABASE_HOST
    value: CENSORED:3306
  - name: GF_DATABASE_MAX_IDLE_CONN
    value: '20'
  - name: GF_DATABASE_MAX_OPEN_CONN
    value: '100'
  - name: GF_DATABASE_SSL_MODE
    value: 'false'
  - name: GF_DATABASE_TYPE
    value: mysql
  - name: GF_DATAPROXY_TIMEOUT
    value: '600'
  - name: GF_INSTALL_PLUGINS
    value: >-
      grafana-piechart-panel,raintank-worldping-app,jdbranham-diagram-panel,goshposh-metaqueries-datasource,vonage-status-panel,petrslavotinek-carpetplot-panel,grafana-polystat-panel,snuids-radar-panel,yesoreyeram-boomtheme-panel,digrich-bubblechart-panel,agenty-flowcharting-panel,citilogics-geoloop-panel,michaeldmoore-multistat-panel,corpglory-progresslist-panel,marcuscalidus-svg-panel,fetzerch-sunandmoon-datasource,snuids-trafficlights-panel,http://grafana-streamer-plugin.default.svc.cluster.local:8080/streamer.zip;streamer,marcusolsson-treemap-panel,grafana-sentry-datasource
  - name: GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS
    value: >-
      grafana-piechart-panel,raintank-worldping-app,jdbranham-diagram-panel,goshposh-metaqueries-datasource,vonage-status-panel,petrslavotinek-carpetplot-panel,grafana-polystat-panel,snuids-radar-panel,yesoreyeram-boomtheme-panel,digrich-bubblechart-panel,agenty-flowcharting-panel,citilogics-geoloop-panel,michaeldmoore-multistat-panel,corpglory-progresslist-panel,marcuscalidus-svg-panel,fetzerch-sunandmoon-datasource,snuids-trafficlights-panel,streamer,marcusolsson-treemap-panel
  - name: GF_RENDERING_RENDERER_TOKEN
    value: >-
      map[valueFrom:map[secretKeyRef:map[key:token
      name:grafana-renderer]]]
  - name: GF_SECURITY_ALLOW_EMBEDDING
    value: 'true'
  - name: GF_SECURITY_COOKIE_SECURE
    value: 'true'
  - name: GF_SERVER_ROOT_URL
    value: 'https://grafana.uat/'
  - name: GF_SESSION_COOKIE_NAME
    value: session
  - name: GF_SESSION_COOKIE_SECURE
    value: 'true'
  - name: GF_SESSION_PROVIDER
    value: mysql
  - name: GF_SESSION_PROVIDER_CONFIG
    value: >-
      CENSORED@tcp(CENSORED:3306)/grafana
  - name: GF_SMTP_ENABLED
    value: 'true'
  - name: GF_SMTP_FROM_ADDRESS
    value: noreply@CENSORED
  - name: GF_SMTP_FROM_NAME
    value: Entain Grafana
  - name: GF_SMTP_HOST
    value: 'smtp.CENSORED:587'
  - name: GF_USERS_VIEWERS_CAN_EDIT
    value: 'true'

here is the env values the renderer has:

env:
  - name: HTTP_PORT
    value: '8081'
  - name: AUTH_TOKEN
    value: >-
      map[valueFrom:map[secretKeyRef:map[key:token
      name:grafana-renderer]]]
  - name: HTTP_HOST
    value: 0.0.0.0
  - name: IGNORE_HTTPS_ERRORS
    value: 'true'
  - name: RENDERING_ARGS
    value: >-
      --no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758
  - name: RENDERING_CLUSTERING_MAX_CONCURRENCY
    value: '5'
  - name: RENDERING_CLUSTERING_MODE
    value: context
  - name: RENDERING_MODE
    value: clustered

[AgnesToulet]

I don't have a lot of knowledge around Kubernetes but after a few tests, I would say this is incorrect:

  - name: AUTH_TOKEN
    value: >-
      map[valueFrom:map[secretKeyRef:map[key:token
      name:grafana-renderer]]]

And should be replaced with:

  - name: AUTH_TOKEN
    valueFrom:
      secretKeyRef:
        key: token
        name: grafana-renderer

With the image renderer 3.6.4, you didn't use your Kubernetes secret but the string:

map[valueFrom:map[secretKeyRef:map[key:token
      name:grafana-renderer]]]

In 3.7.0, it broke because I had to chose a separator when allowing multiple tokens and decided to use the whitespace character as it's usually not used in authentication tokens.

[Red-M]

Thanks, I'll let the rest of my team know, I may provide a patch so that this works correctly in the helm chart at least.