grafana_image_renderer fails with http 401

[emtie]

What happened:
I have configured icingaweb2 to show grafana dashboards inplace to have a nice overview incinga. But when I open a page which should show grafana graphs I get an errormessage which says: "Cannot fetch graph with curl: Operation timed out after 5001 milliseconds with 0 bytes received. When I click on the link the grafana webpage opens and shows the expected graph just fine!

The grafana environment is running as docker containers in a docker swarm. the definition is like that:

 version: '3.7'
 services:
   grafana:
     image: grafana/grafana:latest
     ports:
       - "3000:3000"
     volumes:
       - /gluster/docker/volume/grafana/etc:/etc/grafana
       - /gluster/docker/volume/grafana/lib:/var/lib/grafana
     environment:
       - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource
       - GF_RENDERING_SERVER_URL=http://grafana_renderer:8081/render
       - GF_RENDERING_CALLBACK_URL=https://grafana_grafana:3000/
       - GF_LOG_FILTERS=rendering:debug
   renderer:
     image: grafana/grafana-image-renderer
     ports:
       - "8081:8081"
     environment:
       - IGNORE_HTTPS_ERRORS=true
       - BROWSER_TZ=Europe/Berlin
 networks:
   default:
     external:
       name: my_network

The containerlogs show the following:

grafana_renderer:

{"level":"info","message":"HTTP Server started, listening at http://localhost:8081"},
{"msg":"WebSocket connection to 'wss://grafana_grafana:3000/api/live/ws' failed: HTTP Authentication failed; no valid credentials available","url":"https://grafana_grafana:3000/public/build/vendors~app.7f5a241f55df4a1c66ae.js","line":1,"level":"error","message":"Browser console error"},
{"msg":"Failed to load resource: the server responded with a status of 401 ()","url":"https://grafana_grafana:3000/api/dashboards/uid/ZK27sqXmz","level":"error","message":"Browser console error"},
{"msg":"Failed to load resource: the server responded with a status of 401 ()","url":"https://grafana_grafana:3000/api/login/ping","level":"error","message":"Browser console error"},
{"url":"https://grafana_grafana:3000/public/build/grafanaPlugin.7f5a241f55df4a1c66ae.js","method":"GET","failure":"net::ERR_ABORTED","level":"error","message":"Browser request failed"},
{"msg":"Failed to load resource: the server responded with a status of 401 ()","url":"https://grafana_grafana:3000/d-solo/ZK27sqXmz/icinga2-default?var-hostname=mon01.mydomain&var-service=hostalive&var-command=hostalive&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now&render=1","level":"error","message":"Browser console error"},
{"url":"/render?deviceScaleFactor=1.000000&domain=grafana_grafana&encoding=&height=280&renderKey=4w1c1YCO6uDfxF30gZrPEWyBdL8QJN9v&timeout=60&timezone=&url=https%3A%2F%2Fgrafana_grafana%3A3000%2Fd-solo%2FZK27sqXmz%2Ficinga2-default%3Fvar-hostname%3Dmon01.mydomain%26var-service%3Dhostalive%26var-command%3Dhostalive%26panelId%3D1%26orgId%3D1%26width%3D640%26height%3D280%26theme%3Dlight%26from%3Dnow-6h%26to%3Dnow%26render%3D1&width=640","stack":"TimeoutError: waiting for function failed: timeout 60000ms exceeded\n    at new WaitTask (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:388:34)\n    at DOMWorld.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:303:16)\n    at Frame.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/FrameManager.js:402:32)\n    at Page.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/Page.js:805:33)\n    at Browser.<anonymous> (/usr/src/app/build/browser/browser.js:195:24)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/app/build/browser/browser.js:5:58)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)","level":"error","message":"Request failed"},
{"msg":"WebSocket connection to 'wss://grafana_grafana:3000/api/live/ws' failed: HTTP Authentication failed; no valid credentials available","url":"https://grafana_grafana:3000/public/build/vendors~app.7f5a241f55df4a1c66ae.js","line":1,"level":"error","message":"Browser console error"},
{"msg":"Failed to load resource: the server responded with a status of 401 ()","url":"https://grafana_grafana:3000/api/dashboards/uid/ZK27sqXmz","level":"error","message":"Browser console error"},
{"msg":"Failed to load resource: the server responded with a status of 401 ()","url":"https://grafana_grafana:3000/api/login/ping","level":"error","message":"Browser console error"},
{"url":"https://grafana_grafana:3000/public/build/grafanaPlugin.7f5a241f55df4a1c66ae.js","method":"GET","failure":"net::ERR_ABORTED","level":"error","message":"Browser request failed"},
{"msg":"Failed to load resource: the server responded with a status of 401 ()","url":"https://grafana_grafana:3000/d-solo/ZK27sqXmz/icinga2-default?var-hostname=mon01.mydomain&var-service=hostalive&var-command=hostalive&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now&render=1","level":"error","message":"Browser console error"},
{"url":"/render?deviceScaleFactor=1.000000&domain=grafana_grafana&encoding=&height=280&renderKey=xBdaRNxyW6AWgoLYa2ELDErIwfTcPAGG&timeout=60&timezone=&url=https%3A%2F%2Fgrafana_grafana%3A3000%2Fd-solo%2FZK27sqXmz%2Ficinga2-default%3Fvar-hostname%3Dmon01.amydomain%26var-service%3Dhostalive%26var-command%3Dhostalive%26panelId%3D1%26orgId%3D1%26width%3D640%26height%3D280%26theme%3Dlight%26from%3Dnow-6h%26to%3Dnow%26render%3D1&width=640","stack":"TimeoutError: waiting for function failed: timeout 60000ms exceeded\n    at new WaitTask (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:388:34)\n    at DOMWorld.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:303:16)\n    at Frame.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/FrameManager.js:402:32)\n    at Page.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/Page.js:805:33)\n    at Browser.<anonymous> (/usr/src/app/build/browser/browser.js:195:24)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/app/build/browser/browser.js:5:58)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)","level":"error","message":"Request failed"},,

grafana

t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console",
t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana",
t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana",
t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins",
t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning",
t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_LOG_FILTERS=rendering:debug",
t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_RENDERING_SERVER_URL=http://grafana_renderer:8081/render",
t=2021-06-17T09:34:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_RENDERING_CALLBACK_URL=https://grafana_grafana:3000/",
t=2021-06-17T09:34:37+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana,
t=2021-06-17T09:34:37+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana,
t=2021-06-17T09:34:37+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana,
t=2021-06-17T09:34:37+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins,
t=2021-06-17T09:34:37+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning,
t=2021-06-17T09:34:37+0000 lvl=info msg="App mode production" logger=settings,
t=2021-06-17T09:34:37+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3,
t=2021-06-17T09:34:37+0000 lvl=info msg="Starting DB migrations" logger=migrator,
t=2021-06-17T09:34:37+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=329 duration=1.906371ms,
t=2021-06-17T09:34:37+0000 lvl=info msg="Starting plugin search" logger=plugins,
t=2021-06-17T09:34:37+0000 lvl=info msg="Registering plugin" logger=plugins id=grafana-plugin-admin-app,
t=2021-06-17T09:34:37+0000 lvl=info msg="Registering plugin" logger=plugins id=input,
t=2021-06-17T09:34:39+0000 lvl=info msg="Registering plugin" logger=plugins id=grafana-clock-panel,
t=2021-06-17T09:34:39+0000 lvl=info msg="Registering plugin" logger=plugins id=grafana-simple-json-datasource,
t=2021-06-17T09:34:39+0000 lvl=warn msg="Some plugin scanning errors were found" logger=plugins errors="plugin 'grafana-image-renderer' is unsigned",
t=2021-06-17T09:34:39+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http,
t=2021-06-17T09:34:39+0000 lvl=dbug msg="calling remote rendering service" logger=rendering renderer=http url=http://grafana_renderer:8081/render/version,
t=2021-06-17T09:34:39+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=h2 subUrl= socket=,
t=2021-06-17T09:34:40+0000 lvl=info msg="Backend rendering via external http server" logger=rendering renderer=http version=3.0.1,
t=2021-06-17T10:19:46+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/dashboards/uid/ZK27sqXmz status=401 remote_addr=10.10.10.16 time_ms=15 size=32 referer="https://grafana_grafana:3000/d-solo/ZK27sqXmz/icinga2-default?var-hostname=mon01.mydomain&var-service=hostalive&var-command=hostalive&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now&render=1",
t=2021-06-17T10:19:45+0000 lvl=eror msg="Failed to get render key from cache" logger=rendering renderer=http error="cache item not found",
t=2021-06-17T10:21:38+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname="Matthias Tripp" method=GET path=/api/live/ws status=400 remote_addr=10.0.0.3 time_ms=49 size=12 referer=,
t=2021-06-17T10:19:45+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/live/ws status=401 remote_addr=10.10.10.16 time_ms=23 size=32 referer=,
t=2021-06-17T10:19:46+0000 lvl=eror msg="Failed to get render key from cache" logger=rendering renderer=http error="cache item not found",
t=2021-06-17T10:19:46+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/login/ping status=401 remote_addr=10.10.10.16 time_ms=13 size=32 referer="https://grafana_grafana:3000/d-solo/ZK27sqXmz/icinga2-default?var-hostname=mon01.mydomain&var-service=hostalive&var-command=hostalive&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now&render=1",
t=2021-06-17T10:19:46+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/d-solo/ZK27sqXmz/icinga2-default status=401 remote_addr=10.10.46.16 time_ms=18 size=32 referer="https://grafana_grafana:3000/d-solo/ZK27sqXmz/icinga2-default?var-hostname=mon01.mydomain&var-service=hostalive&var-command=hostalive&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now&render=1",
t=2021-06-17T10:20:38+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname="hans" method=GET path=/api/live/ws status=400 remote_addr=10.0.0.3 time_ms=54 size=12 referer=

grafana.ini looks like this:

[server]
protocol = h2
;http_addr =
;http_port = 3000
domain = localhost
root_url = https://grafana.mydomain
...
[auth.anonymous]
enabled = false
org_name = Main Org.
org_role = Viewer
...
[auth.basic]
enabled = true
#################################### Auth LDAP ##########################
[auth.ldap]
enabled = true
;config_file = /etc/grafana/ldap.toml
allow_sign_up = true
config_file = /etc/grafana/ldap.toml
...
[log]
;mode = console file
;level = info
;filters =
filters = rendering:debug
...
[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true
rendering_verbose_logging = true
rendering_args = --no-sandbox,--no-proxy-server
...
[external_image_storage]
provider = local
...

What you expected to happen:
I would love to see the beautyful grafana graphs to be displayed in icingaweb2

How to reproduce it (as minimally and precisely as possible):
see description above

Anything else we need to know?:
don't no. you are welcome to give me some advice which further information I can provide to help solve the issue...

Environment:

• Grafana Image Renderer version: 3.0.1
• Grafana version: Grafana v8.0.2 (5c5d45e19f)
• Installed plugin or remote renderer service: remote renderer service as docker container (see above)
• OS Grafana Image Renderer is installed on: Ubuntu 20.04
• User OS & Browser: Windows 10 Firefox 89
• Others: docker and grafana is behind a nginx reverse proxy

Thanks in advance!

[AgnesToulet]

I think you need to find a way to increase the timeout on icingaweb2 because 5s can be too short for the rendering requests depending on the requests you are doing. We are looking into ways to improve performance but it's a different issue (see #124).