Grafana Rendering Failed to send request to remote rendering service

[jiragrid]

What happened:
Grafana Rendering Failed to send request to remote rendering service

In pod grafana

t=2021-07-07T06:56:39+0000 lvl=eror msg="Failed to send request to remote rendering service" logger=rendering renderer=http error="Get "http://grafana-image-renderer.grafana:8081/render?deviceScaleFactor=1.000000&domain=grafana.grafana&encoding=&height=500&renderKey=ZTdINNnPBbdsvE8f5UHCacizxmig93bf&timeout=15&timezone=&url=http%3A%2F%2Fgrafana.grafana%3A3000%2Fd-solo%2FF1C_WQg7k%2Fdit%3ForgId%3D1%26panelId%3D4%26render%3D1&width=1000\": context deadline exceeded"
t=2021-07-07T06:56:39+0000 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=15 error="failed to send request to remote rendering service: Get "http://grafana-image-renderer.grafana:8081/render?deviceScaleFactor=1.000000&domain=grafana.grafana&encoding=&height=500&renderKey=ZTdINNnPBbdsvE8f5UHCacizxmig93bf&timeout=15&timezone=&url=http%3A%2F%2Fgrafana.grafana%3A3000%2Fd-solo%2FF1C_WQg7k%2Fdit%3ForgId%3D1%26panelId%3D4%26render%3D1&width=1000\": context deadline exceeded"

In pod grafana-image-renderer

pod grafana

Ports: 443/TCP, 3000/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Wed, 07 Jul 2021 13:09:25 +0700
Ready: True
Restart Count: 0
Limits:
cpu: 200m
memory: 512Mi
Requests:
cpu: 200m
memory: 512Mi
Liveness: http-get http://:3000/api/health delay=60s timeout=30s period=10s #success=1 #failure=10
Readiness: http-get http://:3000/api/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
GF_SECURITY_ADMIN_USER: <set to the key 'admin-user' in secret 'grafana'> Optional: false
GF_SECURITY_ADMIN_PASSWORD: <set to the key 'admin-password' in secret 'grafana'> Optional: false
GF_INSTALL_PLUGINS: <set to the key 'plugins' of config map 'grafana'> Optional: false
GF_RENDERING_SERVER_URL: http://grafana-image-renderer.grafana:8081/render
GF_RENDERING_CALLBACK_URL: http://grafana.grafana:3000/
GF_LOG_FILTERS: rendering:debug
GF_AUTH_ANONYMOUS_ENABLED: true
Mounts:
/etc/grafana/grafana.ini from config (rw,path="grafana.ini")
/etc/grafana/provisioning/datasources/datasources.yaml from config (rw,path="datasources.yaml")
/var/lib/grafana from storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from grafana-token-bwjdd (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: grafana
Optional: false
storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: grafana-2
ReadOnly: false
grafana-token-bwjdd:
Type: Secret (a volume populated by a Secret)
SecretName: grafana-token-bwjdd
Optional: false

pod grafana-image-renderer

Port: 8081/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 07 Jul 2021 12:55:00 +0700
Ready: True
Restart Count: 0
Limits:
cpu: 200m
memory: 512Mi
Requests:
cpu: 200m
memory: 512Mi
Environment:
HTTP_PORT: 8081
HTTP_HOST: 0.0.0.0
ENABLE_METRICS: true
Mounts:
/tmp from image-renderer-tmpfs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rxdc9 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
image-renderer-tmpfs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
default-token-rxdc9:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rxdc9
Optional: false

Environment:

• Grafana Image Renderer version: 3.0.1
• Grafana version: 8.0.4
• OS Grafana Image Renderer is installed on: run on pod in kubernetes

[edvegas]

I faced timeout when CPU limit is too low, when I increased it to 2000m everything is rendered much faster

[jiragrid]

@edvegas I increased cpu to 2000m. also I change GF_RENDERING_CALLBACK_URL to ingress .but throw new error 400 Browser console error ( failed: Error during WebSocket handshake: Unexpected response code: 400")

[AgnesToulet]

Hello @jiragrid, if you still have this issue with latest versions of Grafana and the image renderer, could you please share your updated configuration file and the full log stacks for a rendering request please?

[jiragrid]

Hello @edvegas, thank you for your response. I've deployed Grafana version 9.x, and I haven't encountered any issues related to the image renderer. at this version thanks.