Plugin v3.8.0 does not load: "Could not find Chrome (ver. 116.0.5845.96)" from puppeteer

[chennin]

What happened:

Image renderer plugin version 3.8.0 does not load, giving logs:

logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654884325Z level=debug msg="Error: Could not find Chrome (ver. 116.0.5845.96). This can occur if either"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654903285Z level=debug msg=" 1. you did not perform an installation before running the script (e.g. `npm install`) or"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654910725Z level=debug msg=" 2. your cache path is incorrectly configured (which is: /usr/share/grafana/.cache/puppeteer)."
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654914485Z level=debug msg="For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration."
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654917835Z level=debug msg="    at ChromeLauncher.resolveExecutablePath (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/ProductLauncher.js:300:27)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654922826Z level=debug msg="    at ChromeLauncher.executablePath (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/ChromeLauncher.js:180:25)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654926176Z level=debug msg="    at PuppeteerNode.executablePath (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/PuppeteerNode.js:170:31)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654929406Z level=debug msg="    at main (/snapshot/src/build/app.js:24:46)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654932416Z level=debug msg="    at Object.<anonymous> (/snapshot/src/build/app.js:75:1)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654935376Z level=debug msg="    at Module._compile (pkg/prelude/bootstrap.js:1926:22)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654938246Z level=debug msg="    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654941006Z level=debug msg="    at Module.load (node:internal/modules/cjs/loader:981:32)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654943776Z level=debug msg="    at Function.Module._load (node:internal/modules/cjs/loader:822:12)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.654948296Z level=debug msg="    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)"
logger=plugin.grafana-image-renderer t=2023-08-25T21:15:31.660405449Z level=error msg="plugin process exited" path=/var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64 pid=27 error="exit status 1"

And, the usual:

logger=modules t=2023-08-25T21:15:31.674892305Z level=info msg="module stopped" module=http-server
Error: ✗ *rendering.RenderingService run error: Unrecognized remote plugin message:
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake

Additional notes about plugin:
  Path: /var/lib/grafana-plugins/grafana-image-renderer/plugin_start_linux_amd64
  Mode: -rwxr-xr-x
  Owner: 472 [grafana] (current: 472 [grafana])
  Group: 0 [root] (current: 0 [root])
  ELF architecture: EM_X86_64 (current architecture: amd64)

log.txt

How to reproduce it (as minimally and precisely as possible):

Build the official(?) Grafana-with-Image-Renderer Dockerfile, then run the docker image.

mkdir -p /tmp/work && cd /tmp/work && \
wget https://raw.githubusercontent.com/grafana/grafana/main/packaging/docker/custom/Dockerfile && \
docker build --build-arg GF_INSTALL_IMAGE_RENDERER_PLUGIN=true -t grafana-plugin-test . < ./Dockerfile && \
docker run --rm -e GF_LOG_LEVEL="debug" grafana-plugin-test | tee log.txt

Anything else we need to know?:

3.7.2 works.

Directly using the official images grafana/grafana or grafana/grafana:latest-ubuntu, and passing -e GF_INSTALL_PLUGINS="grafana-image-renderer", also does not work. 3.8.0 still does not load; 3.7.2 does load (but would not actually work on those images due to missing dependencies).

Setting GF_PLUGIN_RENDERING_CHROME_BIN does not fix it.

If I use strace and try to run 3.8.0 directly, I can see an error on a path that doesn't exist just before the error message is printed. No folders named puppeteer exist in the container. 3.7.2 more correctly tells me not to run a plugin directly.

148   pread64(17, "syntax = \"proto3\";\n\npackage grpc.health.v1;\n\nmessage HealthCheckRequest {\n  string service = 1;\n}\n\nmessage HealthCheckResponse {\n  enum ServingStatus {\n    UNKNOWN = 0;\n    SERVING = 1;\n    NOT_SERVING = 2;\n  }\n  ServingStatus status = 1;\n}\n\nservice Health"..., 323, 42591608) = 323
148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_EXEC) = 0
148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_EXEC) = 0

148   access("/home/grafana/.cache/puppeteer/chrome/linux-116.0.5845.96/chrome-linux64/chrome", F_OK) = -1 ENOENT (No such file or directory)

148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_EXEC) = 0
148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
148   mprotect(0x7f73ce903000, 245760, PROT_READ|PROT_EXEC) = 0
148   write(18, "Error: Could not find Chrome (ver. 116.0.5845.96). This can occur if either\n 1. you did not perform an installation before running the script (e.g. `npm install`) or\n 2. your cache path is incorrectly configured (which is: /home/grafana/.cache/puppeteer).\n"..., 1238) = 1238
148   futex(0x342d794, FUTEX_WAKE_PRIVATE, 2147483647) = 2

Environment:

• Grafana Image Renderer version: 3.8.0
• Grafana version: 10.1.0 and 10.0.3
• Installed plugin or remote renderer service: plugin
• OS Grafana Image Renderer is installed on: Alpine/Ubuntu/Docker
• Host OS used to build Docker image: Ubuntu 22.04, with package docker.io 20.10.25-0ubuntu1~22.04.2 (which is after the recent chown issue)

[Hipska]

I have the same issue (saying it is compiled for another architecture) and it keeps restarting Grafana because of this. Reverting to 3.7.x resolves the issue.

I'm not running the plugin or Grafana in Docker, but plain on the system (RHEL 7).

Running the reported faulty plugin_start_linux_amd64 gives the following:

Error: Could not find Chrome (ver. 116.0.5845.96). This can occur if either
 1. you did not perform an installation before running the script (e.g. `npm install`) or
 2. your cache path is incorrectly configured (which is: /home/grafana/.cache/puppeteer).
For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration.
    at ChromeLauncher.resolveExecutablePath (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/ProductLauncher.js:300:27)
    at ChromeLauncher.executablePath (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/ChromeLauncher.js:180:25)
    at PuppeteerNode.executablePath (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/PuppeteerNode.js:170:31)
    at main (/snapshot/src/build/app.js:24:46)
    at Object.<anonymous> (/snapshot/src/build/app.js:75:1)
    at Module._compile (pkg/prelude/bootstrap.js:1926:22)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)

# ldd /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-118.0.5968.0/chrome-linux64/chrome
/var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-118.0.5968.0/chrome-linux64/chrome: /lib64/libnss3.so: version `NSS_3.31' not found (required by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-118.0.5968.0/chrome-linux64/chrome)
/var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-118.0.5968.0/chrome-linux64/chrome: /lib64/libdbus-1.so.3: no version information available (required by /var/lib/grafana/plugins/grafana-image-renderer/chrome/linux-118.0.5968.0/chrome-linux64/chrome)
	linux-vdso.so.1 =>  (0x00007ffc539db000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007ff947598000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff94737c000)
	libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007ff94712b000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007ff946e15000)
	libnss3.so => /lib64/libnss3.so (0x00007ff946aeb000)
	libnssutil3.so => /lib64/libnssutil3.so (0x00007ff9468bd000)
	libsmime3.so => /lib64/libsmime3.so (0x00007ff946696000)
	libnspr4.so => /lib64/libnspr4.so (0x00007ff946458000)
	libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x00007ff946231000)
	libatk-bridge-2.0.so.0 => /lib64/libatk-bridge-2.0.so.0 (0x00007ff946002000)
	libcups.so.2 => /lib64/libcups.so.2 (0x00007ff945d99000)
	libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007ff9459f9000)
	libdrm.so.2 => /lib64/libdrm.so.2 (0x00007ff9457ea000)
	libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007ff9455a2000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007ff945377000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007ff94514f000)
	libxkbcommon.so.0 => /lib64/libxkbcommon.so.0 (0x00007ff944f0f000)
	libatspi.so.0 => /lib64/libatspi.so.0 (0x00007ff944cde000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007ff9449a0000)
	libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007ff94479d000)
	libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007ff944599000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007ff944387000)
	libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007ff944181000)
	libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007ff943f75000)
	libgbm.so.1 => /lib64/libgbm.so.1 (0x00007ff943d69000)
	libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x00007ff943b1e000)
	libcairo.so.2 => /lib64/libcairo.so.2 (0x00007ff9437f5000)
	libasound.so.2 => /lib64/libasound.so.2 (0x00007ff9434f7000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ff9431f5000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff942fde000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff942c1b000)
	/lib64/ld-linux-x86-64.so.2 (0x000055e78d315000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007ff9429b9000)
	libffi.so.6 => /lib64/libffi.so.6 (0x00007ff9427b0000)
	libplc4.so => /lib64/libplc4.so (0x00007ff9425ab000)
	libplds4.so => /lib64/libplds4.so (0x00007ff9423a7000)
	librt.so.1 => /lib64/librt.so.1 (0x00007ff94219e000)
	libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007ff941f9a000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007ff941d4c000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007ff941a64000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007ff941831000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007ff94162c000)
	libssl.so.10 => /lib64/libssl.so.10 (0x00007ff9413ba000)
	libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007ff940f57000)
	libavahi-common.so.3 => /lib64/libavahi-common.so.3 (0x00007ff940d49000)
	libavahi-client.so.3 => /lib64/libavahi-client.so.3 (0x00007ff940b38000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007ff940901000)
	libz.so.1 => /lib64/libz.so.1 (0x00007ff9406ea000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007ff9404c3000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007ff9402a9000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007ff940067000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007ff93fe63000)
	libXrender.so.1 => /lib64/libXrender.so.1 (0x00007ff93fc57000)
	libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007ff93fa55000)
	libthai.so.0 => /lib64/libthai.so.0 (0x00007ff93f849000)
	libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007ff93f59f000)
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007ff93f362000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007ff93f0bc000)
	libEGL.so.1 => /lib64/libEGL.so.1 (0x00007ff93ee8e000)
	libpng15.so.15 => /lib64/libpng15.so.15 (0x00007ff93ec63000)
	libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007ff93ea5f000)
	libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007ff93e850000)
	libGL.so.1 => /lib64/libGL.so.1 (0x00007ff93e5de000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007ff93e3cf000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007ff93e1cb000)
	libfreebl3.so => /lib64/libfreebl3.so (0x00007ff93dfc7000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007ff93dd88000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007ff93db83000)
	libX11-xcb.so.1 => /lib64/libX11-xcb.so.1 (0x00007ff93d980000)
	libxcb-dri2.so.0 => /lib64/libxcb-dri2.so.0 (0x00007ff93d77b000)
	libxcb-xfixes.so.0 => /lib64/libxcb-xfixes.so.0 (0x00007ff93d573000)
	libxcb-dri3.so.0 => /lib64/libxcb-dri3.so.0 (0x00007ff93d36f000)
	libxcb-present.so.0 => /lib64/libxcb-present.so.0 (0x00007ff93d16c000)
	libxcb-sync.so.1 => /lib64/libxcb-sync.so.1 (0x00007ff93cf65000)
	libxshmfence.so.1 => /lib64/libxshmfence.so.1 (0x00007ff93cd62000)
	libglapi.so.0 => /lib64/libglapi.so.0 (0x00007ff93cb33000)
	libxcb-glx.so.0 => /lib64/libxcb-glx.so.0 (0x00007ff93c917000)
	libXxf86vm.so.1 => /lib64/libXxf86vm.so.1 (0x00007ff93c711000)

[olafrauch]

Exactly the same error here. Running it native on AWS EC2 centos 8 stream AMI based.
Downgrade to 3.7.2 was the quick workaround.

We also tested to manually install the documented yum dependencies according to https://grafana.com/docs/grafana/latest/setup-grafana/image-rendering/troubleshooting/
No difference.

[AgnesToulet]

Hello! Thank you for reporting this, we are investigating this issue and we'll keep you updated.