Avoid dependency on external circumstances on restart

[bukovjanmic]

Current behavior of Grafana operator is, when there are declared custom plugins in e.g. Grafana dashboad, to download those plugins during Grafana startup from grafana.com website.

However, there is a catch. If the grafana website is not available (e.g. there is no internet connection), grafana deployment fails to start and ends up in a crash loop.

I suggest the following enahncements:

• if there is a failure downloading a plugin, Grafana should issue a warning into logs, but should proceed and start (i.e. ignore the download error)
• to avoid unexpected failures on startup, it would be great to have a possibility to declare a PVC/mount for /var/lib/grafana/plugins (separate from /var/lib/grafana mount), so that we could keep Grafana ephemeral, but allow to cache already downloaded plugins. Also, the plugins should not be redownloaded if already present, if possible.
• another possibility would be to be able to declare a mirror (do not go directly to Grafana.com) and download plugins from a local mirror

This would not only make Grafana start faster, but also make it less dependent on external circumstances (e.g. Internet connection availability) when starting the deployment.

[weisdd]

Well, the operator does not install the plugins itself, it only generates an environment variable with a list of plugins that Grafana reads upon start.
So, I'd say you're rather looking for changes in Grafana itself, so it would make sense to create an issue in its repository.

The current implementation of CR already allows you to have full access to any workload properties, so there are plenty of things you could do:

• if your list of plugins is more or less static, you can prepackage your plugins either directly into Grafana image or into another image the plugins would be copied from through an init container with a shared volume. You can also download them from somewhere else;
• I think it should be possible to write a reverse-proxy (or maybe reuse something existing) to conditionally forward requests to a mirror and make Grafana use that proxy.

[bukovjanmic]

I think there used to be Grafana init plugin in v4 operator - https://github.com/grafana-operator/grafana_plugins_init, but this seems to have changed.

So it seems that the upstream Grafana image is responsible to download plugins according to GF_INSTALL_PLUGINS env variable?

It seems Grafana v5 CR should allow me to add my own init container, so maybe this is the way to go, providing that Grafana will respect already downloaded plugins and will not try to redownload them (via the GF_INSTALL_PLUGIN variable) again even if init container has already done so.

I will test this out next week.

[weisdd]

That is correct, v4 had a different implementation. With changes that came with v5, we just wanted to let Grafana do the things it's already capable of, so we don't have to overcomplicate our code base :)

[bukovjanmic]

It seems it will be necessary to open an issue for Grafana image directly. Studying the Docker code, it seems that:

• although grafana cli supports a different repo (mirror) than grafana.com, there is no way to specify this via operator - the script that is interpreting GF_INSTALL_PLUGINS, which is created and passed by the operator, have no mirror support (the --repo switch). Maybe if the operator would put a mirror URL for each plugin, although this is arguably not very nice. There is no way to specify a mirror repo via env variable, it seems
• the same goes for custom CA, there is no way to inject custom CA for HTTPS communication, nor invoke the --insecure switch

I will try to create a custom init image and put the plugin into plugins directory for the mount, and see if Grafana will find the plugin and skip the download. Otherwise the options for download are quite limited.

[pb82]

@bukovjanmic I good solution could be to bundle the plugins in a custom Grafana image and deploy that using the operator. You can override the base image.

[bukovjanmic]

Do you know a place (path) where these plugins should be put?

Normally, we have a mount grafana-data on the /var/lib/grafana mount path, which means that /var/lib/grafana/plugins, where plugins are downloaded, cannot be used by image, as it will be overridden by the mount.

[pb82]

@bukovjanmic you can use a setting to override the path where Grafana looks for plugins: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#plugins

[bukovjanmic]

Good point, we will try this out. Hopefully, if the plugins are already present, Grafana will not try to redownload the plugins (as GF_INSTALL_PLUGINS will still be passed by the operator).

[github-actions]

This issue hasn't been updated for a while, marking as stale, please respond within the next 7 days to remove this label