Detect changes to datasources when using provisioning

[bgagnon]

What Grafana version are you using?

Grafana v5.0.3 (commit: 91162f3)

When changing datasource files, grafana should reload them automatically (without restart).
Grafana requires a restart to pick up changes to provisioned datasources, unlike dashboards.

The user case is a "git ops" setup where Grafana is exclusively configured by Git and CI jobs. This works well for dashboards, but not datasources.

[chrisduong]

Hi, while waiting for this feature. Do we have a command (API call) to reload its datasource manually.

[marefr]

@chrisduong No, the only http api available is the update datasource: http://docs.grafana.org/http_api/data_source/#update-an-existing-data-source

[bergquist]

The reason we decided that datasources should not be reloaded is that we consider it configuration rather than content (dashboards) and I don't expect Grafana to reload its configuration when its configuration files changes.

I do understand the need for controlling datasources completly from configuration.. But right now I think a better way would be to add an API endpoint for reloading the configuration.

[dtshepherd]

I ran into this issue today. However, my problem was during initial deploy of the grafana chart + configmap. There is a race condition where the sidecar may not load the new `datasources.yaml file into the provisioning folder before the main grafana pod loads. This means we need to bounce grafana no matter what during initial provisioning of a stack.

It seems if grafana itself isn't planning to continuous update datasources from the provisioning directory, the kubernetes deployment should help synchronize the install. What makes things worse, you can't guarantee when the grafana pod restarts that the sidecar container will start first.

[swtch1]

@dtshepherd I'm guessing from your comment that you're using the stable/grafana helm chart. I was having the same issue so I forked their chart and have the datasource processing happening in an init container. Also doing secret injection there.

[dtshepherd]

@swtch1 Yep, I've been trying to avoid forking the stable charts if at all possible. I may go down that route. Not sure what's worse, forking the chart, or creating a custom grafana docker image.

[lswith]

I'd love if someone just made a PR against the chart. It would help immensely. It seems this bug could take a while.

[dtshepherd]

What do you think would be the fix against the chart? Just support an init container? Or volume mount for provisioning?

[lswith]

yeah I think initContainer is probably the way to go considering its what @swtch1 is using.

[swtch1]

I've got some changes I can submit. No guarantee they'll take it since it changes the methodology a bit but I'll link the PR here in case anyone wants to fork it. Give me a week to get caught up at work.

[swtch1]

I have submitted the PR to the helm charts repository. No idea if they will merge it in but forking mine should get this initContainer behavior you're looking for and avoid the race condition that causes this to miss some of all of the datasources from configmaps.

@lswith @dtshepherd

[michaelbannister]

Should this issue (as originally raised) be closed, now that #16579 is merged? (Thanks very much for that, @aocenas!)

[marefr]

@michaelbannister it depends of the issue author is satisfied with the support of manually reloading via HTTP API endpoint.

[aocenas]

@marefr I remember talking with @bergquist about eventually implementing this feature, and the manual reloading API being a precursor to this but not sure if it got planned (do not see it in the backend project) or was just in some backlog for unscheduled future. In any case even if author can manage with the manual reload this probably should be kept open to at least get feedback whether more people want this feature.

[michaelbannister]

In which case, I'll say that having provisioning configs auto-reload would be really good for my team - we sync dashboards from git to a GCS bucket and have a process sitting next to Grafana that periodically syncs from GCS to the filesystem. Having discovered the reload API, we're using that, but if Grafana automatically picked up the changes that'd be simpler.

Or, even better: provision dashboards from remote storage like GCS/S3/Git
Edit: I see that's been rejected at #11879

[marefr]

@aocenas yeah lets keep it open so that people interested can upvote it.

[vijayakumarbathini]

+1

we are looking for this feature

[sancyx]

+1

also looking for this feature

[Morriz]

+1

[anarcher]

+1

[Hmerac]

+1

[njbarber]

+1

I have a custom solution for the Helm chart issue with makes changes to the kiwigrid/k8s-sidecar and makes use of the reload and delete API when ConfigMaps are added and deleted, respectively. The changes are quite specific, so there probably isn't a set of parameters that could be added to the image to make it generally useful.

Would this be a potential solution? To have the chart use a custom image which does the former?

[sancyx]

Hi @njbarber, I've also prepared a similar change to grafana chart helm/charts@master...sancyx:reaload_datasources, however this only works for the reload, it's not able to delete the datasource because it uses the stock k8s-sidecar. I I think your solution is slightly better, do you have this grafana specific k8s-sidecar in a PR? I think if you can make that change / image public, you should make a PR to grafana chart to use that side-car. I don't see a better solution right now.

[StevenCyb]

Currently I am working on a mechanism to distribute configurations to some Grafana instances. Now I am also facing this challenge. In my case reloading the data sources via API is tedious, because I cannot get the admin user/pw without a lot of overhead and manual administration.

Maybe we can add an command to reload via CLI?

So I'm also interested.

[LudwigHoff]

+1

For me it is an old fashioned "I don't know where prometheus is" until my single-instance prometheus registers itself with consul.

Hoping to use consul-template to update ip:port using an in-container sidecar (containerpilot).

As a previous poster notes, using the api is tedious due to the security.

As it stands I now need to front prometheus behind a load balancer - probably an nginx instance with consul-template, cos nginx does support CLI config reload.

[qaiserali]

+1
I am also facing the same issue. I tried to import dashboard via configmap but it doesn't show on grafana. #27748

[aniketd]

Case:

• using prometheus-operator
• grafana helm chart: v5.3.0
• grafana: v7.0.3

Observation:

• dashboards are updated automatically upon changes to the ConfigMaps, but
• datasources do not.

Having datasources also update automatically like the dashboards do, would be very helpful.

[gjed]

+1

[Rohlik]

You can use API call to reload Grafana provisioned stuff - https://grafana.com/docs/grafana/latest/http_api/admin/#reload-provisioning-configurations

[vpedosyuk]

The POST /api/admin/provisioning/datasources/reload approach doesn't work for those who use SSO with disabled basic auth. Datasources auto-reload would make perfect sense in this scenario.

[adimitris]

+1

[Breee]

Makes absolutely no sense to me that this does not exist.
This logic already exists for Dashboards: https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards
The grafana team or the community should implement a similar mechanism for datasources and plugins.

[ospiegel91]

please understand the importance of this... still an issue in 2023

[danilocmd]

+1

[haqa]

+1