Data Sources: Provisioning config updated, but changes not reflected in Grafana

[SuperMatt]

What happened?

In Grafana 10.2.x, I have made a change to a datasource, namely a http header value, but also the editable flag, but these changes have not been reflected back in Grafana, they are using the old configuration. When I look at the mysql database for the datasource after the provisioning file has been updated and Grafana restarted, the data in the database is completely unchanged. I used the editable flag as a good canary for this because it's unencrypted and therefore would be easy to see the read_only column in the data_source table switch from 1 to 0, but it has not.

What did you expect to happen?

I expected the changes to the provisioning file to be picked up during a restart of Grafana (or even after 10 minutes of Grafana running), and the changes be visible (or at least the effects of the changes visible WRT changing header values) in the UI.

Did this work before?

10.0.x

How do we reproduce it?

1. Create a datasource provisioning file which looks similar to this:

        {
            "access": "proxy",
            "editable": false,
            "jsonData": {
                "httpHeaderName1": "X-Scope-OrgID",
                "incrementalQuerying": true,
                "prometheusType": "Cortex",
                "prometheusVersion": "1.14.0",
                "timeInterval": "30s"
            },
            "name": "Foobar",
            "orgId": 1,
            "secureJsonData": {
                "httpHeaderValue1": "foobar",
            },
            "type": "prometheus",
            "url": "http://prometheus:9090/",
            "version": 1
        },

2. Start grafana
3. Update the provisioning file like so:

        {
            "access": "proxy",
            "editable": true,
            "jsonData": {
                "httpHeaderName1": "X-Scope-OrgID",
                "incrementalQuerying": true,
                "prometheusType": "Cortex",
                "prometheusVersion": "1.14.0",
                "timeInterval": "30s"
            },
            "name": "Foobar",
            "orgId": 1,
            "secureJsonData": {
                "httpHeaderValue1": "foobar",
            },
            "type": "prometheus",
            "url": "http://prometheus:9090/",
            "version": 1
        },

4. Restart Grafana
5. Check to see that there are no changes

Is the bug inside a dashboard panel?

n/a

Environment (with versions)?

Grafana: 10.2.2 and 10.2.3
OS: Linux / Kubernetes
Browser: Edge

Grafana platform?

Kubernetes

Datasource(s)?

Prometheus

[SuperMatt]

I have a suspicion that for the datasource editing to work a uid needs to be provided to the provisioning config file, but I would love your thoughts.

[SuperMatt]

Now that I have returned from my holidays, I have found that the only way to make this work is to delete the original provisioned datasource and reprovision it. Editing the datasource to add the UID and restarting Grafana without deleting the datasource did not work.

[tonypowa]

hi @SuperMatt

have you tried to increase the version number?

Grafana updates only data sources with the same or lower version number than specified in the config. This prevents old configurations from overwriting newer ones if you have different versions of the datasource.yaml file that don’t define version numbers

https://grafana.com/docs/grafana/latest/administration/provisioning/#running-multiple-grafana-instances

[SuperMatt]

Hi @tonypowa, thanks for this but I did just notice something interesting in my test environment. Imagine I have a datasource which is already set to version 2 internally. If I set my datasource provisioning version to say 2 with a change in it, that change will be applied, but now the version is being reported as 3 in the db.

I think this is the cause of my issues in a nutshell. I think that all my versions were 1 previously, then something happened (I don't know what) and now internally they're all version 2. Then when I went to update my datasources I couldn't because I didn't know that the rug had been swept out from underneath me.

Could it be something has changed in the Grafana codebase to automatically increment the number on update?

[tonypowa]

thanks for your reply @SuperMatt

If I set my datasource provisioning version to say 2 with a change in it, that change will be applied, but now the version is being reported as 3 in the db.

I will add the issue to the team's board for review

[SuperMatt]

Thanks very much!

[undef1nd]

@grafana/plugins-platform-backend as per the codeowners file it's within your area of ownership. Could you please take a look at this issue? Thank you

[andresmgot]

Hi,

This is the expected behavior. The version field ensures that the provisioning file does not overwrite manual changes. If the version set in the provisioning file is not bigger than the one in the database, it will not be updated.

This behavior was fixed in 10.2.3, that's why you may have seen a different behavior in previous versions: #77428

To ensure that the changes in the provisioning file are applied, either increment the version field or simply remove it.

Let us know if you still think there is an issue about this.

[SuperMatt]

Hi @andresmgot I've just checked in Grafana 10.3.1, and the database is still updating the version number to be that of a higher number than the one set.

Here are my replication steps:

1. I connected directly into my mysql DB I reset all of my version numbers to 1. There was 39 of them that needed resetting.
2. I then updated my Prometheus-foobar data source provisioning files, also with version 1 set.
3. I restarted my Grafana instance and watched the logs. In the logs I saw this: {"level":"debug","logger":"provisioning.datasources","msg":"updating datasource from configuration","name":"Prometheus-foobar","t":"2024-01-25T11:20:42.888233271Z","uid":"PFD248D0FD B10D960"}
4. I looked at the version numbers in the Grafana DB and all of the 39 Prometheus datasources that I had previously set to Version 1 have been set to Version 2. That's 38 datasources that I didn't ask to be changed having their versions numbers increased by 1, and 1 datasource which was updated, but I specifically requested the version number remain at 1.

There's clearly a bug here if all of my Prometheus datasources are updating to a new version number despite my provisioning files not requesting them.

[andresmgot]

The version field of the provisioning file is not meant to set the version in the database. It's a safeguard to avoid overwriting changes made manually to the datasource.

Any update made to the datasource, regardless if it's triggered by a provisioning file or a manual change, cause the version in the database to be increased by 1. That's why you are seeing Version 2 after all of them have been updated. The next time the provisioning is run, since version 2 is bigger than version 1 from the file, it won't apply any changes.

If you want your provisioning files to always take precendence over what's stored in the database you need to remove the version field form it.