Making dashboards compatible with grafana-API

[florianmulatz]

We want to import grafana-dashboards with Grafana own API. Since we make some specific modification on our dashboard (own variables for region like in the screenshot for example) it would be nice to support the own Grafana-API as well.

From my current perspective it should be only needed to add some specific identifiers to get them compatible (importable):
URL

Expected behavior
Grafana Dashboard should be importable via grafana-API

Actual behavior
own modifications to json-files are needed - afterwards dashboard is empty (not sure if this is maybe a self-made error)

[rahulguptajss]

@florianmulatz We are working on this area and plan to simplify grafana import for upcoming release. There is a PR open #831 . I don't see the screenshot in your comment above. Could you provide more information about your usecase.

We are planning to support below commands. Let us know if your use case is covered.

Import:

bin/grafana import --addr my.grafana.server:3000
bin/grafana import --addr my.grafana.server:3000 --serverfolder [grafanafoldername] --directory [path]

Export:

bin/grafana export --addr my.grafana.server:3000 --serverfolder [grafanafoldername] --directory [path]

[rahulguptajss]

@florianmulatz Above mentioned PR is now merged into main. You can give it a try with harvest nightly builds https://github.com/NetApp/harvest/releases/tag/nightly

[florianmulatz]

Hi @rahulguptajss, first of all the missing screenshots

I've created a own label on that within the prometheus configuration for example

  - job_name: 'netapp'
    scrape_interval: 60s
    static_configs:
      - targets:
        - '10.162.170.102:12000' ## vihsdnclu01
        labels:
          device: vihsdnclu01
          region: EMEA

[florianmulatz]

@florianmulatz Above mentioned PR is now merged into main. You can give it a try with harvest nightly builds https://github.com/NetApp/harvest/releases/tag/nightly

I'll set up a test-machine where I can try nightly-builds and releases before rolling-out in production - thx for that!
My knowledge about that is that importing the dashboards currently simply deletes the old one and creates the new one right?

[rahulguptajss]

@florianmulatz Above mentioned PR is now merged into main. You can give it a try with harvest nightly builds https://github.com/NetApp/harvest/releases/tag/nightly

I'll set up a test-machine where I can try nightly-builds and releases before rolling-out in production - thx for that! My knowledge about that is that importing the dashboards currently simply deletes the old one and creates the new one right?

Latest changes in nightly do not overwrite dashboards if uploaded to a different grafana folder. We are using grafana api's only inside harvest grafana import.

Thanks for the screenshot but it is not clear to me the problem statement here. If you could elaborate.

[florianmulatz]

@florianmulatz Above mentioned PR is now merged into main. You can give it a try with harvest nightly builds https://github.com/NetApp/harvest/releases/tag/nightly

I'll set up a test-machine where I can try nightly-builds and releases before rolling-out in production - thx for that! My knowledge about that is that importing the dashboards currently simply deletes the old one and creates the new one right?

Latest changes in nightly do not overwrite dashboards if uploaded to a different grafana folder. We are using grafana api's only inside harvest grafana import.

Thanks for the screenshot but it is not clear to me the problem statement here. If you could elaborate.

Hi @rahulguptajss - screenshot does not figure a problem - just wanted to visualize our own changes (I guess it's useless for this thread 😆 )

Thx for your answer regarding the importer - that's what I wanted to have. I'll try this within my testmachine today and give a update if it works (at least for me)

[florianmulatz]

Hey, unfortunately import does not work

[root@vihlv0493 bin]# ./grafana import --addr "grafana.internal.url" --config ../harvest.yml -f "Harvest 2.0" --directory "/opt/harvest/grafana/dashboards/cmode/test/" -S
using API token from config
connected to Grafana server (version: 8.3.4)
folder [Harvest 2.0] exists in Grafana - OK
preparing to import dashboards...
error importing [harvest_dashboard_volume_details.json] - server response (400 - 400 Bad Request) map[message:uid contains illegal characters]

Anyhow I tried to set the uid in the source-json file to "null" as stated in the Grafana-API documentation for importing but - still the same message.

BR

[florianmulatz]

PS: It looks like it has todo with my Grafana-Folder Name which contains a WhiteSpace

[root@vihlv0493 bin]# ./grafana import --addr "´grafana.internal.url" --config ../harvest.yml -f "Harvest2" --directory "/opt/harvest/grafana/dashboards/cmode/test/" -S
using API token from config
connected to Grafana server (version: 8.3.4)
created Grafana folder [Harvest2] - OK
preparing to import dashboards...
OK - imported Harvest2 / [harvest_dashboard_volume_details.json]
Imported 1 dashboards to [Harvest2] from [/opt/harvest/grafana/dashboards/cmode/test/]

I tried renaming my "real" folder to something without white-space and "." - and this worked. Anyhow It does exactly what I was afraid of - on Importing the new-dashboard it overwrites all our own modifications (region variable) and just replace it with the new one.

Maybe this is "works as desgined" - I don't know - that's what my first question was targetet on

[rahulguptajss]

PS: It looks like it has todo with my Grafana-Folder Name which contains a WhiteSpace

[root@vihlv0493 bin]# ./grafana import --addr "´grafana.internal.url" --config ../harvest.yml -f "Harvest2" --directory "/opt/harvest/grafana/dashboards/cmode/test/" -S
using API token from config
connected to Grafana server (version: 8.3.4)
created Grafana folder [Harvest2] - OK
preparing to import dashboards...
OK - imported Harvest2 / [harvest_dashboard_volume_details.json]
Imported 1 dashboards to [Harvest2] from [/opt/harvest/grafana/dashboards/cmode/test/]

I tried renaming my "real" folder to something without white-space and "." - and this worked. Anyhow It does exactly what I was afraid of - on Importing the new-dashboard it overwrites all our own modifications (region variable) and just replace it with the new one.

Maybe this is "works as desgined" - I don't know - that's what my first question was targetet on

Thanks @florianmulatz for the test. Can you try uploading to a different folder to avoid overwriting if that helps? Inside same folder, we want to avoid multiple dashboards with same name. I'll handle white-space or special chars for folder name

[cgrinds]

hi @florianmulatz we'll take a look and see how we can improve this. In the case that the dashboard was overwritten, do you see the version tab and can you restore the previous version?

[florianmulatz]

hi @florianmulatz we'll take a look and see how we can improve this. In the case that the dashboard was overwritten, do you see the version tab and can you restore the previous version?

yes - that works

[cgrinds]

hi @florianmulatz here's what we've determined about the Grafana import API - there are lots of permutations :)

A combination of a dashboard's uid, dashboard title, folder name and overwrite flag are used by Grafana to determine whether a dashboard should be replaced or newly created when using the POST /api/dashboards/db API.

Given the logic described here, and our desire to not overwrite customer dashboard changes when importing, we're going to change Harvest's import logic. When importing, by default, Harvest will use the Grafana API POST /api/dashboards/db with overwrite: false. Grafana will return an error instead of overwriting your dashboard. When that happens, Harvest will provide you with two options:

• you can ignore the warning and overwrite anyway by using the --overwrite command line flag
• specify a different Grafana server folder to import the dashboards into

Does this work for you?

[florianmulatz]

Of course this will work - but this does not address the main problem (for us) that our own modifications has to be re-done in the replaced / new created dashboard. Maybe there is any chance to diff the files (maybe just for variable adoptions) or something like that?

To be honest, I have no reasonable idea myself at the moment how to address this cleanly 😕

[cgrinds]

@florianmulatz yes I wish I had a better answer about diffing. That's something we run into also. Grafana does provide a json diff across different versions of the same dashboard. Not directly applicable to your usecase, but perhaps something we can use in the future.

[florianmulatz]

I've updated the harvester to 22.02.0 on our test-machine and imported some dashboards for test. Unfortunately, it still overwrites our own changes.

[rahulguptajss]

@florianmulatz

./bin/grafana import should give error if grafana folder and dashboard title is same. If you have changes then suggestion would be to use a seperate grafana folder by passing arg --serverfolder and relevant local dir folder as --directory

Example command:
bin/grafana import --addr my.grafana.server:3000 --serverfolder [grafanafoldername] --directory [path]

Could you provide the commands you have tried for us to take a look.

[florianmulatz]

Hi @rahulguptajss - I'm confused. The changelog from 22.02.0 states clearly --

bin/grafana cli should not overwrite dashboard changes, ....

how is this to understand then?
Cheers

[rahulguptajss]

yes, that means by default, overwrite will not happen and cli will fail like below. To overwrite --overwrite needs to be passed

./bin/grafana import       
using API token from config
connected to Grafana server (version: 8.1.2)
folder [Harvest-22.02.2620-cDOT] exists in Grafana - OK
folder [Harvest-22.02.2620-7mode] exists in Grafana - OK
preparing to import dashboards...
error importing [harvest_dashboard_aggregate.json] to folder [Harvest-22.02.2620-cDOT] - the server replied with [412 Precondition Failed]
That means you are trying to overwrite an existing dashboard.

If you want to overwrite run with the --overwrite flag or choose a different Grafana folder with the --serverfolder flag

Server response follows:
    message => A dashboard with the same name in the folder already exists
    status => name-exists

error importing [harvest_dashboard_aggregate7.json] to folder [Harvest-22.02.2620-7mode] - the server replied with [412 Precondition Failed]
That means you are trying to overwrite an existing dashboard.

If you want to overwrite run with the --overwrite flag or choose a different Grafana folder with the --serverfolder flag

Server response follows:
    message => A dashboard with the same name in the folder already exists
    status => name-exists

[cgrinds]

hi @florianmulatz with 22.02 your dashboards shouldn't be overwritten. When Harvest calls the Grafana API here it sets overwrite=false unless the command line argument is included.

Let's see if we can figure out what's different in your environment. When you get a chance, can you paste the Global Flags returned when running?

bin/grafana import --help