Dashboard workflow for provisioned dashboards

[bergquist]

Grafana currently allows operators to load dashboard JSON files from the filesystem into Grafana using provisioning. The user can edit the dashboard in the browser but cannot save it. To update the dashboard the user has to copy the edited JSON to the files themselves.

One idea that we have is to allow Grafana to…

1. Provision dashboards directly from Github
2. Create pull requests within Grafana
3. Review pull requests within Grafana by rendering dashboards from pull requests.

Questions:

• Would you like to use Duplicate panel with 3 x colspan 3 panels will add not add to empty row #1, Add all Graphite functions to function definition list #2 and Integrate / link function help to graphite functions docs #3 or just one of them?
• Would you have dedicated repositories for dashboards or mix dashboards within other repositories?
• What source code versioning system are you using in your organization? Github? Gitlab? Bitbucket? Stand-alone git server?

There are still many things to consider. So if you are interested in such a feature. Please reach out to us! We want to talk more with you! :) Either in this issue or over mail carl@grafana.com

[jpvincent]

Hi
In my case :

• the operators are not the maintainers and operators dont necessarily have Git credentials, so option 1 (directly from github) is not useful for us.
  As a maintainer, I already use the new provisionning system, so yes I have to copy/paste my modifications on local files, which can lead to errors, git diff them, validate it still works on a local instance and redeploy.

• option 2/3 : having a PR interface directly into Grafana would be nice, but from my experience, using git diff to understand modifications is hard.

My local Git diff helps me spot small modifications, but big ones like creating a new widget or changing the position are hard to get because it moves a lot of text. Also, most of the time it lacks context, so the quickest way for me to review a modification is run side by side the production grafana instance, to a local one.

I'm afraid to ask too much at once, but a PR-like system in Grafana could be more human-friendly and instead of just diffing code, it could name the difference.

Take this default git diff :

diff --git a/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json b/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json
index b418f13..50737c0 100644
--- a/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json
+++ b/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json
@@ -11,12 +11,13 @@
         "type": "dashboard"
       },
       {
-        "datasource": "graphite",
+        "datasource": "${DS_GRAPHITE}",
         "enable": true,
         "iconColor": "rgb(96, 255, 103)",
         "limit": 100,
         "name": "runs",
-        "tags": "$base $path $domain $page $browser $connectivity",
+        "showIn": 0,
+        "tags": "$base $path $domain $page $browser $connectivity $location",
         "type": "alert"
       }
     ]

There is a lack of context, I can't know that the changes on the tags fields are related to the annotations list if the operator has not told it to the administrator.
In a perfect Grafana's PR UI interface, it could tell :

* Annotations > Query name "runs" : 
** Field `datasource` changed from "graphite" TO "${DS_GRAPHITE}"
** Hidden : from "true" to "false"
** Field `Tags` changed from "$base $path $domain $page $browser $connectivity" TO "$base $path $domain $page $browser $connectivity $location"

• For the first field, the UI could link to the datasources
• For the 2nd field, I replace the showIn key and its value (0 or 1) by its english translation as seen currently in the UI and the numerical values by a more meaningful word. You can imagine such keys to human translations for the widgets options.
  EG:

      "yaxes": [
        {
-          "format": "short",
+          "format": "dtdurationms",

would spell "Widget 'XXX' > Axes > Y > Unit > changed from "short" to "duration (ms)"

• The 3rd field could be the default : just diffing the innertext is enough. It guess it would also be enough for the changes in widget queries

Next to each line, we could have, like in the Github PR UI, a system to discuss each modification.

We could also refuse certain modifications, line by line, or modify them directly.

Under the modification list, we can display the reviewed dashboard, to check if it actually works. Maybe even display the old one, in order to easily compare them (useful for positions or visual modifications).

Regarding your other questions :

• I already provision the dashboards, so I have all the JSON files in one directory
• We are using a local Git or bitbucket, but among my clients, gitlab seems to be popular

Thank you for asking, and sorry if I think to a too complex UI :)

[roidelapluie]

We would not use any of this. We use jsonnet so that would not be possible at the moment.

[dbluxo]

Hi,

Would you like to use #1, #2 and #3 or just one of them?

we would really like to use 1., 2. and 3.
(But "1." only with our own/private GitLab servers, not the public GitHub)

Would you have dedicated repositories for dashboards or mix dashboards within other repositories?

We are using a config repo for our whole monitoring (dashboards, datasources etc.):

.
├── alertmanager
│   ├── receivers
│   │   ├── r1.yaml
│   │   ├── r2.yaml
│   │   └── r3.yaml
│   └── routes
│       └── r1.yaml
├── grafana
│   ├── dashboards
│   │   ├── folder1
│   │   │   └── dashboard.json
│   │   ├── folder2
│   │   │   └── dashboard.json
│   │   ├── k8s-cluster.json
│   │   ├── k8s-container.json
│   │   ├── k8s-deployments.json
│   │   ├── k8s-node.json
│   │   └── prometheus.json
│   ├── datasources
│   │   ├── alertmanager.json
│   │   ├── elasticsearch.json
│   │   └── prometheus.json
│   └── notification-channels
│       └── nc.json
└── prometheus
    ├── jobs
    │   ├── blackbox-external-endpoints.yml
    │   ├── blackbox-internal-endpoints.yml
    │   ├── kubernetes-endpoints.yml
    │   ├── kubernetes-kubelets.yml
    │   ├── kubernetes-nodes.yml
    │   ├── kubernetes-pods.yml
    │   └── kubernetes-services.yml
    └── rules
        ├── rules1.yaml
        ├── rules2.yaml
        ├── rules3.yaml
        └── rules4.yaml

What source code versioning system are you using in your organization? Github? Gitlab? Bitbucket? Stand-alone git server?

We are hosting our own GitLab.

@bergquist Thank you for opening this issue 👍

[arioch]

We would not use any of this. We use jsonnet so that would not be possible at the moment.

I'd love to see you elaborate how this jsonnet is so much better and why it's not possible to use anything but...
Without a single argument given your response sounds off a smidge more harsh than probably intended to.

[vsliouniaev]

As I understand it, jsonnet can be used to template out the dashboards when deploying them into the cluster. This implies that the entire dashboard configuration is managed as a set of jsonnet files that don't look like the json that Grafana stores.

I don't think that this approach necessarily precludes leveraging scm-managed dashboard necessarily though, it's possible to load dashboards from jsonnet files as well as sourcing them from scm.

---

We are also looking for some solution to manage dashboards without having grafana persistence - but in our case there are two slightly different use-cases

1. We have different grafanas deployed across prod, qa, dev and would like to sync dashboards between them. The current solution involves copying out a dashboard from grafana and loading it into a git repository, which is then synced to the other grafanas via the API. The dashboards are all tagged with git2grafana to indicate that they are liable to get overwritten with changes, For this reason, editing a tagged dashboard usually involves making a copy, testing out the configuration and then replacing the other one in source-control. We have not used this solution for very long but there are some subtleties about having two people editing a dashboard in different environments at the same time - it would be nice to know if a dashboard I'm going to push to scm has some kind of conflict - in our current flow it's quite apparent.

2. In another use-case the dashboards exist on a readonly grafana that is deployed using the stable/grafana helm chart and dashboards are loaded using the sidecars from configmaps. It would be very convenient to include a way to export a saved dashboard to be stored as a configmap as well as being stored in source control. This use-case seems like it's a lot closer to what the proposal is talking about, so I'd be interested in further discussion around this.

[roidelapluie]

@arioch

Grafana does not know about jsonnet right now. This issue is about generating pull requests at the Json level, which is after jsonnet is compiled. This feature makes sense in some use cases but not when you compile your dashboards.

Jsonnet enables us to reuse dashboards, overrides and many other things across a huge amounts of dashboards and customers. Unless grafana gets deep understanding of it, this feature will not be useful.

[peterbourgon]

I guess Fastly is a pretty heavy power-user of Grafana, with almost 100 dashboards between two Grafana systems, each deployed from Git and maintained by individual teams directly. From our perspective,

1. Provision dashboards directly from Github

I guess this means making Grafana "GitHub Aware" and pulling dashboards from the remote? We already have simple and robust tools in place to continuously deploy to relevant machines from Git repos as sources of truth, so this is not useful for us.

2. Create pull requests within Grafana

This would be very useful. Ideally, any dashboard whose definition had deviated from the remote version would get a little green "make a PR" button in the corner somewhere, which would shuttle the relevant changes over to the GitHub UI.

3. Review pull requests within Grafana by rendering dashboards from pull requests.

GitHub PR workflows are well-understood by everyone, and used for all of our change management. We would certainly not want to split our processes between GitHub and Grafana UIs. However, if a Grafana webhook (or whatever GitHub calls them now... actions maybe?) could render dashboards in PRs and post, as a comment, an image, or a link, to the proposed changes, that would be useful.

Some comments here about change set readability, making diffs easier to parse, less noisy, etc. IMO a simple evolutionary improvement of the existing JSON data format, decreasing dashboard definition sizes by (say) half, is not enough to be useful. A more revolutionary change in definition is required, where the ratio of number of panels in a dashboard vs. number of lines in the definition file is ideally ~2x, and no more than (say) ~5x. Native Grafana understanding of Jsonnet might be one way to do this. I guess you could also do it with regular JSON, if there were much more aggressive "layout defaults" or something that each file could opt-in to, and would set dashboard-wide defaults for things like panel size, heuristic interpretation of data for Y-axis labels, fill styles, etc.

[bergquist]

Ideally, any dashboard whose definition had deviated from the remote version would get a little green "make a PR"

Right now, I can't see how this will be possible unless Grafan is Github/Git aware.

I guess you could also do it with regular JSON, if there were much more aggressive "layout defaults" or something that each file could opt-in to, and would set dashboard-wide defaults for things like panel size, heuristic interpretation of data for Y-axis labels, fill styles, etc.

Sounds similar to what we described in this issue #13888

[peterbourgon]

@bergquist

Ideally, any dashboard whose definition had deviated from the remote version would get a little green "make a PR"

Right now, I can't see how this will be possible unless Grafan is Github/Git aware.

Yeah, I agree, I think some degree of Grafana/GitHub awareness would be good, especially a webhook or action or something enhancing PRs on specific e.g. GitHub repos would be nice. I was pushing back on the idea that Grafana could own, or take over, the PR workflow from GitHub somehow.

[Xopherus]

Thanks for opening this @bergquist. I just started using the provisioning feature a few months ago, I'm glad that others are starting to think about improvements.

Would you like to use #1, #2 and #3 or just one of them?

I agree with others including @peterbourgon that option 2 is the best approach. The workflow for me and my team is to make modifications to a dashboard, export, then paste the JSON in our repo and open a PR. Implementing a feature to make that process more seamless is 👍. I don't think the right solution is to implement a peer review system within Grafana. Though I do agree that peer review of these dashboard JSON files is very tricky, so that should be addressed, but I think it can be decoupled from this particular issue.

Would you have dedicated repositories for dashboards or mix dashboards within other repositories?

I've decided to have all dashboards within 1 repository (we ship them with our grafana docker image). We have ~30-40 dashboards in version control now. I'm not sure if others agree, but it's worked for us so far. This issue and #12896 are really the 2 big pain points for us right now.

What source code versioning system are you using in your organization? Github? Gitlab? Bitbucket? Stand-alone git server?

GitHub.

[Chris-Hicks]

We have been looking into the idea of using the dashboard provisioning and was considering how we could integrate into GitHub

Would you like to use #1, #2 and #3 or just one of them?

Certainly 1 & 2. Being able to pull dashboards down from GitHub would be really useful. It also makes managing separate instances with the same dashboards straight forward. This was how we were considering our own integration. Similarly being able to push changes back by an authenticated user would be useful.

3 would be a nice to have feature but at this moment in time we do not peer review dashboard changes and would be happy from them to be merged from 2.

Would you have dedicated repositories for dashboards or mix dashboards within other repositories?

We would have a dedicated repository

What source code versioning system are you using in your organization?

GitHub - Private Repo

[BertHartm]

As users of git for our dashboards, but not Github:

I think it would be helpful to somehow allow creating commits/PRs from the grafana UI. PRs would be difficult since we don't use github, but anything that simplified the process of copying the json from the settings page, and pasting into the git repo would be nice.

It would also be nice to have some sort of api endpoint for validating or generating renders of what a dashboard would look like so we could build a process to show a visual diff on PRs to make reviewing easier.

[mgomersbach]

Would indeed much prefer a general export/import functionality with git, s3, gcp etc as backends

[tiarebalbi]

Is this a feature that the Grafana team would consider? More than happy to help in the development of this feature.

[stavalfi]

what is the status of this issue? this is extremely needed to work with grafana

[seuf]

A feature i would love to see is the provisioning from dashboards ids.
in the dashboards.yml provisioning file, allow the possibility to install dashboard from grafana.com/dashboards with their ids.
example :

apiVersion: 1
providers:
  - name: grafana-dashboards-public
    folder: ''
    type: id
    updateIntervalSeconds: 300
    allowUiUpdates: true
    options:
      ids:
        - 11074
        - 11802

[0dragosh]

Has there been any progress on this?

[dnck]

@0dragosh apparently not...

I'd be happy to help with it though.

[mkonopi]

Hi,

Would you like to use #1, #2 and #3 or just one of them?

we would really like to use 1., 2. and 3. (But "1." only with our own/private GitLab servers, not the public GitHub)

Would you have dedicated repositories for dashboards or mix dashboards within other repositories?

We are using a config repo for our whole monitoring (dashboards, datasources etc.):

.
├── alertmanager
│   ├── receivers
│   │   ├── r1.yaml
│   │   ├── r2.yaml
│   │   └── r3.yaml
│   └── routes
│       └── r1.yaml
├── grafana
│   ├── dashboards
│   │   ├── folder1
│   │   │   └── dashboard.json
│   │   ├── folder2
│   │   │   └── dashboard.json
│   │   ├── k8s-cluster.json
│   │   ├── k8s-container.json
│   │   ├── k8s-deployments.json
│   │   ├── k8s-node.json
│   │   └── prometheus.json
│   ├── datasources
│   │   ├── alertmanager.json
│   │   ├── elasticsearch.json
│   │   └── prometheus.json
│   └── notification-channels
│       └── nc.json
└── prometheus
    ├── jobs
    │   ├── blackbox-external-endpoints.yml
    │   ├── blackbox-internal-endpoints.yml
    │   ├── kubernetes-endpoints.yml
    │   ├── kubernetes-kubelets.yml
    │   ├── kubernetes-nodes.yml
    │   ├── kubernetes-pods.yml
    │   └── kubernetes-services.yml
    └── rules
        ├── rules1.yaml
        ├── rules2.yaml
        ├── rules3.yaml
        └── rules4.yaml

What source code versioning system are you using in your organization? Github? Gitlab? Bitbucket? Stand-alone git server?

We are hosting our own GitLab.

@bergquist Thank you for opening this issue 👍

@dbluxo how did you configure your grafana.dashboards to get files from GitLab?
Specifically url - did you use API url or plain link to file ?
And if "b64content: true"

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!

[GreyTeardrop]

I believe this feature request is still relevant

[beasteers]

Unless I misunderstand, it looks like this issue #52916 is tracking progress made on this front, though it looks like development has fallen off so I'm not sure what that means. Though it looks like a lot more progress has been made compared to what the checkmarks show.

I'm also holding out for version controlled store/backup/restore with grafana. Ideally with more than just dashboards (folders, plugins, alerts). Having a dedicated PR UI would be immensely useful for syncing staging and production environments