Dashboard workflow for provisioned dashboards

[bergquist]

Grafana currently allows operators to load dashboard JSON files from the filesystem into Grafana using provisioning. The user can edit the dashboard in the browser but cannot save it. To update the dashboard the user has to copy the edited JSON to the files themselves.

One idea that we have is to allow Grafana to…

1. Provision dashboards directly from Github
2. Create pull requests within Grafana
3. Review pull requests within Grafana by rendering dashboards from pull requests.

Questions:

• Would you like to use Duplicate panel with 3 x colspan 3 panels will add not add to empty row #1, Add all Graphite functions to function definition list #2 and Integrate / link function help to graphite functions docs #3 or just one of them?
• Would you have dedicated repositories for dashboards or mix dashboards within other repositories?
• What source code versioning system are you using in your organization? Github? Gitlab? Bitbucket? Stand-alone git server?

There are still many things to consider. So if you are interested in such a feature. Please reach out to us! We want to talk more with you! :) Either in this issue or over mail carl@grafana.com

[jpvincent]

Hi
In my case :

• the operators are not the maintainers and operators dont necessarily have Git credentials, so option 1 (directly from github) is not useful for us.
  As a maintainer, I already use the new provisionning system, so yes I have to copy/paste my modifications on local files, which can lead to errors, git diff them, validate it still works on a local instance and redeploy.

• option 2/3 : having a PR interface directly into Grafana would be nice, but from my experience, using git diff to understand modifications is hard.

My local Git diff helps me spot small modifications, but big ones like creating a new widget or changing the position are hard to get because it moves a lot of text. Also, most of the time it lacks context, so the quickest way for me to review a modification is run side by side the production grafana instance, to a local one.

I'm afraid to ask too much at once, but a PR-like system in Grafana could be more human-friendly and instead of just diffing code, it could name the difference.

Take this default git diff :

diff --git a/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json b/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json
index b418f13..50737c0 100644
--- a/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json
+++ b/user/nextinteractive/dashboards/WebPageTestPageTimingMetrics.json
@@ -11,12 +11,13 @@
         "type": "dashboard"
       },
       {
-        "datasource": "graphite",
+        "datasource": "${DS_GRAPHITE}",
         "enable": true,
         "iconColor": "rgb(96, 255, 103)",
         "limit": 100,
         "name": "runs",
-        "tags": "$base $path $domain $page $browser $connectivity",
+        "showIn": 0,
+        "tags": "$base $path $domain $page $browser $connectivity $location",
         "type": "alert"
       }
     ]

There is a lack of context, I can't know that the changes on the tags fields are related to the annotations list if the operator has not told it to the administrator.
In a perfect Grafana's PR UI interface, it could tell :

* Annotations > Query name "runs" : 
** Field `datasource` changed from "graphite" TO "${DS_GRAPHITE}"
** Hidden : from "true" to "false"
** Field `Tags` changed from "$base $path $domain $page $browser $connectivity" TO "$base $path $domain $page $browser $connectivity $location"

• For the first field, the UI could link to the datasources
• For the 2nd field, I replace the showIn key and its value (0 or 1) by its english translation as seen currently in the UI and the numerical values by a more meaningful word. You can imagine such keys to human translations for the widgets options.
  EG:

      "yaxes": [
        {
-          "format": "short",
+          "format": "dtdurationms",

would spell "Widget 'XXX' > Axes > Y > Unit > changed from "short" to "duration (ms)"

• The 3rd field could be the default : just diffing the innertext is enough. It guess it would also be enough for the changes in widget queries

Next to each line, we could have, like in the Github PR UI, a system to discuss each modification.

We could also refuse certain modifications, line by line, or modify them directly.

Under the modification list, we can display the reviewed dashboard, to check if it actually works. Maybe even display the old one, in order to easily compare them (useful for positions or visual modifications).

Regarding your other questions :

• I already provision the dashboards, so I have all the JSON files in one directory
• We are using a local Git or bitbucket, but among my clients, gitlab seems to be popular

Thank you for asking, and sorry if I think to a too complex UI :)

[roidelapluie]

We would not use any of this. We use jsonnet so that would not be possible at the moment.

[dbluxo]

Hi,

Would you like to use #1, #2 and #3 or just one of them?

we would really like to use 1., 2. and 3.
(But "1." only with our own/private GitLab servers, not the public GitHub)

Would you have dedicated repositories for dashboards or mix dashboards within other repositories?

We are using a config repo for our whole monitoring (dashboards, datasources etc.):

.
├── alertmanager
│   ├── receivers
│   │   ├── r1.yaml
│   │   ├── r2.yaml
│   │   └── r3.yaml
│   └── routes
│       └── r1.yaml
├── grafana
│   ├── dashboards
│   │   ├── folder1
│   │   │   └── dashboard.json
│   │   ├── folder2
│   │   │   └── dashboard.json
│   │   ├── k8s-cluster.json
│   │   ├── k8s-container.json
│   │   ├── k8s-deployments.json
│   │   ├── k8s-node.json
│   │   └── prometheus.json
│   ├── datasources
│   │   ├── alertmanager.json
│   │   ├── elasticsearch.json
│   │   └── prometheus.json
│   └── notification-channels
│       └── nc.json
└── prometheus
    ├── jobs
    │   ├── blackbox-external-endpoints.yml
    │   ├── blackbox-internal-endpoints.yml
    │   ├── kubernetes-endpoints.yml
    │   ├── kubernetes-kubelets.yml
    │   ├── kubernetes-nodes.yml
    │   ├── kubernetes-pods.yml
    │   └── kubernetes-services.yml
    └── rules
        ├── rules1.yaml
        ├── rules2.yaml
        ├── rules3.yaml
        └── rules4.yaml

What source code versioning system are you using in your organization? Github? Gitlab? Bitbucket? Stand-alone git server?

We are hosting our own GitLab.

@bergquist Thank you for opening this issue 👍

[arioch]

We would not use any of this. We use jsonnet so that would not be possible at the moment.

I'd love to see you elaborate how this jsonnet is so much better and why it's not possible to use anything but...
Without a single argument given your response sounds off a smidge more harsh than probably intended to.

[vsliouniaev]

As I understand it, jsonnet can be used to template out the dashboards when deploying them into the cluster. This implies that the entire dashboard configuration is managed as a set of jsonnet files that don't look like the json that Grafana stores.

I don't think that this approach necessarily precludes leveraging scm-managed dashboard necessarily though, it's possible to load dashboards from jsonnet files as well as sourcing them from scm.

---

We are also looking for some solution to manage dashboards without having grafana persistence - but in our case there are two slightly different use-cases

1. We have different grafanas deployed across prod, qa, dev and would like to sync dashboards between them. The current solution involves copying out a dashboard from grafana and loading it into a git repository, which is then synced to the other grafanas via the API. The dashboards are all tagged with git2grafana to indicate that they are liable to get overwritten with changes, For this reason, editing a tagged dashboard usually involves making a copy, testing out the configuration and then replacing the other one in source-control. We have not used this solution for very long but there are some subtleties about having two people editing a dashboard in different environments at the same time - it would be nice to know if a dashboard I'm going to push to scm has some kind of conflict - in our current flow it's quite apparent.

2. In another use-case the dashboards exist on a readonly grafana that is deployed using the stable/grafana helm chart and dashboards are loaded using the sidecars from configmaps. It would be very convenient to include a way to export a saved dashboard to be stored as a configmap as well as being stored in source control. This use-case seems like it's a lot closer to what the proposal is talking about, so I'd be interested in further discussion around this.

[roidelapluie]

@arioch

Grafana does not know about jsonnet right now. This issue is about generating pull requests at the Json level, which is after jsonnet is compiled. This feature makes sense in some use cases but not when you compile your dashboards.

Jsonnet enables us to reuse dashboards, overrides and many other things across a huge amounts of dashboards and customers. Unless grafana gets deep understanding of it, this feature will not be useful.

[peterbourgon]

I guess Fastly is a pretty heavy power-user of Grafana, with almost 100 dashboards between two Grafana systems, each deployed from Git and maintained by individual teams directly. From our perspective,

1. Provision dashboards directly from Github

I guess this means making Grafana "GitHub Aware" and pulling dashboards from the remote? We already have simple and robust tools in place to continuously deploy to relevant machines from Git repos as sources of truth, so this is not useful for us.

2. Create pull requests within Grafana

This would be very useful. Ideally, any dashboard whose definition had deviated from the remote version would get a little green "make a PR" button in the corner somewhere, which would shuttle the relevant changes over to the GitHub UI.

3. Review pull requests within Grafana by rendering dashboards from pull requests.

GitHub PR workflows are well-understood by everyone, and used for all of our change management. We would certainly not want to split our processes between GitHub and Grafana UIs. However, if a Grafana webhook (or whatever GitHub calls them now... actions maybe?) could render dashboards in PRs and post, as a comment, an image, or a link, to the proposed changes, that would be useful.

Some comments here about change set readability, making diffs easier to parse, less noisy, etc. IMO a simple evolutionary improvement of the existing JSON data format, decreasing dashboard definition sizes by (say) half, is not enough to be useful. A more revolutionary change in definition is required, where the ratio of number of panels in a dashboard vs. number of lines in the definition file is ideally ~2x, and no more than (say) ~5x. Native Grafana understanding of Jsonnet might be one way to do this. I guess you could also do it with regular JSON, if there were much more aggressive "layout defaults" or something that each file could opt-in to, and would set dashboard-wide defaults for things like panel size, heuristic interpretation of data for Y-axis labels, fill styles, etc.

[bergquist]

Ideally, any dashboard whose definition had deviated from the remote version would get a little green "make a PR"

Right now, I can't see how this will be possible unless Grafan is Github/Git aware.

I guess you could also do it with regular JSON, if there were much more aggressive "layout defaults" or something that each file could opt-in to, and would set dashboard-wide defaults for things like panel size, heuristic interpretation of data for Y-axis labels, fill styles, etc.

Sounds similar to what we described in this issue #13888

[peterbourgon]

@bergquist

Ideally, any dashboard whose definition had deviated from the remote version would get a little green "make a PR"

Right now, I can't see how this will be possible unless Grafan is Github/Git aware.

Yeah, I agree, I think some degree of Grafana/GitHub awareness would be good, especially a webhook or action or something enhancing PRs on specific e.g. GitHub repos would be nice. I was pushing back on the idea that Grafana could own, or take over, the PR workflow from GitHub somehow.