Future of this library

[andreasgerstmayr]

Is this library still the best way to generate Grafana dashboards from code, or will it be replaced with Cue sooner or later?

There was an example of a Cue generated dashboard at the "Dashboards as code" talk at GrafanaCon 2021 (https://grafana.com/go/grafanaconline/2021/dashboards-as-code/, around minute 21), and this repo looks to be on the move (manually maintained jsonnet -> generated jsonnet for grafana 7.0 -> ??). grafana/grafana#31038 still mentions grafonnet however, so I'm wondering what's the current status of generating dashboards from code?

[nvvu]

I have the same question.

[sdboyer]

It's a super-reasonable question! tl;dr - parts, possibly all of grafonnet will gradually move towards deprecation. CUE will be the recommended path, though depending on demand/contributions, a bridge from grafonnet to CUE is not inconceivable.

There's plenty of caveats and uncertainty, to be applied, but it is absolutely clear that by introducing CUE schemas for Grafana dashboards, we are fundamentally changing the playing field on which grafonnet has historically operated. That, in turn, means that grafonnet's role will necessarily change. The clearest way to talk about this, given that there's still a road ahead and attendant uncertainty, is in terms of the properties of a functioning [dashboards-]as-code system, how grafonnet satisfies those today, and how they're likely to change in the future.

Key properties/components:

• A schema specifying what keys and value types are permissible in the config-code representation of the entity we think of as "a Grafana dashboard"
• General-purpose default values for at least some of the keys in that schema
• Helpers that make it easier to create dashboard JSON that achieves some well-defined, high level goal

Today, these lines are pretty blurry. There is no formal schema; instead, there's just the informal "does the Grafana frontend application seem to handle this pile of JSON well enough?" In practice, that places the responsibility for schema somewhere awkwardly halfway between grafonnet and the frontend app - the frontend app is the actual decider about what's valid, but grafonnet is the de facto gateway that users encounter.

The other two are thoroughly murky. Some defaults are effectively encoded within the frontend application - i.e., there's some typescript code that decides, "if a particular key is absent from a dashboard/panel object, i'll treat that absence as equivalent to some value, X". The defaults grafonnet provides are more obvious - here's one simple, clear case.

Helpers are clearly only a grafonnet thing, not a frontend thing, but they're not cleanly distinguishable from providing default values. This is largely due to the nature of jsonnet - because every value in every tree can be overridden at any time during jsonnet evaluation, that makes everything essentially a default - a value that might change later if something decides to override it.

With the CUE schemas in place, the lines between these things will become clearer. The first item is obvious - there'll be explicit schema. (Grafana's frontend is still the proper owner of the schema, but rather than an implicit behavioral outcome, it's now an explicit contract that can be statically observed and interacted with, without the need to run the frontend app.)

CUE also allows the specification of default values. That means the upstream schema itself will take on the clear responsibility there, largely obviating the need for these long lists of arguments.

The logical position that grafonnet - or something like it - may still occupy is in providing helpers. I wrote more about what such helpers could look like in a CUE-centric world over here. That post also discusses out that, for folks who really want to stay in a jsonnet-centric toolchain, that should be a feasible path.

[andreasgerstmayr]

Thanks for this detailed response!

Cue sounds quite exciting. By making the default values uniform and explicit, less code will be required for basic dashboards, removing the need of many jsonnet helpers - imho many of them just filled in boilerplate code required for the resulting dashboard json file.

My question is answered - shall I close this ticket, or should it be pinned to raise awareness of the new direction regarding grafonnet/Cue?

[sdboyer]

i am, needless to say, very excited about the future of CUE :)

Oh, yeah, pinning is a good idea! I'll do that.

re: this point:

removing the need of many jsonnet helpers - imho many of them just filled in boilerplate code required for the resulting dashboard json file.

Serendipitously, this just happened: pollypkg/polly#26. It's all still early and handwavy, but it's indicative of how (especially if you have time to watch the video), once reliable schema move us past the relatively banal problems of boilerplate reduction, a whole new universe of "helpers" comes into view.

[MatanRubin]

Thanks for all the details and explanations! I must say this thread has left me confused 😀
What's the bottom line here? I just learned about Grafonnet and was excited to give it a try, but after reading this - should I? If the answer is no, what's a better alternative?

I understand that going with Cue is the direction, but is that something that's available right now? If not, is there a timeline for it?

Thanks!

[Sanhajio]

This is available right now, you can still use it, to a certain extent, when the new version is stable migrate to the newer version.

[malcolmholmes]

@MatanRubin Grafonnet works, and has been in use for 3yrs+. For simple/medium projects, it can work fine. What it struggles with is keeping up with the development of Grafana itself. New features are added to Grafana faster than we can keep up adding them to grafonnet, especially given that this library is maintained because it is useful to us - we're not here to maintain it full time.

If you need to do dashboards as code, the Jsonnet approach is all there is at present. The Cue approach is under development, and hopefully soon will be usable by early adopters. Keep checking back though, I suspect there's gonna be some interesting things coming.

[MatanRubin]

Thanks @malcolmholmes , that's much clearer!

[justinTM]

@sdboyer thank you very much for the detailed response!

[justinTM]

For others interested: grafana/grafana#39593

[malcolmholmes]

We have a new version of Grafonnet under development. It is early, and we are open to feedback. It has a new repo, as it is essentially a new codebase: https://github.com/grafana/grafonnet. This repo will eventually be deprecated/archived in a way that keeps existing code running, but doesn't accept updates. Please try the library and give us your feedback, over in the new repo.