Epic: bring new CUE-based config schema system to release-readiness

[sdboyer]

#32527 introduces the skeleton of the "scuemata" system for schematizing Grafana objects (primarily dashboards and their constituent parts to start). There's lots of work left to do before the whole system is actually mature enough to power Grafana 8. This epic is intended to enumerate the remaining work.

This doc contains background on the architecture of the system. As indicated in the original PR, the goal of the system is to create patterns for defining canonical schemas for Grafana config objects, such that those schema can be consumed for a variety of purposes across the three target languages - TypeScript, Go, and CUE.

The outstanding TODOs are broken down into some categories.

Automation

We need commands to perform some tasks automatically. Some of these need to work for plugin authors, as well.

• Assemble files into cue/* dirs from packages/* dirs (+ do in CI). (Replace the hacky cuegen.sh)
• CLI command for running cuetsy generation of various plugins (+ do in CI)
• CLI command to perform CUE linting (+ do in CI)
• CLI command to validate basic scuemata family invariants (+ do in CI)
• CLI command to validate resources (e.g. dashboards) against the appropriate schema
• cue fmt step for catching syntax/format errors (git pre-commit hook)
• System for snapshotting existing CUE files at release time for future CI checks to prevent unintentional breakages

Docs

We'd like to avoid much direct user-facing interaction with this system. A side effect of that would be that, at least for Grafana 8, this system ought not require a big user-facing section in our docs. But still, we need some things.

• Example models.cue/whatever files for plugin authors to follow
• Docs on schema and load packages need to be finalized
• Some example-laden docs explaining our reasoning behind when fields should be optional, and when they should have defaults

Backend

The backend is a, and initially the, key consumer of the system's Go API.

• REST endpoint for retrieving JSON Schema representing panels and dashboards, generated from CUE
• REST endpoint for trimming defaults from an object
• Selectively wrap CUE errors with better explanations on paths that are particularly common for people to hit
• InstanceDashboardFamily() to accompany Base and Dist loader implementations (requires registry, plugin dir, etc.)

Frontend

The frontend will be insulated from details of families and lineages, dealing only with the most recent version of schemas.

• Use JSON Schema in panel JSON view
• Add button to remove defaults in JSON view

CUE

• Create scuemata family definition(s?) for datasource plugins, like #PanelFamily
  • Create family defs for all datasource plugins using the above
• Create family defs for all panel plugins
• Create schema for key core components; also, finalize how these work (file & package names, etc.)
• Finalize the actual dashboard schema so we can, y'know, use it

Scuemata system

• Make the Resource type into something actually useful
• Generalize compositional scuemata; convert dashboard scuemata to the generic system
• General approach to composing scuemata version metadata into schema
• Implement ApplyDefaults() (can it be a func instead of a method? 🙏)
• Implement TrimDefaults() (can it be a func instead of a method? 🙏)
• Fix up Migrate() implementations
• Make CueSchema.Validate() take a parameter in order to choose doing open validation or close validation.

Verification

Most things above are assumed to include tests. This is more about systems for testing than individual tests themselves.

• Testing harness + cases for migrations
• Testing harness + cases for panel plugins
• Testing harness + cases for datasource plugins

[sdboyer]

Closing this, we're no longer following it for a roadmap, haven't been for a while.

[justinTM]

@sdboyer hi Sam does this mean CUE is not being worked on by Grafana?

[sdboyer]

@justinTM oh sorry, no! we definitely are, heavily. See the area/scuemata label, and this discussion is a quick sketch of what we're planning on.

We've been doing our work sorta stealthily to this point, but in 8.3 we plan to put a clearer public face on it.

[justinTM]

@sdboyer beautiful. appreciate your candid details in all your replies!

i just took a step back from two months of trying to templating dashboards-as-code for my employer and i think i will freeze development on that nasty Terraform/JSON solution i came up with and instead keep a close eye on CUE and changes to grafonnet.

thanks again

[sdboyer]

I'm quite hopeful you'll be pleased with what we've come up with 🙂