POC: Generate jsonnet library from upstream schema #376
Conversation
|
Amazing to see this! We've got about ten panel plugins schematized, though no datasource plugins yet. However, the pipeline is built, so as soon as the respective teams get their schemas written, they'll come rolling in automatically. It's a lot of chicken-or-egg, but being able to show the outputs - such as this, here - is crucial to demonstrating the value of actually getting the schemas written. Which, in turn, makes it easier to prioritize the schematization work division-wide. The primary interface core grafana is presenting to all the schemas is for importing them from Go. i've taken some care to make it possible to do what you've done here with raw However, now that i can see the kind of output pattern you're going for, i can throw together what should be a fairly trivial Go program (LoC-wise) that generates equivalent output! |
sdboyer
force-pushed
the
duologic/poc_lib_from_schema
branch
from
e3e1ea1
to
f54618c
Compare
|
OK, i spent a while on this yesterday, and discovered quite the frustrating bug in the course of getting the Go code pushed up. i've made some inline comments for explanation on the Go code itself. Overall, my take is that, while it's possible and (IMO) roughly equivalent ergonomics using |
sdboyer
force-pushed
the
duologic/poc_lib_from_schema
branch
from
f54618c
to
da54d8b
Compare
| "PanelLayout", | ||
| "PanelOptions" |
There was a problem hiding this comment.
Panel composable kinds are what i'm tentatively calling a grouped lineage - see grafana/thema#62. There isn't official support for these in Thema yet. Consequently, these are generated as a single schema object, rather than the (maybe, depending on how you plan to use these) preferable approach of generating them as two distinct schemas.
My guess is that, at least for now, you can hardcode what you need in jsonnet
There was a problem hiding this comment.
It looks like dashlist.PanelLayout and dashlist.PanelOptions needs to be a separate schema under components.schemas instead of a property of the dashlist schema. The ref further down already seems to know this:
"layout": {
"$ref": "#/components/schemas/dashlist.PanelLayout"
},
grafonnext/mkschema.go
Outdated
| // There's a bug in generating openapi from Thema with imports are loaded. | ||
| // Normally an edge case in Thema, but needed in Grafana. Commenting this out | ||
| // until bug is fixed and just dropping errors instead | ||
| // die(fmt.Errorf("%s: %w", ptree.RootPlugin().Meta().Id, err)) |
There was a problem hiding this comment.
If this is uncommented, then we get errors like the following:
barchart: package "github.com/grafana/grafana/packages/grafana-schema/src/schema" imported but not defined in (and 5 more errors)
This happens because of a fugly hack i have to do in Thema in order to be able to generate OpenAPI at all, because the OpenAPI encoder uses the old, deprecated cue.Instance. Basically, that func writes out a text version of the lineage and recompiles it through a backdoor - but when recompiling, external imports can't be sanely satisfied.
They fixed the API in the recent alpha release, so i no longer have to export, but...well, there's another problem, and i probably just need to submit a patch upstream. Hoping to do that this week - this is also a prerequisite for grok, so it needs to get done regardless.
grafonnext/mkschema.go
Outdated
| base := filepath.Join("schemas", grafanaVersion, "core") | ||
| die(os.MkdirAll(base, 0777)) | ||
|
|
||
| for _, kind := range corekind.NewBase(nil).AllStructured() { |
There was a problem hiding this comment.
Probably the main thing that using the kind as represented in Go offers is all the cue metadata/DSL we'll be wrapping around the kind declarations. In a Go context, these structs will be kept in sync with what's in those cue framework definitions, and are easy to access. It will still probably be possible with generic CLI tooling - use thema to grab out lineages from known subpaths, and then cue to extract certain known metadata from other subpaths, emit it as JSON, etc. But i suspect that will get increasingly unwieldy compared to just using the system designed to make this all easy to access from Go, in an external repo, as we're doing here.
There was a problem hiding this comment.
The main area of interest right now with this is probably maturity flags, btw - it seems wise to indicate something about the maturity levels and corresponding guarantees that come with them for each schema.
My stance on grok is that i'm not willing to put it out at all until we have clear comms in that regard. I understand how there are different considerations here, where there were never guarantees in the first place. So probably fine to not consider that part of MVP here.
grafonnext/mkschema.go
Outdated
| // this is a meaningless hack, we have to work out versioning | ||
| var grafanaVersion = "latest" |
There was a problem hiding this comment.
This is just very clearly not what you're looking for here. And it's equally clear to me that we do not currently have sufficient information available in Go scope to organize these outputs by Grafana release version.
That said, working on this gave me a thought about how we might be able to make that information available in a tidy way that would be useful here and elsewhere. Might be a good thing for @malcolmholmes to look at, as it straddles some known overlaps with release engineering.
There was a problem hiding this comment.
Looks like this will be covered by grafana/grafana#58498, right?
|
General agreement @Duologic and i reached in backchannel is that the Go logic to generate JSON Schema here is probably something we can do in grok (and be owned by @grafana/grafana-as-code), and then this can just depend on grok for those generated files. That fits with the larger pattern grok is aiming to fill, anyway! |
- moves go version to separate directory so that jb install will not pull in the go-code. - update main.libsonnet to process the extra schemas
|
POC has served its purpose, closing this. |
What's this?
Upstream on github.com/grafana/grafana, work is in progress to provide a schema (in CUE)
for Grafana objects. This PR takes a first pass at providing a jsonnet library based on
this schema.
How does it work?
The script
bin/mkschema.shclones a specific Grafana version and uses Thema to generatea Json Schema (OpenAPI schema actually). It then uses CRDsonnet to convert this schema
into a library with jsonnet functions. In
main.libsonnet, we also add a few extrashortcuts in addition to the generated functions. In
example.libsonnetwe use thislibrary to generate in simple dashboard.
What does it give me?
It is still early days for the schema, the upstream schema currently limited to the
dashboard schema. And even in that there are lots of missing bits and pieces that are
currently covered for much better in the existing Grafonnet, this makes this POC a lot
less useful. To make this more useful, we need at least schemas for the different built-in
panels and datasource targets.
This is actually a second pass for me personally, I played with this almost a year ago and
am currently a bit dissapointed that there was almost no progress in schematizing other
compontents (panels/datasources/plugins) since then. I might be wrong on this end, so
please tell me if I'm missing something here.
Why is this important?
We get very regular questions from Grafana users and customers on different channels what
the best way is to compose dashboards as-code. Grafonnet is currently our best offer we
can give them, but it is horribly out-of-date and undermaintained. With the upstream
schemas at the horizon, this is unlikely going to improve.
With an upstream schema that gets maintained along with Grafana itself, Grafonnet can
offer an up-to-date library that can be maintained with very little effort.