Series order should be guaranteed by the SDK

[srclosson]

What happened: This bug is probably debatable... So IMHO

Many plugins don't order the series, and many customer endpoints do not return series in the same order either. When data arrives at the panel, it's confusing when series change order because the "green" trend is always changing, or if you are viewing data in the table panel, the series, or the order of the fields in the series will change.

What you expected to happen:

The SDK should guarantee some sort of ordering in the series it's being given. In the case of timeseries data, if panels are always looking for the "time" field to be the first series or "column", then the SDK should ensure this ordering exists.

From a users perspective, my "green" trend should always stay green.

Alternatively, if multiple queries are used, then the order of those queries in the editor should be retained.
(Ie: if you have a refID of "A", "B", and "C", The "A" query results should be ordered first.

How to reproduce it (as minimally and precisely as possible):

I used the splunk plugin, and this query. Any panel will show the issue:

The three queries were:

| mstats avg(_value) prestats=true WHERE metric_name="cpu.idle.value" AND index="grafana_metrics_index" span=10s
| timechart avg(_value) AS CPUIdle span=60s
| fields - _span

| mstats avg(_value) prestats=true WHERE metric_name="cpu.system.value" AND index="grafana_metrics_index" span=10s
| timechart avg(_value) AS CPUSystem span=60s
| fields - _span

| mstats avg(_value) prestats=true WHERE metric_name="cpu.user.value" AND index="grafana_metrics_index" span=10s
| timechart avg(_value) AS CPUUser span=60s
| fields - _span

Environment:

• SDK version: 0.91.0
• Grafana version: Grafana 8.0 (https://g80-beta.grafana.net/d/bmTVvMLGk/splunk-alerting?orgId=1)
• Go version: Darwin (Big Sur) Beta 11.5

[kminehart]

Wouldn't this be sort of a problem with how you might be processing results before putting them into a data frame?

If I get a response like this from some serivce,

b := `{
    "results_A": [...],
    "results_B": [...]
}`

And if you unmarshal that as a map,

m := map[string]interface{}

json.Unmarshal(m, b)

and then loop over those results:

for k, v := range m {
    // use the map key `k` as the name of the field
    data.NewFrame("frame", data.NewField(k, v...))
}

you're going to end up with unordered results because maps are unordered in Go, and looping over them can yield a different result each time.

[elliotdobson]

We just recently updated from Grafana v7.3.7 to v8.0.1 and noticed that this has started happening on our panels that are using CloudWatch as their datasource.

Alternatively, if multiple queries are used, then the order of those queries in the editor should be retained.
(Ie: if you have a refID of "A", "B", and "C", The "A" query results should be ordered first.

I'm fairly confident that this is how it was in v7.3.7. Now in v8.0.1 it's just random.

We didn't test any version in between the two that I've mentioned so not sure what version it started happening in exactly.

[JesperCph]

@kxwsd

First noticed this issued from version 8.0.0. Also using the CloudWatch datasource. For the moment it renders most of my dashboards worthless.

I’m considering rolling back to version 7.3.8.

[kminehart]

@kxesd @JesperCph This might be an isssue specifically in the CloudWatch datasource since it's shipped with Grafana. I would open an issue in github.com/grafana/grafana

[leehuk]

@kminehart This was opened against the cloudwatch datasource and closed as an SDK issue: grafana/grafana#35944

[hammond756]

My issue on the Grafana repo was closed because it is marked as a duplicate of this issue. Will this be be picked up as part of the sdk?

[sunker]

We now have three issues related to the CloudWatch plugin that I think are symptoms of this problem.
grafana/grafana#40011 (thanks @hammond756)
grafana/grafana#41773
grafana/grafana#41303

It's not the order of the timeseries in one given query that is non-deterministic - it's the order of the returned queries.

My understanding is that the SDK should guarantee that order of refIds in backend.QueryDataResponse should be the same as the order of refIds in backend.QueryDataRequest. This does not seem to be the case. I can reproduce this locally using the CloudWatch plugin, but also using Prometheus and CloudMonitoring. It seems like the order is incorrect every tenth time or so.

See this example using CloudWatch.

cc @grafana/plugins-platform-backend