Visualizations: Choose color based on series name

[lukepalmer]

What is this feature?

Allows automatic color selection based on the series name instead of by the series index.

Why do we need this feature?

Some datasources, such as prometheus, may return many series as the result of one query. A particular series may or may not be present in the underlying data at a particular time range. When coloring by series index, this results in color assigned to particular series changing based on the time range, which is confusing and may result in flickering colors when a visualization is set to auto-refresh.

By selecting color based on series name the user can achieve consistent coloring across time ranges and also across multiple unrelated visualizations without manual setup.

Who is this feature for?

Users of prometheus with grafana.

Which issue(s) does this PR fix?:

Fixes #221
Fixes #3250
Fixes #36050

Please check that:

• It works as expected from a user's perspective.
• If this is a pre-GA feature, it is behind a feature toggle.
• The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

[CLAassistant]

All committers have signed the CLA.

[nmarrs]

This addition makes sense to me to add, with a few clean up / slight renaming I think this is good to merge 👍

[nmarrs]

A final few nits, I tried to add them to branch directly but didn't have write rights 😬 this is looking a lot cleaner already!

[nmarrs]

Example dashboard to test this feature out with:

Key	Value
Panel	timeseries @ 10.0.0-pre (971b6aa)
Grafana	10.0.0-pre (971b6aa) // Open Source

Panel debug snapshot dashboard

{
  "panels": [
    {
      "datasource": {
        "type": "grafana",
        "uid": "grafana"
      },
      "fieldConfig": {
        "defaults": {
          "custom": {
            "drawStyle": "line",
            "lineInterpolation": "linear",
            "barAlignment": 0,
            "lineWidth": 1,
            "fillOpacity": 0,
            "gradientMode": "none",
            "spanNulls": false,
            "showPoints": "auto",
            "pointSize": 5,
            "stacking": {
              "mode": "none",
              "group": "A"
            },
            "axisPlacement": "auto",
            "axisLabel": "",
            "axisColorMode": "text",
            "scaleDistribution": {
              "type": "linear"
            },
            "axisCenteredZero": false,
            "hideFrom": {
              "tooltip": false,
              "viz": false,
              "legend": false
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "color": {
            "mode": "palette-classic-by-name"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 13,
        "w": 15,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "options": {
        "tooltip": {
          "mode": "single",
          "sort": "none"
        },
        "legend": {
          "showLegend": true,
          "displayMode": "table",
          "placement": "right",
          "calcs": [
            "first",
            "max",
            "count"
          ]
        }
      },
      "targets": [
        {
          "refId": "A",
          "datasource": {
            "type": "grafana",
            "uid": "grafana"
          },
          "queryType": "snapshot",
          "snapshot": [
            {
              "schema": {
                "refId": "A",
                "fields": [
                  {
                    "name": "time",
                    "type": "time",
                    "typeInfo": {
                      "frame": "time.Time",
                      "nullable": true
                    },
                    "config": {}
                  },
                  {
                    "name": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa123456789123456789123456789123456789",
                    "type": "number",
                    "typeInfo": {
                      "frame": "int64",
                      "nullable": true
                    },
                    "config": {}
                  }
                ]
              },
              "data": {
                "values": [
                  [
                    1679601788,
                    1680724988
                  ],
                  [
                    1256,
                    785
                  ]
                ]
              }
            },
            {
              "schema": {
                "refId": "B",
                "fields": [
                  {
                    "name": "time",
                    "type": "time",
                    "typeInfo": {
                      "frame": "time.Time",
                      "nullable": true
                    },
                    "config": {}
                  },
                  {
                    "name": "lalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalal",
                    "type": "number",
                    "typeInfo": {
                      "frame": "int64",
                      "nullable": true
                    },
                    "config": {}
                  }
                ]
              },
              "data": {
                "values": [
                  [
                    1681156988,
                    1678482188
                  ],
                  [
                    1600,
                    360
                  ]
                ]
              }
            },
            {
              "schema": {
                "refId": "C",
                "fields": [
                  {
                    "name": "time",
                    "type": "time",
                    "typeInfo": {
                      "frame": "time.Time",
                      "nullable": true
                    },
                    "config": {}
                  },
                  {
                    "name": "lalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalal",
                    "type": "number",
                    "typeInfo": {
                      "frame": "int64",
                      "nullable": true
                    },
                    "config": {}
                  }
                ]
              },
              "data": {
                "values": [
                  [
                    1681156988,
                    1678482188
                  ],
                  [
                    300,
                    500
                  ]
                ]
              }
            }
          ]
        }
      ],
      "title": "Reproduced with embedded data",
      "type": "timeseries"
    },
    {
      "gridPos": {
        "h": 7,
        "w": 9,
        "x": 15,
        "y": 0
      },
      "id": 5,
      "options": {
        "content": "<table width=\"100%\">\n    <tr>\n      <th width=\"2%\">Panel</th>\n      <td >timeseries @ 10.0.0-pre (971b6aa6c2)</td>\n    </tr>\n    <tr>\n      <th>Queries</th>\n      <td>A[testdata], B[testdata], C[testdata]</td>\n    </tr>\n    \n    <tr><th>Data</th><td> 3 frames, 6 fields, 6 rows </td></tr>\n    \n    <tr>\n      <th>Grafana</th>\n      <td>10.0.0-pre (971b6aa6c2) // Open Source</td>\n    </tr>\n  </table>",
        "mode": "html"
      },
      "title": "Debug info",
      "type": "text"
    },
    {
      "id": 6,
      "title": "Original Panel JSON",
      "type": "text",
      "gridPos": {
        "h": 13,
        "w": 9,
        "x": 15,
        "y": 7
      },
      "options": {
        "content": "{\n  \"datasource\": {\n    \"uid\": \"PD8C576611E62080A\",\n    \"type\": \"testdata\"\n  },\n  \"fieldConfig\": {\n    \"defaults\": {\n      \"custom\": {\n        \"drawStyle\": \"line\",\n        \"lineInterpolation\": \"linear\",\n        \"barAlignment\": 0,\n        \"lineWidth\": 1,\n        \"fillOpacity\": 0,\n        \"gradientMode\": \"none\",\n        \"spanNulls\": false,\n        \"showPoints\": \"auto\",\n        \"pointSize\": 5,\n        \"stacking\": {\n          \"mode\": \"none\",\n          \"group\": \"A\"\n        },\n        \"axisPlacement\": \"auto\",\n        \"axisLabel\": \"\",\n        \"axisColorMode\": \"text\",\n        \"scaleDistribution\": {\n          \"type\": \"linear\"\n        },\n        \"axisCenteredZero\": false,\n        \"hideFrom\": {\n          \"tooltip\": false,\n          \"viz\": false,\n          \"legend\": false\n        },\n        \"thresholdsStyle\": {\n          \"mode\": \"off\"\n        }\n      },\n      \"color\": {\n        \"mode\": \"palette-classic-by-name\"\n      },\n      \"mappings\": [],\n      \"thresholds\": {\n        \"mode\": \"absolute\",\n        \"steps\": [\n          {\n            \"color\": \"green\",\n            \"value\": null\n          },\n          {\n            \"color\": \"red\",\n            \"value\": 80\n          }\n        ]\n      }\n    },\n    \"overrides\": []\n  },\n  \"gridPos\": {\n    \"h\": 13,\n    \"w\": 21,\n    \"x\": 0,\n    \"y\": 0\n  },\n  \"id\": 19,\n  \"options\": {\n    \"tooltip\": {\n      \"mode\": \"single\",\n      \"sort\": \"none\"\n    },\n    \"legend\": {\n      \"showLegend\": true,\n      \"displayMode\": \"table\",\n      \"placement\": \"right\",\n      \"calcs\": [\n        \"first\",\n        \"max\",\n        \"count\"\n      ]\n    }\n  },\n  \"targets\": [\n    {\n      \"datasource\": {\n        \"type\": \"testdata\",\n        \"uid\": \"PD8C576611E62080A\"\n      },\n      \"csvContent\": \"time, aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa123456789123456789123456789123456789\\n1679601788, 1256\\n1680724988, 785\",\n      \"refId\": \"A\",\n      \"scenarioId\": \"csv_content\"\n    },\n    {\n      \"datasource\": {\n        \"type\": \"testdata\",\n        \"uid\": \"PD8C576611E62080A\"\n      },\n      \"csvContent\": \"time, lalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalal\\n1681156988, 1600\\n1678482188, 360\",\n      \"refId\": \"B\",\n      \"scenarioId\": \"csv_content\"\n    },\n    {\n      \"scenarioId\": \"csv_content\",\n      \"refId\": \"C\",\n      \"datasource\": {\n        \"type\": \"testdata\",\n        \"uid\": \"PD8C576611E62080A\"\n      },\n      \"csvContent\": \"time, lalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalallalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalalalallalalalal\\n1681156988, 300\\n1678482188, 500\",\n      \"hide\": false\n    }\n  ],\n  \"title\": \"Panel Title\",\n  \"type\": \"timeseries\"\n}",
        "mode": "code",
        "code": {
          "language": "json",
          "showLineNumbers": true,
          "showMiniMap": true
        }
      }
    },
    {
      "id": 3,
      "title": "Data from panel above",
      "type": "table",
      "datasource": {
        "type": "datasource",
        "uid": "-- Dashboard --"
      },
      "gridPos": {
        "h": 7,
        "w": 15,
        "x": 0,
        "y": 13
      },
      "options": {
        "showTypeIcons": true
      },
      "targets": [
        {
          "datasource": {
            "type": "datasource",
            "uid": "-- Dashboard --"
          },
          "panelId": 2,
          "withTransforms": true,
          "refId": "A"
        }
      ]
    }
  ],
  "schemaVersion": 37,
  "title": "Debug: Panel Title // 2023-04-12 19:07:12",
  "tags": [
    "debug",
    "debug-timeseries"
  ],
  "time": {
    "from": "1970-01-20T10:14:42.188Z",
    "to": "1970-01-20T10:59:16.988Z"
  }
}

[torkelo]

Let me give this a review in the coming days. I explored something like this very early on but could not get it work well (series colors assigned by name hash) created ugly colors with no control and poor contrast between series as you basically have no control over the colors, a query that returns two series could both get a very similar blue color for example. So interested in how this solves that.

[lukepalmer]

@torkelo Thanks for looking! I agree that it's possible a user could get ugly colors that potentially don't contrast well using this feature.

A notable difference between this feature and other previous attempts I've seen is that it's optional as opposed to changing the grafana default. You can turn on this feature to likely solve a very real problem, but you don't have to.

I'd think that if this feature outputs colors the user doesn't like, what this is partly saying is that the user doesn't like the color palette. I'd claim that's a different issue.

[torkelo]

Just not sure how usable this is going to be.

Might need a custom palette, some are very faint

[lukepalmer]

@torkelo It'd be possible to have the color palette programatically include only colors that contrast with the theme background. Would that be better?

Very interested in other suggestions as well.

[torkelo]

@lukepalmer yes, you can use colorManipulator.getContrastRatio to remove colors that produce to low a contrast ratio for the current theme

[lukepalmer]

@torkelo I implemented that, thanks.
I also tried a version that selected from the colors defined in the theme visualization hues; the usability of the outcome was pretty similar.

[nmarrs]

Looks good from dataviz squad perspective - looks like there are a few merge conflicts due to this PR being merged in before. Once merge conflicts are resolved I'm good with merging this in 🎊

[lukepalmer]

Rebased, thanks for the review @nmarrs

[ShawnTuatara]

The colors match in the time series graph although the tooltip and legend have a single color for all series.

Grafana v10.2.0-59422pre (2569081)

[torkelo]

Maybe we should revert this, when testing I only get similar looking colors that are very hard to differentiate

[lukepalmer]

We really do find this feature useful. I can take a look at the legend if you're willing to wait a week or so. I think that issue is disjoint from how people feel about the color choices working well for them, and there are clearly differing opinions there.

…

On Wed, Sep 6, 2023, 1:36 AM Torkel Ödegaard ***@***.***> wrote: Maybe we should revert this, when testing I only get similar looking colors that are very hard to differentiate — Reply to this email directly, view it on GitHub <#66197 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGESAPTQWEDNEOQE7SYVADXZADWPANCNFSM6AAAAAAWXRY3LQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[brad-alexander]

seems like for the prometheus data source, all fields are named Value and get the same color. Switching Format to 'Heatmap' in the series options seems to have this work correctly.

[xforman2]

I want to use this feature. It is very confusing when the colors change while interacting with time range or if colors for same entity are not the same across the panels in one dashboard. It is crucial to be consistent.

Main problem is that I use mySQL datasource and the logic behind this feature is based on different color for different field name.
Let`s consider Time-series graph with wide format.
This can be achieved in my case by renaming the field name which is the column with numeric values of the query.
I want to visualize CPU usage of users on the server. Every user should be a different series.
Now I can do something like this:

SELECT $__timeGroup(TimeCreated, $__interval, 0) as time, ur.ProcessCount as **User_1**
      FROM UserRecord ur
      WHERE $__timeFilter(TimeCreated) 
      ORDER BY time

SELECT $__timeGroup(TimeCreated, $__interval, 0) as time, ur.CpuUsage as **User_2**
      FROM UserRecord ur
      WHERE $__timeFilter(TimeCreated) 
      ORDER BY time

and so on...
The problem with this is that i cannot control how many users will there be and so I want to it to be dynamic.
When I want it to be dynamic and do it with one query, let`s say:

SELECT $__timeGroup(TimeCreated, $__interval, 0) as time, ur.CpuUsage, **u.Name**
      FROM UserRecord ur
      JOIN User u ON ur.UserID = u.ID
      WHERE $__timeFilter(TimeCreated) 
      ORDER BY time

u.Name is now label and I cannot really control the field name which is powering the logic behind the colors.
My questions are:
Am I somehow able to use this future without statically renaming the field name?
Does some possibly some transformation allow for dynamic field names?