Transformations: Add frame source picker to allow transforming annotations #77842

leeoniya · 2023-11-08T02:50:40Z

this ~~begins~~ continues 😅 the story of transformations being able to target specific DataFrames rather than always operating on the full set of data.series frames. one recurring request is to be able hide and reorder specific fields in the Exemplar tooltip. since exemplars live in Annotations and not Series data, they have been inaccessible to transformations...until now!

this PR adds a selector for the DataFrame source, which allows us to use the Organize fields transform to manipulate annotations frames.

one issue with this approach, though, is that Organize fields always removes fields when you "hide" them. this means you cannot hide fields that are necessary for rendering. for exemplars you cannot hide Time and le or Value. to solve this we'd like to enhance Organize fields to allow it to hide without removal by exposing granular control over hideFrom.viz, hideFrom.tooltip, and hideFrom.legend field configs. if all of them are enabled, then the field will be removed, otherwise we will keep the field but set the hideFrom configs.

in the future, we would like to expand this frame selection to be more powerful by adding FrameMatchers that can choose frames containing specific fields, or frames coming from specific data sources.

we should also follow up here and allow field matchers and overrides to get access to Annotation frames so that additional things can be tweaked in them, such as units, decimals, etc.

TODO:

the source selector should probably move up into the header? it's definitely too huge and almost always necessary to prominently show at the top of every transform editor

transform-annotations.json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 836,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "grafana-testdata-datasource",
        "uid": "PD8C576611E62080A"
      },
      "fieldConfig": {
        "defaults": {
          "custom": {
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "scaleDistribution": {
              "type": "linear"
            }
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 25,
        "w": 19,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "maxDataPoints": 100,
      "options": {
        "calculate": false,
        "cellGap": 1,
        "color": {
          "exponent": 0.5,
          "fill": "dark-orange",
          "mode": "scheme",
          "reverse": false,
          "scale": "exponential",
          "scheme": "Spectral",
          "steps": 64
        },
        "exemplars": {
          "color": "rgba(255,0,255,0.7)"
        },
        "filterValues": {
          "le": 1e-9
        },
        "legend": {
          "show": true
        },
        "rowsFrame": {
          "layout": "auto"
        },
        "tooltip": {
          "show": true,
          "yHistogram": false
        },
        "yAxis": {
          "axisPlacement": "left",
          "reverse": false
        }
      },
      "pluginVersion": "10.3.0-pre",
      "targets": [
        {
          "datasource": {
            "type": "grafana-testdata-datasource",
            "uid": "PD8C576611E62080A"
          },
          "rawFrameContent": "[\n  {\n    \"schema\": {\n      \"refId\": \"A\",\n      \"meta\": {\n        \"type\": \"heatmap-rows\",\n        \"typeVersion\": [\n          0,\n          1\n        ],\n        \"custom\": {\n          \"resultType\": \"matrix\"\n        }\n      },\n      \"name\": \"0.005\",\n      \"fields\": [\n        {\n          \"name\": \"Time\",\n          \"type\": \"time\",\n          \"typeInfo\": {\n            \"frame\": \"time.Time\"\n          },\n          \"config\": {\n            \"interval\": 60000\n          }\n        },\n        {\n          \"name\": \"0.005\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"0.005\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"0.005\"\n          }\n        },\n        {\n          \"name\": \"0.01\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"0.01\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"0.01\"\n          }\n        },\n        {\n          \"name\": \"0.025\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"0.025\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"0.025\"\n          }\n        },\n        {\n          \"name\": \"0.05\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"0.05\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"0.05\"\n          }\n        },\n        {\n          \"name\": \"0.1\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"0.1\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"0.1\"\n          }\n        },\n        {\n          \"name\": \"0.25\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"0.25\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"0.25\"\n          }\n        },\n        {\n          \"name\": \"0.5\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"0.5\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"0.5\"\n          }\n        },\n        {\n          \"name\": \"1.0\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"1.0\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"1.0\"\n          }\n        },\n        {\n          \"name\": \"2.5\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"2.5\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"2.5\"\n          }\n        },\n        {\n          \"name\": \"5.0\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"5.0\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"5.0\"\n          }\n        },\n        {\n          \"name\": \"10.0\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"10.0\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"10.0\"\n          }\n        },\n        {\n          \"name\": \"25.0\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"25.0\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"25.0\"\n          }\n        },\n        {\n          \"name\": \"50.0\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"50.0\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"50.0\"\n          }\n        },\n        {\n          \"name\": \"100.0\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"100.0\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"100.0\"\n          }\n        },\n        {\n          \"name\": \"+Inf\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"labels\": {\n            \"le\": \"+Inf\"\n          },\n          \"config\": {\n            \"displayNameFromDS\": \"+Inf\"\n          }\n        }\n      ]\n    },\n    \"data\": {\n      \"values\": [\n        [\n          1699408500000,\n          1699408560000,\n          1699408620000,\n          1699408680000,\n          1699408740000,\n          1699408800000,\n          1699408860000,\n          1699408920000,\n          1699408980000,\n          1699409040000,\n          1699409100000,\n          1699409160000,\n          1699409220000,\n          1699409280000,\n          1699409340000,\n          1699409400000,\n          1699409460000,\n          1699409520000,\n          1699409580000,\n          1699409640000,\n          1699409700000,\n          1699409760000,\n          1699409820000,\n          1699409880000\n        ],\n        [\n          125.86666666666667,\n          123.73333333333333,\n          124.80000000000001,\n          124.80000000000001,\n          124.80000000000001,\n          126.93333333333334,\n          125.86666666666667,\n          124.80000000000001,\n          125.86666666666667,\n          127.99715561876403,\n          126.93333333333334,\n          128,\n          126.93333333333334,\n          124.80000000000001,\n          126.93333333333334,\n          126.93333333333334,\n          124.80000000000001,\n          126.93333333333334,\n          124.80000000000001,\n          124.80000000000001,\n          111.76815384615385,\n          87.94351621463927,\n          82.4377482072613,\n          77.92061131919219\n        ],\n        [\n          9.599999999999994,\n          11.733333333333334,\n          10.666666666666657,\n          10.666666666666657,\n          10.666666666666657,\n          8.533333333333331,\n          9.599999999999994,\n          10.666666666666657,\n          9.599999999999994,\n          7.466500744427918,\n          8.533333333333331,\n          7.466666666666669,\n          8.533333333333331,\n          10.666666666666657,\n          8.533333333333331,\n          8.533333333333331,\n          10.666666666666657,\n          8.533333333333331,\n          10.666666666666657,\n          10.666666666666657,\n          9.959538461538457,\n          5.6464211100863935,\n          13.551410664207339,\n          22.58568444034556\n        ],\n        [\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          15.999644452345507,\n          16,\n          16,\n          16,\n          16,\n          16,\n          14.933333333333337,\n          14.933333333333337,\n          14.933333333333337,\n          14.933333333333337,\n          16,\n          14.385999999999996,\n          10.38600000000001,\n          6.775705332103669,\n          2.2585684440345517\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          1.0666666666666629,\n          1.0666666666666629,\n          1.0666666666666629,\n          1.0666666666666629,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          15.999644452345507,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          16,\n          14.385999999999996,\n          10.385999999999996,\n          9.034273776138235,\n          9.034273776138235\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          -2.842170943040401e-14,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ],\n        [\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0,\n          0\n        ]\n      ]\n    }\n  },\n  {\n    \"schema\": {\n      \"refId\": \"A\",\n      \"meta\": {\n        \"typeVersion\": [\n          0,\n          0\n        ],\n        \"custom\": {\n          \"resultType\": \"exemplar\"\n        },\n        \"dataTopic\": \"annotations\"\n      },\n      \"name\": \"exemplar\",\n      \"fields\": [\n        {\n          \"name\": \"Time\",\n          \"type\": \"time\",\n          \"typeInfo\": {\n            \"frame\": \"time.Time\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"Value\",\n          \"type\": \"number\",\n          \"typeInfo\": {\n            \"frame\": \"float64\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"__name__\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"cluster\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"container\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"instance\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"job\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"le\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"method\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"namespace\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"pod\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"prometheus_job\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"route\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"status_code\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        },\n        {\n          \"name\": \"traceID\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {\n            \"links\": [\n              {\n                \"title\": \"Query with grafanacloud-nativehistogram-traces\",\n                \"url\": \"\",\n                \"internal\": {\n                  \"query\": {\n                    \"query\": \"${__value.raw}\",\n                    \"queryType\": \"traceql\"\n                  },\n                  \"datasourceUid\": \"grafanacloud-traces\",\n                  \"datasourceName\": \"grafanacloud-nativehistogram-traces\"\n                }\n              }\n            ]\n          }\n        },\n        {\n          \"name\": \"ws\",\n          \"type\": \"string\",\n          \"typeInfo\": {\n            \"frame\": \"string\"\n          },\n          \"config\": {}\n        }\n      ]\n    },\n    \"data\": {\n      \"values\": [\n        [\n          1699408545523,\n          1699408605524,\n          1699408665522,\n          1699408725520,\n          1699408785522,\n          1699408845523,\n          1699408905522,\n          1699408965520,\n          1699409025524,\n          1699409085523,\n          1699409145521,\n          1699409205522,\n          1699409265522,\n          1699409325521,\n          1699409385532,\n          1699409445521,\n          1699409505520,\n          1699409565520,\n          1699409625522,\n          1699409789001,\n          1699409848061\n        ],\n        [\n          0.020431832,\n          0.02090233,\n          0.019722001,\n          0.016970139,\n          0.019284203,\n          0.020572653,\n          0.019318701,\n          0.017329633,\n          0.021806494,\n          0.018097813,\n          0.01909374,\n          0.019286653,\n          0.018773903,\n          0.017577304,\n          0.029428525,\n          0.018254914,\n          0.017515206,\n          0.01797259,\n          0.019303786,\n          0.005466718,\n          0.000586483\n        ],\n        [\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\",\n          \"cortex_request_duration_seconds_bucket\"\n        ],\n        [\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\",\n          \"dev-us-central-0\"\n        ],\n        [\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\",\n          \"compactor\"\n        ],\n        [\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\",\n          \"compactor-0:compactor:http-metrics\"\n        ],\n        [\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\",\n          \"mimir-dev-12/compactor\"\n        ],\n        [\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.05\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.025\",\n          \"0.01\",\n          \"0.005\"\n        ],\n        [\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\",\n          \"GET\"\n        ],\n        [\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\",\n          \"mimir-dev-12\"\n        ],\n        [\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\",\n          \"compactor-0\"\n        ],\n        [\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\",\n          \"nativehistogram\"\n        ],\n        [\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"debug_pprof\",\n          \"metrics\",\n          \"debug_pprof\"\n        ],\n        [\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\",\n          \"200\"\n        ],\n        [\n          \"71cf1dfd8f3d01e2\",\n          \"5de8349e39986ecb\",\n          \"365d8c1a9590606d\",\n          \"74bb127c7f30e211\",\n          \"4f4da590eb49335d\",\n          \"3b7a072cf88636a5\",\n          \"094d3f4421f87af4\",\n          \"7e5444a2ac635799\",\n          \"7404c197005a87e5\",\n          \"6412591a61b76b6a\",\n          \"7131e4c73ce16131\",\n          \"0229ed3905458ddb\",\n          \"5fb2200e69616856\",\n          \"709f2e40d15fc13f\",\n          \"6c6e98fa58a6729d\",\n          \"5a4f82574df2e6c4\",\n          \"1981b3f623d27ceb\",\n          \"0d5c6e75a136be98\",\n          \"4d3e66a003190452\",\n          \"384dac0442da6e54\",\n          \"728ceedb3647a28d\"\n        ],\n        [\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\",\n          \"false\"\n        ]\n      ]\n    }\n  }\n]",
          "refId": "A",
          "scenarioId": "raw_frame"
        }
      ],
      "title": "Panel Title",
      "transformations": [],
      "type": "heatmap"
    }
  ],
  "refresh": "",
  "schemaVersion": 39,
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "2023-11-08T01:53:57.047Z",
    "to": "2023-11-08T02:18:04.478Z"
  },
  "timepicker": {},
  "timezone": "",
  "title": "transform-annotations",
  "uid": "e58aa9d0-663e-441a-b30f-5f6c97c8bf62",
  "version": 5,
  "weekStart": ""
}

Peek.2023-11-07.23-00.mp4

…tions

leeoniya · 2023-11-08T02:52:22Z

public/app/features/dashboard/components/TransformationsEditor/TransformationFilter.tsx

  onChange: (index: number, config: DataTransformerConfig) => void;
 }

 export const TransformationFilter = ({ index, data, config, onChange }: TransformationFilterProps) => {
  const styles = useStyles2(getStyles);
  const context = useMemo(() => {
    // eslint-disable-next-line
-    return { data } as StandardEditorContext<unknown>;
+    return { data: data.series } as StandardEditorContext<unknown>;


not sure what TransformationFilter does, but this is probably not sufficient to cover the new functionality?

heh @ryantxu i totally forgot this existed. it only shows up in the toolbar when there's more than one dataframe in series 🤣

i'm gonna stick this new source selector in there for now.

public/app/features/dashboard/components/TransformationsEditor/TransformationOperationRow.tsx

leeoniya · 2023-11-08T02:56:36Z

public/app/features/query/state/PanelQueryRunner.ts

+    let seriesTransformations = transformations.filter((t) => t.source == null || t.source === DataFramesSource.Series);
+    let annotationsTransformations = transformations.filter((t) => t.source === DataFramesSource.Annotations);
+
+    let seriesStream = transformDataFrame(seriesTransformations, data.series, ctx);
+    let annotationsStream = transformDataFrame(annotationsTransformations, data.annotations ?? [], ctx);
+
+    return merge(seriesStream, annotationsStream).pipe(
+      map((frames) => {
+        let isAnnotations = frames.some((f) => f.meta?.dataTopic === DataTopic.Annotations);
+        let transformed = isAnnotations ? { annotations: frames } : { series: frames };
+        return { ...data, ...transformed };
+      }),


not super excited by this. we can probably get away with a single stream if transformDataFrame handled this logic internally and accepted + returned TransformationData, but would need an fn signature change.

github-actions · 2023-11-17T01:16:48Z

⚠️ Possible breaking changes (md version)

grafana-data

Removals

DataTopic.Annotations
_{/home/runner/work/grafana/grafana/base/grafana-data/dist/index.d.ts}

Annotations = "annotations"

DataTopic.AlertStates
_{/home/runner/work/grafana/grafana/base/grafana-data/dist/index.d.ts}

AlertStates = "alertStates"

Console output
Read our guideline

dprokop · 2023-11-23T15:58:43Z

@leeoniya I love the entire idea. But seems like this is bult for exemplars only which already come as a DataFrame with the topic set.

In the current dashboard case, the annotation query results are not decorated with the topic, so the transformation applied to i.e. Grafana annotations doesn't work 😿

Screen.Recording.2023-11-23.at.16.40.56.mov

We are missing data topic set on annotations frame https://github.com/grafana/grafana/blob/leeoniya/transform-annotations/public/app/features/query/state/mergePanelAndDashData.ts#L21 which makes the following incorrect https://github.com/grafana/grafana/blob/leeoniya/transform-annotations/public/app/features/query/state/PanelQueryRunner.ts#L234 is incorrect.

On top of the above:

Think this may deserve a feature flag.
UI - now that transformations can be applied to different topics I think we need an improvement to filtering and some indication which transformations apply to what.

dprokop · 2023-11-23T15:59:24Z

@leeoniya please let me know if/when this is merged, we need to apply appropriate updates in Scenes for this to be supported.

leeoniya · 2023-12-13T02:05:28Z

We are missing data topic set on annotations frame

nice, thanks!

added the DataTopic tagging in 6ed4ade and works as expected now:

Peek.2023-12-12.20-03.mp4

leeoniya · 2023-12-13T02:58:31Z

@dprokop

On top of the above:

Think this may deserve a feature flag.

UI - now that transformations can be applied to different topics I think we need an improvement to filtering and some indication which transformations apply to what.

RE 1: i'm not sure about FF since this is a pretty hidden feature that wont be known or used until we blog about it or rework the UI to surface it better. but it will be immediately useful for the users and purpose stated in #77840.

RE 2: similar to above, i think the UI/filtering changes to make this perfect could take quite some time (figuring out which transformations and options can apply to what wont be quick) and i'd rather get this merged and work out the UI later as time permits instead of having this blocked on UI polish. i mainly want to decide now whether we think this is generally a good feature to add, and get the JSON save model correct to avoid writing migrations later. the original external contribution PR was back in Aug, so i'd prefer if we finally got something useful in place to accommodate that need.

@ryantxu, the saved config currently looks like below, but since dataTopic is part of the frame, should we just roll this into a compound filter with a new byTopic matcherId (in addition to byRefId), instead if it living outside of filter but still acting as a filter?

something like

"filters": [
  {
    "id": "byRefId",
    "options": ""
  },
  {
    "id": "byTopic",
    "options": "annotation"
  }
],

thinking of a future where we can do more complex filters/matchers using something like https://jsonlogic.com/ or even better variants like https://github.com/leeoniya/uExpr ;)

dprokop · 2023-12-14T15:19:49Z

@leeoniya - I'm fine with pushing this forward without polishing the UI. But please make sure we don't forget about this. Unpolished UX may be OK for early adopters, but I'm sure people will discover this feature and eventually struggle to understand the UI. This is just adding complexity to a UI that's already neglected.

But other than that I think this is an excellent addition and we just make sure people can understand what transforms are applied to which slice of the data.

Ref this comment:

@ryantxu, the saved config currently looks like below, but since dataTopic is part of the frame, should we just roll this into a compound filter with a new byTopic matcherId (in addition to byRefId), instead if it living outside of filter but still acting as a filter?

I think this should work and that's how I would expect this to work. But - we would not be able to config transformations for series right? I mean we could but we currently have a linear UI for transformations and such a configured filter would create a very weird (if readable at all) UI. Such a filter creates two paths for transformations config, one for data, and another for annotations. So my earlier comment on the UX becomes even more important I think.

…notations

ryantxu

LGTM -- i cleaned up the UX a bit @dprokop

leeoniya · 2024-01-03T05:30:21Z

k, let's merge this and we'll tackle combining the matchers later, since it'll require changes to more places that have frame / field matchers.

Transformations: Add frame source picker to allow transforming annota…

0c40362

…tions

leeoniya added add to changelog area/transformations no-backport Skip backport of PR add to what's new labels Nov 8, 2023

leeoniya added this to the 10.3.x milestone Nov 8, 2023

leeoniya requested review from torkelo, ryantxu and dprokop November 8, 2023 02:50

leeoniya self-assigned this Nov 8, 2023

leeoniya requested a review from a team as a code owner November 8, 2023 02:50

leeoniya requested a review from a team November 8, 2023 02:50

leeoniya requested review from a team as code owners November 8, 2023 02:50

leeoniya requested review from axelavargas, oscarkilhed, baldm0mma, joshhunt, JoaoSilvaGrafana and leventebalogh and removed request for a team November 8, 2023 02:50

grafana-pr-automation bot added the area/frontend label Nov 8, 2023

leeoniya commented Nov 8, 2023

View reviewed changes

public/app/features/dashboard/components/TransformationsEditor/TransformationOperationRow.tsx Outdated Show resolved Hide resolved

leeoniya commented Nov 8, 2023

View reviewed changes

leeoniya added 2 commits November 7, 2023 21:01

simplify

dd4ba6b

lint

c5af02d

leeoniya added 3 commits November 16, 2023 19:00

add alertStates DataTopic

1d3508f

Merge branch 'main' into leeoniya/transform-annotations

a66378a

more alertStates cue

6f9974e

github-actions bot added the levitate breaking change A label indicating a breaking change and assigned by Levitate. label Nov 17, 2023

leeoniya added 3 commits December 12, 2023 19:32

tmp

a361795

Merge branch 'main' into leeoniya/transform-annotations

d840130

restore

aa916bc

tag annotations frames with DataTopic

6ed4ade

leeoniya force-pushed the leeoniya/transform-annotations branch from 82bd75d to 6ed4ade Compare December 13, 2023 02:26

ryantxu added 2 commits January 1, 2024 15:23

Merge remote-tracking branch 'origin/main' into leeoniya/transform-an…

1c4d3a3

…notations

use emotion css

d90e259

leeoniya requested a review from grafanabot as a code owner January 2, 2024 00:01

ryantxu approved these changes Jan 2, 2024

View reviewed changes

Merge branch 'main' into leeoniya/transform-annotations

9f78fb8

leeoniya merged commit fb79be4 into main Jan 3, 2024
19 of 20 checks passed

leeoniya deleted the leeoniya/transform-annotations branch January 3, 2024 05:33

dprokop mentioned this pull request Jan 19, 2024

Transformations: Allow transforming selected data topic grafana/scenes#537

Closed

summerwollin modified the milestones: 10.3.x, 10.3.0 Jan 22, 2024

This was referenced Jan 25, 2024

Transformations: Disabling transformation is causing "No Data" to appear despite data existing #80503

Closed

Exemplars: Add exemplar tooltip config #73350

Closed

gtk-grafana mentioned this pull request Feb 15, 2024

Prometheus: Lengthy exemplar modals #82810

Closed

dprokop mentioned this pull request Mar 20, 2024

DashboardScene: Allow transforming data by topic #84819

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transformations: Add frame source picker to allow transforming annotations #77842

Transformations: Add frame source picker to allow transforming annotations #77842

leeoniya commented Nov 8, 2023 •

edited

leeoniya Nov 8, 2023

leeoniya Nov 8, 2023 •

edited

leeoniya Nov 8, 2023

github-actions bot commented Nov 17, 2023 •

edited

dprokop commented Nov 23, 2023 •

edited

dprokop commented Nov 23, 2023

leeoniya commented Dec 13, 2023 •

edited

leeoniya commented Dec 13, 2023 •

edited

dprokop commented Dec 14, 2023

ryantxu left a comment

leeoniya commented Jan 3, 2024

Transformations: Add frame source picker to allow transforming annotations #77842

Transformations: Add frame source picker to allow transforming annotations #77842

Conversation

leeoniya commented Nov 8, 2023 • edited

leeoniya Nov 8, 2023

Choose a reason for hiding this comment

leeoniya Nov 8, 2023 • edited

Choose a reason for hiding this comment

leeoniya Nov 8, 2023

Choose a reason for hiding this comment

github-actions bot commented Nov 17, 2023 • edited

grafana-data

Removals

dprokop commented Nov 23, 2023 • edited

dprokop commented Nov 23, 2023

leeoniya commented Dec 13, 2023 • edited

leeoniya commented Dec 13, 2023 • edited

dprokop commented Dec 14, 2023

ryantxu left a comment

Choose a reason for hiding this comment

leeoniya commented Jan 3, 2024

leeoniya commented Nov 8, 2023 •

edited

leeoniya Nov 8, 2023 •

edited

github-actions bot commented Nov 17, 2023 •

edited

dprokop commented Nov 23, 2023 •

edited

leeoniya commented Dec 13, 2023 •

edited

leeoniya commented Dec 13, 2023 •

edited