Using alerts on panel not working

[janisbiz]

Hello!

Using alerts on existing panels are not working.

Version(s):

• Grafana v8.5.13 (38d274060d)
• OS DS Plugin 1.2.2
• OS Version 1.3.2

Example panel:

Example alert:

When clicking "Test rule" in "alerts tab" getting:

{
  ...
  "logs": [
    ...
    {
      "message": "Condition[0]: Query Result",
      "data": {
        "fromDataframe": true,
        "series": []                                          
      }
    },
    {
      "message": "Condition: Eval: false, Query Returned No Series (reduced to null/no value)",
      "data": null
    }
  ]
}

[ivanahuckova]

Hello @janisbiz. To help us investigate, could you please provide panel/dashboard JSON and query from query inspector.

[janisbiz]

@ivanahuckova I can provide it only with obscured values (pretty much anything defined in storage, like variable names, sources e.t.c. will not be readable) due to my company policy. If that's fine, let me know.

[ivanahuckova]

Yes it is absolutely fine! Thank you.

[janisbiz]

@ivanahuckova here you go:

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 125,
  "iteration": 1666786114102,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "alert": {
        "alertRuleTags": {},
        "conditions": [
          {
            "evaluator": {
              "params": [
                3
              ],
              "type": "lt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "ALERT",
                "3m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "sum"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "3m",
        "frequency": "1m",
        "handler": 1,
        "name": "OBSCURED",
        "noDataState": "alerting",
        "notifications": []
      },
      "datasource": {
        "type": "OBSCURED",
        "uid": "OBSCURED"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "bars",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "normal"
            },
            "thresholdsStyle": {
              "mode": "area"
            }
          },
          "mappings": [],
          "max": 1,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 0
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 22,
        "x": 0,
        "y": 0
      },
      "id": 13,
      "interval": "1m",
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "8.1.1",
      "targets": [
        {
          "alias": "",
          "bucketAggs": [
            {
              "field": "FIELD",
              "id": "3",
              "settings": {
                "min_doc_count": "1",
                "order": "desc",
                "orderBy": "_term",
                "size": "0"
              },
              "type": "terms"
            },
            {
              "id": "4",
              "settings": {
                "interval": "auto",
                "min_doc_count": "0",
                "trimEdges": "0"
              },
              "type": "date_histogram"
            }
          ],
          "datasource": {
            "type": "OBSCURED",
            "uid": "OBSCURED"
          },
          "format": "table",
          "metrics": [
            {
              "field": "value",
              "id": "1",
              "type": "sum"
            }
          ],
          "query": "KEY1:VALUE1 AND KEY2:\"VALUE2\" AND KEY3:\"VALUE3\"",
          "queryType": "lucene",
          "refId": "ALERT",
          "timeField": "time"
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "op": "lt",
          "value": 3,
          "visible": true
        }
      ],
      "title": "OBSCURED",
      "type": "timeseries"
    }
  ],
  "refresh": "",
  "schemaVersion": 36,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "OBSCURED",
  "uid": "OBSCURED",
  "version": 5,
  "weekStart": ""
}

[maxwellvarner]

I too am experiencing this issue. I am using grafana OSS 9.1.8, but see the same in 9.2.2 with the opensearch plugin version 2.0.3.

It appears from my grafana container logs that the alert queries are not respecting some of the values configured in the datasource. I expect see the index of [cds_proxies-]YYYY.MM.DD being used as it is configured in the datasource.

but when I check the logs after running the queries from the Alerts tab I see that it is attempting to use a different index

Output in grafana logs:
logger=plugin.grafana-opensearch-datasource t=2022-11-02T16:25:11.322770919Z level=info msg="Creating new client" PPLindex=logstash-* indices=logstash-2022.11.02 timeField=@timestamp version=5.6.0

These values from the datasource appear to be respected when I simply create a new panel for viewing only purposes.

[maxwellvarner]

I'll add that the index it has chosen to use is not one that I even appear to have when I check all my indices configured in Kibana. It would appear that index is being determined by something else either in Grafana or in the plugin itself.

[svennergr]

Thanks @maxwellvarner. I see that you are using the datasource to query an ElasticSearch. Could you provide the exact version of that ElasticSearch?

@janisbiz Are you using the datasource to query ElasticSearch as well, or any other database? Would be great to get that version too.

Thanks!

[svennergr]

@maxwellvarner / @janisbiz

Sorry for the next ping. I discovered a root cause and will get a fix out.

[maxwellvarner]

I've attempted this against 2 different versions of Elasticsearch; versions 6.8.23 and 7.9

I got this information by running
curl -u <username> -XGET 'https://<domain>:9200

[janisbiz]

@svennergr not sure if it is still relevant, I am using OpenSearch 1.3.2

[svennergr]

@janisbiz @maxwellvarner OpenSearch v1.2.4 and OpenSearch v2.0.4 have been released with a fix of this issue.

[janisbiz]

@svennergr just updated. The previous error is gone, but alerts are still not working. It seems like datasets are always returning 0. Will take a look deeper to understand what's the problem.

@maxwellvarner please let us know how it is for you.

[svennergr]

@janisbiz tested in Grafana 8.5.13 and it was working for me:

[janisbiz]

@svennergr this is what I cam having for alert - with sum:

If I change to count - it is all good. Thus not sure why 🤔 I guess sum is not summing actual metric value maybe.

Getting:

{
  "firing": true,
  "state": "pending",
  "conditionEvals": "true = true",
  "timeMs": "527.946ms",
  "matches": [
    "OBFUSCATED"
  ],
  "logs": [
    {
      "message": "Condition[0]: Query",
      "data": "OBFUSCATED"
    },
    {
      "message": "Condition[0]: Query Result",
      "data": {
        "fromDataframe": true,
        "series": [
          {
            "name": " {OBFUSCATED=OBFUSCATED}",
            "points": [
              [
                0,
                1667572980000
              ],
              [
                0,
                1667573040000
              ],
              [
                0,
                1667573100000
              ],
              [
                0,
                1667573160000
              ]
            ],
            "tags": {
              "OBFUSCATED": "OBFUSCATED"
            }
          },
          {
            "name": " {OBFUSCATED=OBFUSCATED}",
            "points": [
              [
                0,
                1667572980000
              ],
              [
                0,
                1667573040000
              ],
              [
                0,
                1667573100000
              ],
              [
                0,
                1667573160000
              ]
            ],
            "tags": {
              "OBFUSCATED": "OBFUSCATED"
            }
          },
          {
            "name": " {OBFUSCATED=OBFUSCATED}",
            "points": [
              [
                0,
                1667572980000
              ],
              [
                0,
                1667573040000
              ],
              [
                0,
                1667573100000
              ],
              [
                0,
                1667573160000
              ]
            ],
            "tags": {
              "OBFUSCATED": "OBFUSCATED"
            }
          }
        ]
      }
    },
    {
      "message": "Condition[0]: Eval: true, Metric:  {OBFUSCATED}, Value: 0.000",
      "data": null
    },
    {
      "message": "Condition[0]: Eval: true, Metric:  {OBFUSCATED}, Value: 0.000",
      "data": null
    },
    {
      "message": "Condition[0]: Eval: true, Metric:  {OBFUSCATED}, Value: 0.000",
      "data": null
    }
  ]
}

[maxwellvarner]

This appears to be working for me against both versions of elasticsearch mentioned above running grafana v9.1.8.

sum() appears to be returning the correct value for me. If I'm understanding what count() is going to return then that would be the number of records returned by the query; whereas sum() will total up all the counts returned for each row based on your grouping.

In the example below where you see the reduce expressions; it is showing that I returned 601 rows (grouped records) and the varying count in each of those rows totaled up by the sum() function is 1997. Sum being what I want as that indicates to me I have had 1997 individual requests return a 401 status code for that environment and type in a 10 min window. I don't let it try and pull stats up to now and instead use now-5m to ensure that I don't query for records that have not made there way to elasticsearch yet from my logstash cluster.

Alert Rule - Results

Panel Results

[janisbiz]

@svennergr are you trying with elastic or OS? As I am using OS, it seems to be not working.

[janisbiz]

So, I re-created panel with same query, same alert (litereally everything the same) and it works now. I guess something was corrupted on my panel. 🤷‍♂️

Anyway - all good now. Thanks 🎉