[Bug] Query using template variable with Multi-value enabled returns no data when 1 entry is selected #7205

Closed
kohpinngee opened this Issue Jan 10, 2017 · 27 comments

Projects

None yet

5 participants

@kohpinngee
kohpinngee commented Jan 10, 2017 edited
  • What Grafana version are you using?
    4.0.1 viewed on Firefox 45.5.1
  • What datasource are you using?
    graphite
  • What OS are you running grafana on?
    Ubuntu 14.04
  • What did you do?
    I setup a template with "Multi-value" checked.
    Then create a graph using the template variable as part of the name.
    If I check 2 or more entries, grafana generates graphite query payload like "a.{b1,b2}.c"
    But if I check only 1 entry, grafana generates graphite query payload like "a.{b1}.c", and graphite returns no data.
  • What was the expected result?
    data should return from graphite.
  • What happened instead?
    With 1 entry checked, grafana should generates graphite query payload should look like "a.b1.c" - without the curly bracket.
@kohpinngee kohpinngee changed the title from [Bug] to [Bug] Query using template variable with Multi-value enabled returns no data when 1 entry is selected Jan 10, 2017
@torkelo
Member
torkelo commented Jan 10, 2017

a.{b1}.c is a valid query and returns data for me. Very strange. This is how it has worked in Grafana (when using multi variable) for ever, and no one has had an issue with it. So I am not sure what is going on.

What version of graphite are you using?

@kohpinngee

Interestingly. In edit mode, if I select 1 entry a.{b1}.c gets sent as before with no data returned. But, if I then save the Panel and press F5, I see that a.b1.c gets sent and now data returns. Not sure if you reproduce it. FYI, I am using Firefox 45.5.1

@torkelo
Member
torkelo commented Jan 12, 2017 edited

So a query like this works for me:

movingAverage(groupByNode(apps.{backend}.{backend_01}.counters.requests.count, 2, 'sum'), 5)

Must be something wrong with your graphite server.

@torkelo torkelo closed this Jan 12, 2017
@torkelo
Member
torkelo commented Jan 12, 2017

Can't understand what though, sorry for closing. But this is how templating has worked for a very long time with no issues. Can't understand why it's causing an issue for your graphite server.

@ps20renar

Some Details from my side i hope ist the same.
Template edit Window of an imported Dashbord with Initial values.
If i want to change something in this view grafana promted there are no datasource selected. IF i choose one, after that the filter variables are no working anymore:
$gpfsMetrics -> query: metrics(gpfs_.*)
tag_names(gpfs_rq_blk_current) this is working.
tag_names($gpfsMetrics) this don't.
I think the Substitution of this stopped after one attemd.
regards Renar

@ps20renar

Datasource is TSDB 2.2,
Firefox 45.1

@torkelo
Member
torkelo commented Jan 12, 2017 edited

does metrics(gpfs_.*) return any values?

@ps20renar
ps20renar commented Jan 13, 2017 edited

Hallo Torkelo,
let me summarize whats going on here.

  1. Import a Dashbord
    Here came an error Window a.metricFindQuery is not a function
  2. The preview of values Show values.
  3. If i want to update these Template edit window, there is a new Error Message
    (Templating Template vaiables could not be initialized: a. metricFindQuery is not a function) after i add the Datasource entry.
  4. But if see the new metatdata entries.
    (yes the metrics(gpfs_.*) has values.
    fail-01
    fail-02
    fail-03
    fail-04
    fail-05
@torkelo
Member
torkelo commented Jan 13, 2017

The dashboard you import seems to broken, seems to be missing a data source on the template variable

@ps7voli
ps7voli commented Jan 13, 2017

Hallo Torkelo,

i'm a colleague of Renar.

The imported dashboard have no entries in datasource,
but hard-codet entries by the templates.
If i modify the dashboard-json-file ("datasource": "SASGrid",)
the templating-error: "Templating variables could not be initialized: a.metricFindQuery is not a function"
is gone.

If we use in a template-query a another template-variable,
e.g. tag_names($gpfsMetrics), it doesn't work. (grafana 4.x.x)
There is no problem in grafana 3.x.x

For testing purpose

  1. I created a new dashboard.

  2. I created the templating-variable "$Metric" with the query "metrics(gpfs_.*)" and the datasource "SASGrid".
    -> there are values e.g. "gpfs_fs_disks"

  3. I created the templating-variable "$NameHard" with the query "tag_names(gpfs_fs_disks)" and the datasource "SASGrid".
    -> there are values e.g. "gpfs_cluster_name"

  4. I created the templating-variable "$Names" with the query "tag_names($Metric)" and the datasource "SASGrid".
    -> there are no values ("None")

beschreibung01
beschreibung02
beschreibung03
beschreibung04
beschreibung05

@torkelo
Member
torkelo commented Jan 13, 2017

Can you open chrome dev tools and inspect the network request being made when you refresh (hit update button) the Names variable? What request is sent to OpenTSDB?

@torkelo
Member
torkelo commented Jan 13, 2017

When I test with this query:

tag_names($hostname)

and $hostname=server2, then Grafana issues the following query to OpenTSDB:

GET api/search/lookup?limit=1000&m=server2

When I test with ``tag_names(server2)` then exact same query is sent to OpenTSDB. So it's very strange that you get different results.

@ps7voli
ps7voli commented Jan 13, 2017

$Metric -> metrics(gpfs_.*) -->
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Df,Metric%3Di,Metric%3Ds,Metric%3D_,Metric%3Db,Metric%3Dy,Metric%3Dt,Metric%3De,Metric%3Ds,Metric%3D_,Metric%3Dr,Metric%3De,Metric%3Da,Metric%3Dd HTTP/1.1

$NameHard -> tag_names(gpfs_fs_disks) -->
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Dd,Metric%3Di,Metric%3Ds,Metric%3Dk,Metric%3Ds HTTP/1.1

$Names -> tag_names($Metric) -->
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=gpfs_fs_disks HTTP/1.1

@torkelo
Member
torkelo commented Jan 13, 2017
$NameHard -> tag_names(gpfs_fs_disks) -->
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Dd,Metric%3Di,Metric%3Ds,Metric%3Dk,Metric%3Ds HTTP/1.1

This looks impossible, cannot replicate that.

tag_names(gpfs_fs_disks) -> generates
GET api/search/lookup?limit=1000&m=gpfs_fs_disks

@ps20renar

Hallo Torkelo,
we want to go Forward on Monday with this.

@torkelo
Member
torkelo commented Jan 13, 2017

yes! I am here to help :)

@Helene
Helene commented Jan 15, 2017

Hello Renar, hello Torkel,

Helene is there. It looks like replacing "$variable" for lookup http request doesn't work any more (at least with 4.1.1). I'm surprised a little bit, because I rember I did work some weeks ago with 4 beta version and it did work.... As Renar and his colleague have mentioned this did work pretty sure with 3.0.4. Here I did run the test with 3.0.4:
grafana3 1 1

And I did test it with the latest build (4.1.1):
grafana4 1 1

@ps7voli
ps7voli commented Jan 16, 2017

Hello Torkelo,

sorry for the mistake.

$Metric -> metrics(gpfs_.*)
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Df,Metric%3Di,Metric%3Ds,Metric%3D_,Metric%3Db,Metric%3Dy,Metric%3Dt,Metric%3De,Metric%3Ds,Metric%3D_,Metric%3Dr,Metric%3De,Metric%3Da,Metric%3Dd HTTP/1.1

$Names -> tag_names($Metric)
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Dd,Metric%3Di,Metric%3Ds,Metric%3Dk,Metric%3Ds HTTP/1.1_

$NameHard -> tag_names(gpfs_fs_disks)
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=gpfs_fs_disks HTTP/1.1

@ps20renar

Hallo Torkelo,
a collegue from IBM Zuerich Lab checked the the openTSDB Semantik here. Here are there comments:
Hi Renar,

I checked the bridge code and as I thought, it only accepts a single metric.
Looking for why it was implemented that way, I think I found the reason. The description of openTSDB /api/search/lookup method
described in http://opentsdb.net/docs/build/html/api_http/search/lookup.htmlstates:
A lookup query consists of at least one metric, tag name (tagk) or tag value (tagv). Each value must be a literal name in the UID table.
If a given name cannot be resolved to a UID, an exception will be returned.
Only one metric can be supplied per query but multiple tagk, tagv or tag pairs may be provided.
The bridge follows this, expecting a metric, and returning as tags the identifier attributes in the key for that metric (e.g., "netdev_bytes_r" -> { "node", "netdev_name" }.
Grafana also states that in expects a single metric with tag_names (http://docs.grafana.org/datasources/opentsdb/)
tag_names(cpu) // return tag names (i.e. keys) for a specific cpu metric

So i think this seems to be bug in the lookup code since 4.x in grafana.
Regards Renar.

@torkelo
Member
torkelo commented Jan 16, 2017
Metric -> metrics(gpfs_.*)
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Df,Metric%3Di,Metric%3Ds,Metric%3D_,Metric%3Db,Metric%3Dy,Metric%3Dt,Metric%3De,Metric%3Ds,Metric%3D_,Metric%3Dr,Metric%3De,Metric%3Da,Metric%3Dd HTTP/1.1

$Names -> tag_names($Metric)
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Dd,Metric%3Di,Metric%3Ds,Metric%3Dk,Metric%3Ds HTTP/1.1_

This is not supported, tag_names only supports a single parameter, so the $Metric variable can only have a single value (so disable Multi select / All value).

@ps7voli
ps7voli commented Jan 16, 2017

I have changed the $Metric variable to

$Metric -> metrics(gpfs_fs_bytes_read)
GET /api/datasources/proxy/1/api/suggest?max=1000&q=gpfs_fs_bytes_read&type=metrics HTTP/1.1
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D
,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Db,Metric%3Dy,Metric%3Dt,Metric%3De,Metric%3Ds,Metric%3D_,Metric%3Dr,Metric%3De,Metric%3Da,Metric%3Dd HTTP/1.1_

Multi select / All value is disabled.

Result:

$Names -> tag_names($Metric)
GET /api/datasources/proxy/1/api/search/lookup?limit=1000&m=g,Metric%3Dp,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Df,Metric%3Ds,Metric%3D_,Metric%3Db,Metric%3Dy,Metric%3Dt,Metric%3De,Metric%3Ds,Metric%3D_,Metric%3Dr,Metric%3De,Metric%3Da,Metric%3Dd HTTP/1.1

beschreibung07

@torkelo
Member
torkelo commented Jan 16, 2017

@ps7voli can you show your dashboard json in a gist? Looks like your template variables are not what you set. or something is wrong with the variables. As you seem to set Metric variable to a single value but it still seems to be all values when it is interpolated.

@ps7voli
ps7voli commented Jan 16, 2017

Dashboard json Export:

{
  "__inputs": [
    {
      "name": "DS_SASGRID",
      "label": "SASGrid",
      "description": "",
      "type": "datasource",
      "pluginId": "opentsdb",
      "pluginName": "OpenTSDB"
    }
  ],
  "__requires": [
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph",
      "version": ""
    },
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "4.1.1"
    },
    {
      "type": "datasource",
      "id": "opentsdb",
      "name": "OpenTSDB",
      "version": "1.0.0"
    }
  ],
  "id": null,
  "title": "NewTest-Board",
  "tags": [],
  "style": "dark",
  "timezone": "browser",
  "editable": true,
  "graphTooltip": 0,
  "hideControls": false,
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {
.
.
.
  },
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_SASGRID}",
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "Metric",
        "options": [],
        "query": "metrics(gpfs_fs_bytes_read)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": null,
        "tagsQuery": null,
        "type": "query"
      },
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_SASGRID}",
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "NameHard",
        "options": [],
        "query": "tag_names(gpfs_fs_disks)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": null,
        "tagsQuery": null,
        "type": "query"
      },
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_SASGRID}",
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "Names",
        "options": [],
        "query": "tag_names($Metric)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": null,
        "tagsQuery": null,
        "type": "query"
      }
    ]
  },
.
.
.
@ps7voli
ps7voli commented Jan 16, 2017

View json:


.
.
.
"templating": {
    "list": [
      {
        "allValue": null,
        "current": {
          "text": "gpfs_fs_bytes_read",
          "value": "gpfs_fs_bytes_read"
        },
        "datasource": "SASGrid",
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "Metric",
        "options": [
          {
            "text": "gpfs_fs_bytes_read",
            "value": "gpfs_fs_bytes_read",
            "selected": true
          }
        ],
        "query": "metrics(gpfs_fs_bytes_read)",
        "refresh": 0,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": null,
        "tagsQuery": null,
        "type": "query"
      },
      {
        "allValue": null,
        "current": {
          "text": "node",
          "value": "node"
        },
        "datasource": "SASGrid",
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "NameHard",
        "options": [
          {
            "text": "node",
            "value": "node",
            "selected": true
          },
          {
            "text": "gpfs_cluster_name",
            "value": "gpfs_cluster_name",
            "selected": false
          },
          {
            "text": "gpfs_fs_name",
            "value": "gpfs_fs_name",
            "selected": false
          }
        ],
        "query": "tag_names(gpfs_fs_disks)",
        "refresh": 0,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": null,
        "tagsQuery": null,
        "type": "query"
      },
      {
        "allValue": null,
        "current": {
          "text": "None",
          "value": "",
          "isNone": true
        },
        "datasource": "SASGrid",
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "Names",
        "options": [
          {
            "text": "None",
            "value": "",
            "isNone": true,
            "selected": true
          }
        ],
        "query": "tag_names($Metric)",
        "refresh": 0,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": null,
        "tagsQuery": null,
        "type": "query"
      }
    ]
  },
.
.
.
@kohpinngee

Are the on going conversations related to this issue raised initially?

@kohpinngee
kohpinngee commented Jan 23, 2017 edited

I deployed graphite 0.9.15 and issue is now gone. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment