[regression] template tags query with ES6.x does not split on comma

[mobidyc]

What happened:
Using Elasticsearch 6.7.1 the group/tags are no more splitted.
My documents have a "@roles" field which can contains for the example: "master,ssh,storage"
using group tags, I can literally see the string "master,ssh,storage" in the list.

What you expected to happen:
Using elasticsearch 2.4 the group/tags are separated and do the job

How to reproduce it (as minimally and precisely as possible):
I tried different format for the "@roles" field, keyword and text, exact same issue.

tags query: {"find": "terms", "field": "@roles"}
tags value query: {"find":"terms", "field": "@hostname", "query": "@roles:$tag"}

Anything else we need to know?:

Environment:

• Grafana version: same issue using grafana-6.0.1-1.x86_64 and version 6.1
• OS Grafana is installed on: Centos 7.5
• User OS & Browser: Chrome 73

[torkelo]

What is the response from Elasticsearch for the templating query? Is it different for ES 6?

[mobidyc]

responses are different, indeed.

using elasticsearch 2.3.4:

curl 'http://217.182.11.53/stats/api/datasources/proxy/18/_msearch?search_type=count' -H 'Pragma: no-cache' -H 'Origin: http://217.182.11.53' -H 'Accept-Encoding: gzip, deflate' -H 'X-Grafana-Org-Id: 1' -H 'Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7' -H 'User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36' -H 'Content-Type: application/json;charset=UTF-8' -H 'Accept: application/json, text/plain, */*' -H 'Cache-Control: no-cache' -H 'Referer: http://217.182.11.53/stats/d/w6NeRVkZk/test?refresh=30s&orgId=1&var-HOSTNAME=All' -H 'Cookie: grafana_user=admin; grafana_remember=0067d4456d9f2491d1a315d58fdfb594fb66993da5fbf3098605d3e61740666a8c; scality_grafana_sess=28dfdf2faac1aba1' -H 'Connection: keep-alive' -H 'DNT: 1' --data-binary $'{"search_type":"count","ignore_unavailable":true,"index":"network-stats-v3"}\n{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1555959973983","lte":"1555961773984","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"*"}}]}},"aggs":{"1":{"terms":{"field":"@roles","size":500,"order":{"_term":"asc"}}}}}\n' --compressed

{"responses":[{"took":55,"timed_out":false,"_shards":{"total":128,"successful":128,"failed":0},"hits":{"total":7160,"max_score":0.0,"hits":[]},"aggregations":{"1":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"ElasticSearch","doc_count":4296},{"key":"NFS","doc_count":6444},{"key":"S3","doc_count":3580},{"key":"SVSD","doc_count":6444},{"key":"Sfused","doc_count":6444},{"key":"Sproxyd","doc_count":4296},{"key":"Storage","doc_count":4296},{"key":"Supervisor","doc_count":716}]}}}]}

using elasticsearch 6.7.1:

curl 'http://10.200.2.208/stats/api/datasources/proxy/21/_msearch?search_type=query_then_fetch' -H 'Accept: application/json, text/plain, */*' -H 'Referer: http://10.200.2.208/stats/d/10tLe7kZz/test?orgId=1&refresh=30s' -H 'Origin: http://10.200.2.208' -H 'X-Grafana-Org-Id: 1' -H 'DNT: 1' -H 'Content-Type: application/json;charset=UTF-8' --data-binary $'{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"network-stats-v4","max_concurrent_shard_requests":256}\n{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1555959028762","lte":"1555960828762","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"*"}}]}},"aggs":{"1":{"terms":{"field":"@roles","size":500,"order":{"_key":"asc"}}}}}\n' --compressed

{"responses":[{"took":6,"timed_out":false,"_shards":{"total":6,"successful":6,"skipped":0,"failed":0},"hits":{"total":2690,"max_score":0.0,"hits":[]},"aggregations":{"1":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Storage,ElasticSearch,S3","doc_count":1350},{"key":"Storage,ElasticSearch,S3,NFS","doc_count":670},{"key":"Supervisor","doc_count":670}]}},"status":200}]}

Do I have to change something on the mapping side or on the grafana request?

[mobidyc]

I did my homework and rechecked everything.
It works using a "text" field AND fielddata=true.

the fielddata error does not appear in Grafana, it is hard to dig into the issue.
at least, I learned how to use the web developer tools ;)

Just for info, below are the tests made:

On elasticsearch 6.x

# curl -s 10.200.2.121:9200/network-stats-v4 |jq '."network-stats-v4-2019-04-22".mappings."network-stats-v4".properties."@roles"'
{
  "type": "keyword"
}
# curl -s -XPOST 10.200.2.121:9200/network-stats-v4/_msearch -H 'Content-Type: application/json;charset=UTF-8' --data-binary $'{"ignore_unavailable":true,"index":"network-stats-v4"}\n{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1555959973983","lte":"2555961773984","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"*"}}]}},"aggs":{"1":{"terms":{"field":"@roles","size":500,"order":{"_term":"asc"}}}}}\n' ; echo

{"responses":[{"took":9,"timed_out":false,"_shards":{"total":6,"successful":6,"skipped":0,"failed":0},"hits":{"total":3590,"max_score":0.0,"hits":[]},"aggregations":{"1":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"Storage,ElasticSearch,S3","doc_count":1800},{"key":"Storage,ElasticSearch,S3,NFS","doc_count":895},{"key":"Supervisor","doc_count":895}]}},"status":200}]}

# curl -s 10.200.2.121:9200/network-stats-v4\* -XDELETE
### change mapping
# curl -s 10.200.2.121:9200/network-stats-v4 |jq '."network-stats-v4-2019-04-22".mappings."network-stats-v4".properties."@roles"'
{
  "type": "text"
}


# curl -s -XPOST 10.200.2.121:9200/network-stats-v4/_msearch -H 'Content-Type: application/json;charset=UTF-8' --data-binary $'{"ignore_unavailable":true,"index":"network-stats-v4"}\n{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1555959973983","lte":"2555961773984","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"*"}}]}},"aggs":{"1":{"terms":{"field":"@roles","size":500,"order":{"_term":"asc"}}}}}\n' ; echo
{"responses":[{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [@roles] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"network-stats-v4-2019-04-22","node":"5KJAzrYXSG2ZLwNJRN6IKw","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [@roles] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [@roles] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [@roles] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}},"status":400}]}

# curl -s 10.200.2.121:9200/network-stats-v4\* -XDELETE
### change mapping
# curl -s 10.200.2.121:9200/network-stats-v4 |jq '."network-stats-v4-2019-04-22".mappings."network-stats-v4".properties."@roles"'
{
  "type": "text",
  "fielddata": true
}

#  
# curl -s -XPOST 10.200.2.121:9200/network-stats-v4/_msearch -H 'Content-Type: application/json;charset=UTF-8' --data-binary $'{"ignore_unavailable":true,"index":"network-stats-v4"}\n{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1555959973983","lte":"2555961773984","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"*"}}]}},"aggs":{"1":{"terms":{"field":"@roles","size":500,"order":{"_term":"asc"}}}}}\n' ; echo

{"responses":[{"took":7,"timed_out":false,"_shards":{"total":3,"successful":3,"skipped":0,"failed":0},"hits":{"total":420,"max_score":0.0,"hits":[]},"aggregations":{"1":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"elasticseach","doc_count":315},{"key":"nfs","doc_count":105},{"key":"s3","doc_count":315},{"key":"storage","doc_count":315},{"key":"supervisor","doc_count":105}]}},"status":200}]}