Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch 7.x support #15622

Closed
UKSFM99 opened this issue Feb 24, 2019 · 37 comments

Comments

@UKSFM99
Copy link

commented Feb 24, 2019

What Grafana version are you using?

6.0.0 beta 3

What datasource are you using?

Elasticsearch 7.0.0 beta 1

What OS are you running grafana on?

Ubuntu

What did you do?

After upgrading a Elasticsearch cluster from 6.6 to 7.0 beta 1, I can no longer add new indices as seperate datasources to grafana, due to the following error:
image

However, current data sources that were created when the cluster was 6.6.0 now no longer work, all charts display the following error:
image

Some useful F12 sceenshots to help:

  1. Response when using old data sources after upgrading to Elasticsearch 7.0:
    image

  2. Response when trying to create new data sources that are 7.0:
    the _mapping Json is the following:

{"logging-2019.02.24":{"mappings":{"properties":{"@timestamp":{"type":"date"},"@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"Host-City":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"Host-Country-Code":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"Host-Env":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"Host-Type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"HostName":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"Log-Event":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},,"host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"port":{"type":"long"},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}

Which clearly shows the @timestamp field as being of type 'date', even though grafana complains that @timestamp field does not exist

@marefr

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

Regarding max_concurrent_shard_requests the semantics has changed to be the maximum and now defaults to 5 instead of 256: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#_semantics_changed_for_literal_max_concurrent_shard_requests_literal

@Julzorensen

This comment has been minimized.

Copy link

commented Apr 11, 2019

Hey guys,

Elastic Stack v7 is now GA, some production cluster will be (or are already) upgraded.
Mine is now in full 7.0.0 (GA) and I encouter the exact same problem.

Basicly, Grafana seems to not support this new version of ElasticSearch.

Any idea when it'll be fixed?

@danielfariati

This comment has been minimized.

Copy link

commented Apr 11, 2019

Same problem here. We upgraded our production cluster and now Grafana stopped working.

The problem did not occured in all of my indexes, though. I think it's related to the amount of data in each of them. Indexes that have small amount of data seems to work, but indexes with lots of documents seems to crash.

Also, when trying to save / edit Datasources, an error is thrown: No date field named @timestamp found (even though the field exists).

@creaticious

This comment has been minimized.

Copy link

commented Apr 12, 2019

Same here...upgraded today, can't update datasource because of @timestamp not found...
Maybe this is related to the deprecation of document types in ES 7.0...?

@c0r3xxx

This comment has been minimized.

Copy link

commented Apr 12, 2019

we upgraded today too.. same here

@robertotrgt

This comment has been minimized.

Copy link

commented Apr 12, 2019

Same over here, upgraded yesterday to ES7, noticed today grafana is not working with that cluster.

@torkelo torkelo added this to the 6.3 milestone Apr 15, 2019

@marefr

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

We're planning to fix this for Grafana v6.3 where stable is ~6 weeks forward, but will probably land in master/nightly earlier.

@hastarin

This comment has been minimized.

Copy link

commented Apr 17, 2019

As another person who forgot all about Grafana and broke their production dashboard by upgrading things without testing first (and I even have an ELK dev stack in Docker Swarm) I would love some hints on what I might be able to do to assist in testing any early releases of a fix for this?

I've had to embed a Kibana dashboard via an iframe for now.

@alcidesv

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2019

@hastarin

Here is a docker image for the PR with this feature: eu.gcr.io/shimmercat-pdi-7108/grafana

@hastarin

This comment has been minimized.

Copy link

commented Apr 17, 2019

@alcidesv Wow thanks for the prompt response. I've now built my own image based on that with extra plugins, etc and it's up and running.

I am pleased to confirm that it has solved my issue for the dashboards I have. Many thanks for your efforts!

@danielfariati

This comment has been minimized.

Copy link

commented Apr 17, 2019

I also tested the docker image.
The dashboards are now working again! Thanks, @alcidesv!

The only thing that appears to not be working is the alerting functionality, always returning: Query Returned No Series (reduced to null/no value).

@marefr

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

@danielfariati Thanks for feedback. Fix for that been pushed to #16646, but currently docker image mentioned above have not been updated as far as I'm concerned.

@PepeLopez

This comment has been minimized.

Copy link

commented Apr 19, 2019

I don't get it. This thread is about ES7 compatibility, which is not released yet, but ppl talking about an docker image which is working!? Has the docker image ES7 compatibility, but not the nightly?
The confusion is real...

@marefr

This comment has been minimized.

Copy link
Member

commented Apr 20, 2019

@PepeLopez

This comment has been minimized.

Copy link

commented Apr 20, 2019

@PepeLopez see #15622 (comment) above

I did. It's not answering my question.
Does that docker image fixed the ES7 compatibility?

@marefr

This comment has been minimized.

Copy link
Member

commented Apr 21, 2019

@PepeLopez this issue is open because official Grafana support does not currently exists for elasticsearch v7. The docker image above is a build of the ongoing work for adding support for elasticsearch v7 and have been built by the author of PR #16646. No alerting support in above image as far as I'm concern, but besides that it sort of works. Please try it out if you want and provide feedback, if any. Thanks

@torkelo

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

  1. Because the features is not implemented / completed yet
  2. we normally don’t backport features and enhancements
@danielfariati

This comment has been minimized.

Copy link

commented Apr 23, 2019

@danielfariati Thanks for feedback. Fix for that been pushed to #16646, but currently docker image mentioned above have not been updated as far as I'm concerned.

@marefr I tested the code pushed to #16646. Everything (including alerts) seems to be working perfectly, now.
Thanks!

marefr added a commit that referenced this issue Apr 25, 2019

elasticsearch: add 7.x version support (#16646)
Adds a new version option 7.0+ (70 internally).
Version 7.0+ doesn't include document types in index mappings 
so support for handling this have been added.
Version 7.0+ returns number of hits in a different way so 
support for handling this have been added.
Version 7.0+ doesn't support sending 
max_concurrent_shard_requests in multisearch header so 
support for sending this in query string have been added.
Update elastic6 docker block and dashboards (devenv) to use 
6.7.1 images, filebeat index name is now filebeat-YYYY.MM.DD 
and dashboard include correct tags and links.
Add elastic7 docker block and provisioning (devenv).
Updates documentation regarding new version.

Closes #15622

@marefr marefr modified the milestones: 6.3, 6.2 Apr 25, 2019

@sirajshe

This comment has been minimized.

Copy link

commented May 1, 2019

I am using the latest nightly build of grafana to connect to ES 7.0.

INFO[05-01|06:38:43] Starting Grafana logger=server version=6.1.6 commit=cf9cb45 branch=HEAD compiled=2019-04-29T13:29:28+0000

While saving the datasource, it gives an error

"no handler found for uri [/5/1/2019og0tam06-2019.04.14-000001/_mapping] and method [GET]"

The request in the grafana log is recorded as follows. Looks like a jumbled up index name being used.

INFO[05-01|06:49:03] Request Completed logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/5/1/2019og0tam06-2019.04.14-000001/_mapping status=400 remote_addr=xx.xx.xx.xx time_ms=2 size=119 referer=http://xxxxx:2000/datasources/edit/1/

Following is my data source config.

image

@dnewsholme

This comment has been minimized.

Copy link

commented May 1, 2019

@marefr

This comment has been minimized.

Copy link
Member

commented May 1, 2019

@sirajshe and you have selected Hourly as index pattern which seems wrong in your case since you have inserted the full index name in the index name field.

@sirajshe

This comment has been minimized.

Copy link

commented May 1, 2019

@dnewsholme - The datasource creation UI only has options upto 6.0+. Not sure if the nightly build I used has the changes. (version=6.1.6 commit=cf9cb45 branch=HEAD compiled=2019-04-29T13:29:28+0000)

@marefr - Thanks for the pointer. I removed the pattern and now I am back to the original issue in this thread, ie. "No date field named @timestamp found".

The data source needs to set to elastic search 7.0 not 6.0

On Wed, 1 May 2019, 07:55 Siraj Sherriff, @.***> wrote: I am using the latest nightly build of grafana to connect to ES 7.0. While saving the datasource, it gives an error "no handler found for uri [/5/1/2019og0tam06-2019.04.14-000001/_mapping] and method [GET]" The request in the grafana log is recorded as follows. Looks like a jumbled up index name being used. INFO[05-01|06:49:03] Request Completed logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/5/1/2019og0tam06-2019.04.14-000001/_mapping status=400 remote_addr=xx.xx.xx.xx time_ms=2 size=119 referer= http://xxxxx:2000/datasources/edit/1/ Following is my data source config. [image: image] https://user-images.githubusercontent.com/46110311/57008196-3f753d00-6ba3-11e9-944e-00f7d519b0a3.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#15622 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHKKEMG4AZSNKNGZJ2CBPLPTE5E5ANCNFSM4GZ2NWJQ .

@sirajshe

This comment has been minimized.

Copy link

commented May 1, 2019

Using a 6.2.0 nightly build now. Will update the thread with the results.

@sirajshe

This comment has been minimized.

Copy link

commented May 1, 2019

Looks good with the latest build. ES 7.0+ option is available and the data source is functional. Thanks for the help.
version=6.2.0-pre commit=157f330 branch=master compiled=2019-05-01T15:20:01+0000

@sirajshe

This comment has been minimized.

Copy link

commented May 2, 2019

@tonobo

This comment has been minimized.

Copy link

commented May 2, 2019

I'm following grafana master branch and having a elasticsearch cluster with version 6.7.1 running. The alert rule is pointing to 6.0+ but i'm still getting the error tsdb.HandleRequest() error json: cannot unmarshal number into Go struct field SearchResponseHits.Total of type map[string]interface {}. This might be related to the 7.0 changes?

@drew-o

This comment has been minimized.

Copy link

commented May 2, 2019

Oh yes, thanks for the tip, I realised that just after posting the question. Works like a charm now.

I see the nightly build you are using is about 15 days old and there are ES 7.0 support related commits like the following that happened 8 days back. 8816d6d I used a most recent nightly build v6.2.0-pre (157f330) and I see the 7.0+ options.

On Wed, May 1, 2019 at 3:32 PM drew-o @.***> wrote: Hello, using Grafana v6.2.0-pre (commit: 9cc67e4 <9cc67e4>) and there's no ES 7.0+ option here. Using nightly build (grafana_6.2.0-91846ebepre_amd64.deb) Maybe I'm missing something? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15622 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AK7ZMZ3ZR35W4GDURRR67VDPTIK7RANCNFSM4GZ2NWJQ .

@marefr

This comment has been minimized.

Copy link
Member

commented May 6, 2019

@tonobo thanks for the feedback. That sounds like a bug and we'll fix that for the 6.2-beta1. Re-opening this issue...

@marefr marefr reopened this May 6, 2019

@marefr marefr self-assigned this May 6, 2019

@marefr marefr closed this in #16904 May 6, 2019

marefr added a commit that referenced this issue May 6, 2019

Elasticsearch: Fix pre-v7.0 and alerting error (#16904)
This fixes a regression introduced in #16646 where using Elasticsearch 
pre-v7.0 and alerting resulted in an error when trying to deserialize the 
response of total number of hits.
Total number of hits is not in use by the backend so we're removing it 
for now to make ES 6 and 7 being able to deserialize search responses 
without errors.

Closes #15622
@tonobo

This comment has been minimized.

Copy link

commented May 6, 2019

Awsome!

ademmers pushed a commit to ademmers/grafana that referenced this issue May 9, 2019

elasticsearch: add 7.x version support (grafana#16646)
Adds a new version option 7.0+ (70 internally).
Version 7.0+ doesn't include document types in index mappings
so support for handling this have been added.
Version 7.0+ returns number of hits in a different way so
support for handling this have been added.
Version 7.0+ doesn't support sending
max_concurrent_shard_requests in multisearch header so
support for sending this in query string have been added.
Update elastic6 docker block and dashboards (devenv) to use
6.7.1 images, filebeat index name is now filebeat-YYYY.MM.DD
and dashboard include correct tags and links.
Add elastic7 docker block and provisioning (devenv).
Updates documentation regarding new version.

Closes grafana#15622
Conflicts:
	devenv/datasources.yaml
	devenv/dev-dashboards/datasource-elasticsearch/elasticsearch_compare.json
	devenv/dev-dashboards/datasource-elasticsearch/elasticsearch_v2.json
	devenv/dev-dashboards/datasource-elasticsearch/elasticsearch_v5.json
	devenv/dev-dashboards/datasource-elasticsearch/elasticsearch_v6.json
	devenv/dev-dashboards/datasource-elasticsearch/elasticsearch_v6_filebeat.json
	devenv/docker/blocks/elastic6/docker-compose.yaml
	devenv/docker/blocks/elastic6/filebeat.yml
	docs/sources/administration/provisioning.md
	docs/sources/features/datasources/elasticsearch.md
	public/app/plugins/datasource/elasticsearch/config_ctrl.ts
@arikfre

This comment has been minimized.

Copy link

commented May 28, 2019

Hi,
Any chance there was a regression?
I'm using Grafana 6.2.1 and Elasticsearch 7.1 and I'm getting the mentioned error:
"no handler found for uri [/pro1559001600000y0q5/28/2019_qu2ry_2ig20t_2019.05/_mapping] and method [GET]"

image

@marefr

This comment has been minimized.

Copy link
Member

commented May 28, 2019

@arikfre please use [query_digest_]YYYY.MM.DD as index name.

@arikfre

This comment has been minimized.

Copy link

commented May 28, 2019

@marefr thank you so much. Works perfectly.

@Ivan-Strahovsky

This comment has been minimized.

Copy link

commented Jun 12, 2019

ES version 7.1.1. Grafana: 6.2.2. This is what I see in alert state history:

image
Error: tsdb.HandleRequest() error elasticsearch version=70 is not supported
what is interesting - when I click test rule it always return me OK state. What can be wrong?

@marefr

This comment has been minimized.

Copy link
Member

commented Jun 13, 2019

@Ivan-Strahovsky that's interesting. Can you try and re-save your dashboard with the alert rule? Can you show a screenshot of the datasource configuration?

@Ivan-Strahovsky

This comment has been minimized.

Copy link

commented Jun 13, 2019

@marefr
image
I tried to recreate alert, dashboard, datasource. Result the same.

@marefr

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

Issue #17558 opened for further discussions around latest comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.