Elasticsearch 7 Support #5933

jalogisch · 2019-05-09T08:15:51Z

Expected Behavior

Graylog 3.0 should work with Elasticsearch 7

2019-05-08 16:36:57,413 ERROR: org.graylog2.periodical.ConfigurationManagementPeriodical - Error while running migration <V20170607164210_MigrateReopenedIndicesToAliases{2017-06-07T16:42:10Z}>
org.graylog2.indexer.ElasticsearchException: Unsupported Elasticsearch version: 7.0.1

Your Environment

Graylog Version: 3

The text was updated successfully, but these errors were encountered:

whitelife · 2019-05-10T15:15:11Z

Hi. jalogisch

We have upgraded to Elasticsearch 7 in our test environment.
Graylog was using 3.0.1.

I checked the parts that did not work.
We have recovered to Elasticsearch 6.7.x.

When will you be supporting Elasticsearch 7?

gruselglatz · 2019-05-20T13:23:44Z

Will this be some major change? Or will it be ready in some days?

gruselglatz · 2019-05-29T08:43:08Z

After reading this: https://www.elastic.co/de/blog/security-for-elasticsearch-is-now-free I think there is a major reason to upgrade to 6.8 and 7.1+ so I hope Graylog will release a 7+ compatible version soon.

breisig · 2019-06-24T17:51:20Z

Elasticsearch 7.1+ functionality should be added right away.

ThaDaVos · 2019-07-25T10:20:51Z

Any eta guys? - I just ran into the issue of 7.2 not being supported and reading this I guess the whole 7.x isn't supported yet

KDGundermann · 2019-08-08T12:57:10Z

at least update the documentation and write it in BOLD that Elasticsearch 7.x is not supported !!

jalogisch · 2019-08-21T08:46:21Z

at least update the documentation and write it in BOLD that Elasticsearch 7.x is not supported !!

that is already done: http://docs.graylog.org/en/3.1/pages/installation.html

alen-z · 2019-10-02T12:42:41Z

Any projections about timeline for milestone 3.2.0 completion? Thank you.

rdoering · 2019-10-25T00:08:02Z

any progress here?

tehpanta · 2019-11-11T11:59:50Z

Any ETA on this?

gruselglatz · 2019-11-11T12:20:34Z

Resolved the Issue, only using Kibana now. Since The 7.4 Stack offers a even a SIEM now, we don't need Graylog anymore, sry guys.

rdoering · 2019-11-11T14:11:49Z

Still no progress :-/

piellick · 2019-11-14T14:04:11Z

+1

devopstales · 2019-11-18T18:25:14Z

+1

entrop-tankos · 2019-11-26T10:03:51Z

+1

dinuta · 2019-12-02T11:37:29Z

Resolved the Issue, only using Kibana now. Since The 7.4 Stack offers a even a SIEM now, we don't need Graylog anymore, sry guys.

I just told them that this is a big issue. I mean you force people to use Kibana.
The people in general wants the latest features and security fixes, thus having latest versions of elasticsearch.

I am using Kibana now, but i want to try Graylog for the nice features and web interface. I saw some articles and looks really cool.

MartinVerges · 2019-12-09T09:27:26Z

same here, please make graylog year 2019 compatible and don't stay behind!

whitelife · 2019-12-10T09:50:48Z

I am using Graylog efficiently.
It's been about two years.

I hope it will be supported in year 2019.

jbguerraz · 2019-12-13T10:27:01Z

Hello!
@jalogisch how can we, as a community, help make this happen ? may someone provide some guidance ? we probably can contribute to move (at least!) a step ahead.
there is still (mostly) about a year before elasticsearch 6 EOL so it's not so critical yet :)

jalogisch · 2019-12-13T16:23:30Z

@jbguerraz you are right Elasticsearch 6 is not EOL yet - if Graylog is supporting ES 7 it will drop the ES 5 support and according to our active user base this is quite common setup. Because environments with heavy load are not always on the latest stable release and tend to move slowly.

I already asked several times - what exact feature from ES7 do you need and why?

piellick · 2020-04-30T11:09:53Z

Hello, it's cancelled for 3.3.0 yet ?

hulkk · 2020-05-25T08:55:20Z

Any updates?

breisig · 2020-05-25T15:46:25Z

This is crazy not have Elasticsearch 7 support.

hulkk · 2020-05-26T06:56:43Z

I'm concerned because there is no (public) decision or roadmap about adding support. Elasticsearch EOL is expected in six months.

MartinVerges · 2020-05-26T07:15:55Z

Very sad how the future of a great product is endangered by absolute oversleeping of the present.

ismaelpuerto · 2020-05-26T07:35:54Z

I have some clusters with elasticsearch 2. I don't understand because some people want have the latest version.

kroepke · 2020-05-26T08:34:13Z

Hi!

Updated Elasticsearch support is scheduled for Graylog 4.0, which will come out later this year.

While we appreciate the interest and concern, I can assure you that we are not ignoring it.

That being said, major Elasticsearch upgrades are always a huge deal for users and customers with large clusters, due to the way elastic handles backward compatibility in various APIs, notable mapping changes, and (Lucene) index version support. This presents tremendous challenges for upgrading.

At the same time, larger organizations are not as quick to update as y'all might think, because of the time and testing involved, and the considerable amount of resources invested.

Long story short, work has already begun for supporting more recent elasticsearch versions and we are again committed (as we were in Graylog 3.0) to make this migration as seamless as possible.
If you are watching commits and pull requests, you will notice significant activity over the next weeks on this topic.

Thanks for your support.

hulkk · 2020-05-26T09:49:00Z

While we appreciate the interest and concern, I can assure you that we are not ignoring it.

But due to the limited/missing communication we have had no knowledge about your thoughts. In the future I think the community would appreciate some kind of roadmaps for major component support.

kalibyrn · 2020-06-03T11:45:38Z

Perhaps too often unsaid: We really appreciate the hard work that you put into a great product.

Thanks for clarifying that work has started on ES7 support and what version it will arrive in. I have a (at least perceived) need to use the 7.x series of ES products.

Filebeat 6.x doesn't support the "add_fields" processor type. We use that (in Filebeat 7.x) to add fields to messages when a regex catches a specific error pattern. This lets us filter on that new field and send the message to another stream in Graylog.

When we use Filebeat 7.x however, a lot of root level fields are listed as "unknown" such as source and name.

gmeks · 2020-06-18T11:09:17Z

With Graylog 4.0 will this bring support for date_nano ?
That would posibile resolve #2741

kroepke · 2020-06-18T11:44:12Z

With Graylog 4.0 will this bring support for date_nano ?

That would posibile resolve #2741

I believe date_nanos is incompatible with regular date fields, so it would be an either/or choice because queries and aggregations would not work across indices anymore.

Out of the box we would therefore not have support for it. I’m also not sure the Java libraries we are using can all deal with nanosecond resolution in all the places, so that would be a separate effort for sure.

gmeks · 2020-06-18T12:32:20Z

Perhaps its wishfull thinkinking. I was hoping for milisecond resolution ( I assume java supports that). And when reasearching this i found date_nano and ES 7 behing "solution" to milisecond resolution on log entires.

Thats why i thought these 2 issues might be linked. :)

Thank you for taking the time to respond.

dennisoelkers · 2020-08-21T14:57:25Z

If there is anyone particularly adventurous, the current master branch supports ES7 now. If you feel like it, please take it our for a test ride (do NOT do anything mission critical with it) and give us some feedback. All you need to do is upgrade ES to ES7, and set elasticsearch_version = 7 in your config file.

Update (Oct 2020): The elasticsearch_version setting is not required anymore, the version should be auto-sensed.

dinuta · 2020-08-31T13:41:18Z

I only see hooray, but did anyone tested?

LATER EDIT:

I was using docker-compose to quickly validate this. It does not work with the latest docker hub image 3.3.5-1. I also attach the compose file. docker compose -f docker-compose-graylog.yml up --build

2020-08-31 14:36:27,821 INFO : org.graylog2.indexer.MongoIndexSet - Did not find a deflector alias. Setting one up now.
2020-08-31 14:36:27,823 INFO : org.graylog2.indexer.MongoIndexSet - There is no index target to point to. Creating one now.
2020-08-31 14:36:27,825 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <none> to <fluentd_0>.
2020-08-31 14:36:27,825 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <fluentd_0>.
2020-08-31 14:36:27,827 ERROR: org.graylog2.periodical.IndexRotationThread - Couldn't point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Unsupported Elasticsearch version: 7.8.0
        at org.graylog2.indexer.IndexMappingFactory.indexMappingFor(IndexMappingFactory.java:51) ~[graylog.jar:?]
        at org.graylog2.indexer.IndexMappingFactory.createIndexMapping(IndexMappingFactory.java:42) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.ensureIndexTemplate(Indices.java:364) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.create(Indices.java:411) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.create(Indices.java:397) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:293) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:261) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_265]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_265]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_265]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]

fluentd_efk.zip

jbguerraz · 2020-08-31T17:38:28Z

@dinuta as @dennisoelkers said, it's on master branch, not on version 3.3.5. If you wanna give it a try you gonna have to run the master branch.
We'll be happy to give it a test ride, but sadly not before a couple of months (maybe even a bit more).

dinuta · 2020-08-31T17:45:06Z

How I can compile graylog? Where I get the master artifact?

jbguerraz · 2020-09-01T15:30:42Z

@dinuta probably https://community.graylog.org/t/compiling-graylog-server-for-production/2271 could helps. https://github.com/Graylog2/graylog2-server/blob/master/.travis.yml shows what travis run to build it.
Hope it helps :)

zez3 · 2020-09-17T17:57:32Z

@dinuta Did you managed to test the latest version supporting ESv7 ?
Seara faina

cubed-it · 2020-09-19T16:36:45Z

Are there no precompiled master artifacts available anywehre, as dinuta already asked for?

avolmensky · 2020-09-21T06:32:49Z

I have compiled the master branch, also modified the Dockerfile if anyone is interested.

https://github.com/avolmensky/graylog

cubed-it · 2020-09-21T16:06:24Z

@avolmensky thanks for that!

2020-09-21 15:54:58,790 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.

However there are some errors in the log. Are they "normal"?

logs.txt

dinuta · 2020-09-22T20:15:24Z

logs.txt
Hi @zez3 ,

For me it did not work. Graylog does not create indexes in ES. No errors on ES, but saw them on graylog. Keep in mind if you re-create the env to start graylog with delay, otherwise will throw ClientConnectException.

2020-09-22 19:23:01,657 ERROR: org.graylog2.periodical.ConfigurationManagementPeriodical - Error while running migration <V20200730000000_AddGl2MessageIdFieldAliasForEvents{2020-07-30T00:00:00Z}>
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [gl-events*]]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1897) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1867) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1624) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1596) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1563) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.putMapping(IndicesClient.java:213) ~[?:?]
        at org.graylog.storage.elasticsearch7.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.addGl2MessageIdFieldAlias(V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.java:37) ~[?:?]
        at org.graylog.plugins.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEvents.upgrade(V20200730000000_AddGl2MessageIdFieldAliasForEvents.java:75) ~[graylog.jar:?]
        at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:43) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
        Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch:9200], URI [/gl-events*,gl-system-events*/_mapping?master_timeout=30s&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false&ignore_throttled=false&timeout=30s], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [gl-events*]","resource.type":"index_or_alias","resource.id":"gl-events*","index_uuid":"_na_","index":"gl-events*"}],"type":"index_not_found_exception","reason":"no such index [gl-events*]","resource.type":"index_or_alias","resource.id":"gl-events*","index_uuid":"_na_","index":"gl-events*"},"status":404}
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:261) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1611) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1596) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1563) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.putMapping(IndicesClient.java:213) ~[?:?]
                at org.graylog.storage.elasticsearch7.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.addGl2MessageIdFieldAlias(V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.java:37) ~[?:?]
                at org.graylog.plugins.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEvents.upgrade(V20200730000000_AddGl2MessageIdFieldAliasForEvents.java:75) ~[graylog.jar:?]
                at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:43) [graylog.jar:?]
                at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
2020-09-22 19:23:01,907 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:14, serverValue:14}] to mongo:27017

ES 7.8.0

cubed-it · 2020-09-23T18:51:37Z

@dinuta your log contains almost the same messages as mine. Also your log reports that your graylog is up an running.
I could not observe any misbehaviour on my graylog despite the error messages.

dennisoelkers · 2020-10-06T12:08:39Z

Hey @dinuta, @cubed-it,

thanks a lot for testing the 4.0 snapshot and reporting back this issue. It helped us to uncover an issue before we are hitting beta. Much appreciated!

The cause for this is that we were handling an empty list of indices (what could happen as the result of a race condition on a fresh install) differently between ES6 and ES7. Besides the error on startup there should be no further impact. I have fixed this issue and created a PR for it: #9094. Once it's merged, the error should not happen anymore for fresh installs.

If you are seeing any other errors during testing, please open a new issue, that would help us a lot.

dennisoelkers · 2020-10-06T12:10:22Z

As ES7 support is basically completed, I am now closing this issue. Any further issues/bugs/requests should go to separate issues. Thanks a lot for all of your patience and support!

zez3 · 2020-10-07T09:47:40Z

@dennisoelkers
Does the new beta perhaps commit to the ECS?

dennisoelkers · 2020-10-07T13:40:03Z

Hey @zez3, we currently support the GIM - Graylog Information Model, which is similar to ECS, but is our own schema, with full mapping to our Graylog Illuminate content. To see the schema, you can visit https://schema.graylog.org

john-larson · 2020-10-11T11:17:08Z

Since this issue is closed, does this mean there is a version of Graylog available that supports ES7?

Nothing4You · 2020-10-11T13:06:02Z

as explained in an earlier comment in this issue it's included for graylog 4 which is planned for release later this year, not graylog 3

deeshe added infrastructure to-test labels May 13, 2019

deeshe added this to the 3.1.0 milestone May 13, 2019

deeshe added the elasticsearch label May 13, 2019

bernd modified the milestones: 3.1.0-legacy, 3.1.0 Jul 25, 2019

bernd modified the milestones: 3.1.0, 3.2.0 Aug 13, 2019

bernd added the #XL > 1w label Oct 28, 2019

jalogisch mentioned this issue Dec 2, 2019

Graylog 3.x does not work with Elasticsearch 7.x! #6912

Closed

dennisoelkers closed this as completed Oct 6, 2020

Elasticsearch 7 Support #5933

Elasticsearch 7 Support #5933

Comments

jalogisch commented May 9, 2019

Expected Behavior

Your Environment

whitelife commented May 10, 2019

gruselglatz commented May 20, 2019

gruselglatz commented May 29, 2019

breisig commented Jun 24, 2019

ThaDaVos commented Jul 25, 2019

KDGundermann commented Aug 8, 2019

jalogisch commented Aug 21, 2019

alen-z commented Oct 2, 2019

rdoering commented Oct 25, 2019

tehpanta commented Nov 11, 2019

gruselglatz commented Nov 11, 2019

rdoering commented Nov 11, 2019

piellick commented Nov 14, 2019

devopstales commented Nov 18, 2019

entrop-tankos commented Nov 26, 2019

dinuta commented Dec 2, 2019

MartinVerges commented Dec 9, 2019

whitelife commented Dec 10, 2019 • edited Loading

jbguerraz commented Dec 13, 2019

jalogisch commented Dec 13, 2019

piellick commented Apr 30, 2020

hulkk commented May 25, 2020

breisig commented May 25, 2020

hulkk commented May 26, 2020

MartinVerges commented May 26, 2020

ismaelpuerto commented May 26, 2020

kroepke commented May 26, 2020

hulkk commented May 26, 2020

kalibyrn commented Jun 3, 2020 • edited Loading

gmeks commented Jun 18, 2020

kroepke commented Jun 18, 2020

gmeks commented Jun 18, 2020

dennisoelkers commented Aug 21, 2020 • edited Loading

dinuta commented Aug 31, 2020 • edited Loading

jbguerraz commented Aug 31, 2020

dinuta commented Aug 31, 2020

jbguerraz commented Sep 1, 2020

zez3 commented Sep 17, 2020

cubed-it commented Sep 19, 2020

avolmensky commented Sep 21, 2020

cubed-it commented Sep 21, 2020

dinuta commented Sep 22, 2020 • edited Loading

cubed-it commented Sep 23, 2020

dennisoelkers commented Oct 6, 2020

dennisoelkers commented Oct 6, 2020

zez3 commented Oct 7, 2020

dennisoelkers commented Oct 7, 2020

john-larson commented Oct 11, 2020

Nothing4You commented Oct 11, 2020

whitelife commented Dec 10, 2019 •

edited

Loading

kalibyrn commented Jun 3, 2020 •

edited

Loading

dennisoelkers commented Aug 21, 2020 •

edited

Loading

dinuta commented Aug 31, 2020 •

edited

Loading

dinuta commented Sep 22, 2020 •

edited

Loading