-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Elasticsearch 7 Support #5933
Comments
Hi. jalogisch We have upgraded to Elasticsearch 7 in our test environment. I checked the parts that did not work. When will you be supporting Elasticsearch 7? |
Will this be some major change? Or will it be ready in some days? |
After reading this: https://www.elastic.co/de/blog/security-for-elasticsearch-is-now-free I think there is a major reason to upgrade to 6.8 and 7.1+ so I hope Graylog will release a 7+ compatible version soon. |
Elasticsearch 7.1+ functionality should be added right away. |
Any eta guys? - I just ran into the issue of 7.2 not being supported and reading this I guess the whole 7.x isn't supported yet |
at least update the documentation and write it in BOLD that Elasticsearch 7.x is not supported !! |
that is already done: http://docs.graylog.org/en/3.1/pages/installation.html |
Any projections about timeline for milestone 3.2.0 completion? Thank you. |
any progress here? |
Any ETA on this? |
Resolved the Issue, only using Kibana now. Since The 7.4 Stack offers a even a SIEM now, we don't need Graylog anymore, sry guys. |
Still no progress :-/ |
+1 |
2 similar comments
+1 |
+1 |
I just told them that this is a big issue. I mean you force people to use Kibana. I am using Kibana now, but i want to try Graylog for the nice features and web interface. I saw some articles and looks really cool. |
same here, please make graylog year 2019 compatible and don't stay behind! |
I am using Graylog efficiently. I hope it will be supported in year 2019. |
Hello! |
@jbguerraz you are right Elasticsearch 6 is not EOL yet - if Graylog is supporting ES 7 it will drop the ES 5 support and according to our active user base this is quite common setup. Because environments with heavy load are not always on the latest stable release and tend to move slowly. I already asked several times - what exact feature from ES7 do you need and why? |
Hello, it's cancelled for 3.3.0 yet ? |
Any updates? |
This is crazy not have Elasticsearch 7 support. |
I'm concerned because there is no (public) decision or roadmap about adding support. Elasticsearch EOL is expected in six months. |
Very sad how the future of a great product is endangered by absolute oversleeping of the present. |
I have some clusters with elasticsearch 2. I don't understand because some people want have the latest version. |
Hi! Updated Elasticsearch support is scheduled for Graylog 4.0, which will come out later this year. While we appreciate the interest and concern, I can assure you that we are not ignoring it. That being said, major Elasticsearch upgrades are always a huge deal for users and customers with large clusters, due to the way elastic handles backward compatibility in various APIs, notable mapping changes, and (Lucene) index version support. This presents tremendous challenges for upgrading. At the same time, larger organizations are not as quick to update as y'all might think, because of the time and testing involved, and the considerable amount of resources invested. Long story short, work has already begun for supporting more recent elasticsearch versions and we are again committed (as we were in Graylog 3.0) to make this migration as seamless as possible. Thanks for your support. |
But due to the limited/missing communication we have had no knowledge about your thoughts. In the future I think the community would appreciate some kind of roadmaps for major component support. |
Perhaps too often unsaid: We really appreciate the hard work that you put into a great product. Thanks for clarifying that work has started on ES7 support and what version it will arrive in. I have a (at least perceived) need to use the 7.x series of ES products. Filebeat 6.x doesn't support the "add_fields" processor type. We use that (in Filebeat 7.x) to add fields to messages when a regex catches a specific error pattern. This lets us filter on that new field and send the message to another stream in Graylog. When we use Filebeat 7.x however, a lot of root level fields are listed as "unknown" such as source and name. |
I believe Out of the box we would therefore not have support for it. I’m also not sure the Java libraries we are using can all deal with nanosecond resolution in all the places, so that would be a separate effort for sure. |
Perhaps its wishfull thinkinking. I was hoping for milisecond resolution ( I assume java supports that). And when reasearching this i found date_nano and ES 7 behing "solution" to milisecond resolution on log entires. Thats why i thought these 2 issues might be linked. :) Thank you for taking the time to respond. |
If there is anyone particularly adventurous, the current Update (Oct 2020): The |
I only see hooray, but did anyone tested? LATER EDIT: I was using docker-compose to quickly validate this. It does not work with the latest docker hub image 3.3.5-1. I also attach the compose file.
|
@dinuta as @dennisoelkers said, it's on |
How I can compile graylog? Where I get the master artifact? |
@dinuta probably https://community.graylog.org/t/compiling-graylog-server-for-production/2271 could helps. https://github.com/Graylog2/graylog2-server/blob/master/.travis.yml shows what travis run to build it. |
@dinuta Did you managed to test the latest version supporting ESv7 ? |
Are there no precompiled master artifacts available anywehre, as dinuta already asked for? |
I have compiled the master branch, also modified the Dockerfile if anyone is interested. |
@avolmensky thanks for that!
However there are some errors in the log. Are they "normal"? |
For me it did not work. Graylog does not create indexes in ES. No errors on ES, but saw them on graylog. Keep in mind if you re-create the env to start graylog with delay, otherwise will throw ClientConnectException.
ES 7.8.0 |
@dinuta your log contains almost the same messages as mine. Also your log reports that your graylog is up an running. |
thanks a lot for testing the 4.0 snapshot and reporting back this issue. It helped us to uncover an issue before we are hitting beta. Much appreciated! The cause for this is that we were handling an empty list of indices (what could happen as the result of a race condition on a fresh install) differently between ES6 and ES7. Besides the error on startup there should be no further impact. I have fixed this issue and created a PR for it: #9094. Once it's merged, the error should not happen anymore for fresh installs. If you are seeing any other errors during testing, please open a new issue, that would help us a lot. |
As ES7 support is basically completed, I am now closing this issue. Any further issues/bugs/requests should go to separate issues. Thanks a lot for all of your patience and support! |
@dennisoelkers |
Hey @zez3, we currently support the GIM - Graylog Information Model, which is similar to ECS, but is our own schema, with full mapping to our Graylog Illuminate content. To see the schema, you can visit https://schema.graylog.org |
Since this issue is closed, does this mean there is a version of Graylog available that supports ES7? |
as explained in an earlier comment in this issue it's included for graylog 4 which is planned for release later this year, not graylog 3 |
Expected Behavior
Graylog 3.0 should work with Elasticsearch 7
Your Environment
The text was updated successfully, but these errors were encountered: