Add support for new Elasticsearch versions

[Flole998]

As Elasticsearch is vulnerable to the log4j exploit it is necessary for secure operations to upgrade to version with a fix. However, according to the documentation Elasticsearch versions 7.11 or later should not be used with Graylog. In order to ensure secure operations please support newer, secure versions of Elasticsearch.

[loganmarchione]

Couple things:

1. Elastic says:

"Elasticsearch is not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager."

2. Graylog uses the open-source version of Elasticsearch, called elasticsearch-oss. This has a different license, which itself might not be compatible with Graylog's license. Also, it appears that elasticsearch-oss stopped development at v7.10.2 (at least according to their docker repos). My point is, upgrading Elasticsearch might not be an easy fix...

[Flole998]

Just because remote code execution is not possible it is still vulnerable to information disclosure caused by this vulnerability.

The license does not restrict how the server can be used/which software is allowed to call the endpoints. Also that change was introduced with 6.3 already, so if this hasn't been an issue since 3 years, why now?

You seem to only look at docker there, but just because no docker image exists that doesn't mean that the software doesn't exist.

[xoxys]

It's not only this special issue with log4j, every other bug or future security issue will not be back ported by Elastic and Graylog users are forced to Elastic 7.7.x-7.10.x which is a bit annoying...

Besides that, the documentation for the supported ES versions is confusing at all... The latest Graylog release it 4.2.x and not 4.0.x and there is absolutely nothing in the docs https://docs.graylog.org/v1/docs/elasticsearch

[xoxys]

"Elasticsearch is not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager."

That doesn't even make it better, especially as there is a permanent fix available in 7.16.1 which Graylog users can't apply....

[bendem]

A plan is needed (switch to something else, make it compatible, archive graylog and make it legacy), if the project decides to freeze elasticsearch to 7.10, it is basically a dead project, you cannot decide to stop updating a critical part of your infrastructure that's actively developed like it's no big deal.

The users have a right to know if they still have to invest time and energy in something that's not willing to work with up to date versions of its critical components, or if they are better off looking for alternatives.

It's been almost a year since the version was frozen, what's the plan?

[bendem]

Also, this is a duplicate of #11686

[kroepke]

We will announce a plan shortly with regards to Elasticsearch compatibility and support going forward.
Thanks for your patience.

[1tft]

@kroepke We are waiting for this information, because it decides should we upgrade to ES 7.10 or better wait with upgrade for official OpenSearch or Elasticsearch OSS support or anything else...
In times of Log4Shell and many more vulnerabilities we have extreme trouble to explain somebody, that we cant update our base components. We are using Graylog Enterprise, so it should act like enterprise software.

[KlavsKlavsen]

opensearch just released 1.2.1 which has updated log4j - and since it forked at v7.10 - it SHOULD "just work" - with graylog?

[kroepke]

OpenSearch works, but you might need to put it into compatibility mode to report itself as elasticsearch 7.10: https://opensearch.org/docs/latest/clients/agents-and-ingestion-tools/index/ and the current master (which will become Graylog 4.3) also will contain direct support for it: #11435

[AnonymousWP]

I very much support this request. Yesterday I tried using ElasticSearch 7.16.1 in combination with Graylog 4.2.3, but that may have caused issues, so I'm back at the last version of ElasticSearch-OSS again (which its latest release is from January 2021 if I recall correctly...) Couldn't really find a reason why Graylog chose ES-OSS to begin with, but probably because of some extra features (https://medium.com/codex/implementing-security-in-elasticsearch-oss-distribution-using-open-distro-security-plugin-d1d106e62ca6).

[leowinterde]

We will announce a plan shortly with regards to Elasticsearch compatibility and support going forward. Thanks for your patience.

A decision here continues to be critical to the use of Graylog as a reliable logging solution.

What is the planned for the old Elasticsearch dependency? Using OpenSearch could be a solution, will there be an official announcement or even a migration path - what is planned? @kroepke

With Elasticsearch 6.8.22 (last patch with Log4j 2.17.0) we are also running towards EOL on 2022-02-08, same goes for Elasticsearch 7.10.x on 2022-05-11.

[KKeXX]

Are there any news on this?
The announcement about a plan has been posted over a month ago. @kroepke in #11804 (comment)

Whats the plan for ES in the future?

[PacifistRiot]

Any news yet on the future of ES for graylog?

[coffee-squirrel]

They posted this on March 10th: https://www.graylog.org/post/graylog-to-add-support-for-opensearch

[xoxys]

What the heck? So the result, after waiting for months to get any feedback, is to remove Elasticsearch entirely? Nice one...

[loganmarchione]

What the heck? So the result, after waiting for months to get any feedback, is to remove Elasticsearch entirely? Nice one...

Ehhh. Graylog is in a tough place due to Elastic licensing and no more updates for Elastic OSS. This is the clearest path forwards, right?

[Flole998]

No they aren't: The license does not restrict how applications may interface with an elastic search instance. In fact it looks like all the elasticsearch related code was written for this project and no dependencies were used, so the license of elasticsearch doesn't even matter. It's like saying graylog may only receive log data from applications under a certain license....

In my opinion the reason this isn't done is the effort required, and blaming it on the license provides an excellent excuse. Of course I could be wrong, but in that case please show me where graylog uses code from elasticsearch which can not legally be used under the new license so that I can understand why my point of view is not correct.

[JTabel]

No they aren't: The license does not restrict how applications may interface with an elastic search instance. In fact it looks like all the elasticsearch related code was written for this project and no dependencies were used, so the license of elasticsearch doesn't even matter. It's like saying graylog may only receive log data from applications under a certain license....

In my opinion the reason this isn't done is the effort required, and blaming it on the license provides an excellent excuse. Of course I could be wrong, but in that case please show me where graylog uses code from elasticsearch which can not legally be used under the new license so that I can understand why my point of view is not correct.

The license does restrict usage in SaaS environments however:

You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software.

From https://www.elastic.co/de/licensing/elastic-license

Now this wouldn't be a problem for self-hosting, as most users probably have a private instance, not public and using elasticsearch in that context is usually not problematic. However, with graylog cloud this becomes a different topic, as elasticsearch is basically offered as a service and that is prohibited. So graylog really is in a tough spot: support elasticsearch for on-prem private users or go forward with their SaaS strategy, which brings them probably more money. I think supporting both isn't a feasible way forward, as the two products will diverge in the future even more than they already did. So while I can understand the decision (and I expected it already), I am still not happy with it.

[kroepke]

Hi everyone!

Sorry for not posting the blog post here, too, my poor excuse is that I was away and then forgot to do it earlier this week.

That being said, some of the reasons have been touched upon in the recent comments, but I wanted to expand on them a bit.
The license situation poses problems for us, but also for some of our customers because they provide services to others.
Due to the way the Elastic v2 license is worded, you really cannot use binaries licensed in that way. Unfortunately, it's the only way official binaries are distributed, so either you have to accept that uncertainty or you build everything from scratch yourself under SSPL.
Neither is a great option for us and when it comes to licenses you need to be sure you comply, doubly more so when you take money for services or products.

For that reason, we stuck with the last Apache 2.0 licensed artifacts. Elastic has stated that the majority of the downloads had been Elastic-licensed for a long time (both 1.0 and 2.0, I guess), but that's just due to the fact that the majority of users never understood that they were actually downloading and running commercially licensed binaries. Graylog has always stuck to the -oss packages. Those are simply not available for SSPL licensed code. What's more, elastic's source trees contain code that is only licensed under Elastic v2 mixed with dual-licensed code, which creates yet another nightmare for building your own artifacts, because you need to make very sure you don't accidentally publish something that you don't have a license for.

Long story short, the only viable alternative is OpenSearch, which is why we have added direct support for it. We have not removed support for Elasticsearch 6.8 or 7.10 at this point, and we will support running with Elasticsearch 7.10 for as long as it is technically feasible.

Those are just the reasons for the status quo, if we ever want to extend or modify behavior we are in a dead-end with elastic's codebase, which is not a great place to be. Seeing that OpenSearch has a good set of important features that were previously unavailable to use (since we stayed away from all x-pack code), we will be adding direct support for this over the next few versions, too.

OpenSearch is Apache 2.0 licensed, which we feel is important for such a component in our stack, and as others here have said is the logical step.
We are committed to providing products for all of our users, whether they are on the free open version, a self-managed subscription, or our cloud service, but we cannot support three different backends forever, that is simply unsustainable for us, so we had to make a choice.

Thanks again for the civil tone everyone!

[loganmarchione]

FYI for everyone here. OpenSearch support was added today in v4.3.0.

https://docs.graylog.org/docs/changelog

[boosty]

As we do not plan to offer support for Elasticsearch past v7.10, we are closing this issue.

The details have been written down by @kroepke in the comment above, and in this blog post:
https://www.graylog.org/post/graylog-to-add-support-for-opensearch