OpenSearch support (Elasticsearch fork)

[killerbees19]

Steps to reproduce the problem

1. Install OpenSearch instead of Elasticsearch.
2. Enable Elasticsearch: ES_ENABLED=true
3. Go to admin dashboard.

Expected behaviour

No warning message.

Actual behaviour

Incompatible Elasticsearch version: Elasticsearch X.Y.Z is running while 7.x is required

Specifications

OpenSearch is a fork of Elasticsearch.

It seems v1.3.1 (and even v2.0.0) are fully supported by Mastodon (v3.5.1 - v3.5.3), but the version check at the admin dashboard is still annoying:

Considerations

Is OpenSearch really supported by Mastodon?

• Yes: Update version check in file elasticsearch_check.rb (if possible)
• No: Make it happen in the future 😁

Details

Maybe the relevant data is available at Chewy.client.info['version']['distribution']?

$ curl localhost:9200

{
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

(Irrelevant lines removed.)

[kescherCode]

It works fine - I suppose the warning is purely visual.

[kescherCode]

On OpenSearch 1.x, you can set compatibility.override_main_response_version in opensearch.yml to silence this warning. On 2.x, this setting has been removed though.

[kescherCode]

What's more: Both ElasticSearch and OpenSearch currently return minimum_wire_compatiblity_version and minimum_index_compatiblity_version when accessed at /. For OpenSearch 2.0.0:

{
...
  "version" : {
...
    "number" : "2.0.0",
...
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  ...
}

Perhaps these should be checked instead of merely the running version.

[marians]

I would definitely recommend to move to OpenSearch officially. Elasticsearch as configured here in the docker-compose setup of v4.0.0rc2

https://github.com/mastodon/mastodon/blob/v4.0.0rc2/docker-compose.yml#L28

is not even an Open Source (OSS) image. Also the way how these images are created and maintained is completely intransparent and have been for a while.

In addition, instead of promoting an open ecosystem, Elastic (the company) tries to create a walled garden where only certain clients interoperate with certain versions of their software. As an example, their official Go client (https://github.com/elastic/go-elasticsearch) as of v7 does not interoperate with the elasticsearch-oss v7 server.

Here is the server info for a server running docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2-amd64:

{
  "name" : "4c2875dfc572",
  "cluster_name" : "mastodon",
  "cluster_uuid" : "OMtR9IvgSdasx8k69rv4Nw",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

And here is the error the Go client produces when connecting to this server: the client noticed that the server is not a supported distribution of Elasticsearch.

I'm certain that it is in the interest of the Mastodon project to move to the openly licensed fork by AWS that is OpenSearch.

[abdessalaam]

Is there a way to access OpenSearch Dashboards? Trying to go (after insatallation) to https://example.org:5601 doesn't work...

[VyrCossont]

I fixed the admin check in #22422 to check against the OpenSearch ES wire compatibility version instead of the version of OpenSearch itself. Note that the check doesn't actually prevent Mastodon instances from operating with an OpenSearch cluster.

Compatibility-wise, the currently shipping OpenSearch 2.4.0 is compatible with ES 7.10 and seems to handle everything Mastodon demands of it. I'd like to second or third or fourth the suggestion that Mastodon Docker images should start using OpenSearch instead of mainline proprietary ES, and that Mastodon shouldn't start using any future ES features that would break compatibility with OpenSearch.