Cannot activate Elasticsearch 8.9.1 amd64 in Mastodon v4.2.0-beta1

[DalekDave]

Steps to reproduce the problem

1. Followed the instructions at https://github.com/martindale/mastodon-docs/blob/master/Running-Mastodon/Elasticsearch-guide.md and at https://docs.joinmastodon.org/admin/optional/elasticsearch/
2. Executed the final step of running the deployment command: RAILS_ENV=production bin/tootctl search deploy
3. Got the following command line error output:

mastodon@burnout:~/live$ RAILS_ENV=production bin/tootctl search deploy
/home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [403] {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/read/search] is unauthorized for user [anonymous_user] with effective roles [] (assigned roles [mastodon_search] were not found) on indices [chewy_specifications], this action is granted by the index privileges [read,all]"}],"type":"security_exception","reason":"action [indices:data/read/search] is unauthorized for user [anonymous_user] with effective roles [] (assigned roles [mastodon_search] were not found) on indices [chewy_specifications], this action is granted by the index privileges [read,all]"},"status":403} (Elasticsearch::Transport::Transport::Errors::Forbidden)
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/client.rb:192:in `perform_request'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/elasticsearch-api-7.13.3/lib/elasticsearch/api/actions/search.rb:104:in `search'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/search/request.rb:1026:in `block in perform'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.6/lib/active_support/notifications.rb:206:in `block in instrument'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.6/lib/active_support/notifications.rb:206:in `instrument'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/search/request.rb:1025:in `perform'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/search/request.rb:105:in `response'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/search/request.rb:47:in `wrappers'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/search/request.rb:50:in `to_a'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/search/request.rb:886:in `first'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/index/specification.rb:37:in `locked'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/index/specification.rb:57:in `changed?'
        from /home/mastodon/live/lib/mastodon/cli/search.rb:47:in `block in deploy'
        from /home/mastodon/live/lib/mastodon/cli/search.rb:47:in `select'
        from /home/mastodon/live/lib/mastodon/cli/search.rb:47:in `deploy'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/invocation.rb:116:in `invoke'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor.rb:243:in `block in subcommand'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
        from bin/tootctl:9:in `block in <main>'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy/strategy.rb:60:in `wrap'
        from /home/mastodon/live/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.3/lib/chewy.rb:154:in `strategy'
        from bin/tootctl:8:in `<main>'

Expected behaviour

Deployment command should have completed without error and Elasticsearch should have been working in the Mastodon Web interface (after a Mastodon systemd services restart)

Actual behaviour

Command completed with error output and search is broken in the Mastodon Web interface until you disable it in .env.production

Detailed description

See the technical description below for detailed info.

Mastodon instance

burnout.cafe

Mastodon version

v4.2.0-beta1

Technical details

If this is happening on your own Mastodon server, please fill out those:

• Ruby version: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
• Node.js version: v18.17.1
• Elasticsearch 8.9.1
• Server RAM: 64GB (16GB configured for Elasticsearch in /etc/elasticsearch/ config)

The problem is that the Elasticsearch version provided by the Elasticsearch PPA is a recent version in which Elasticsearch user authentication is enabled by default.

Unfortunately, the Mastodon documentation does not cover how to configure authentication in Elasticsearch. It would be possible either to configure an ES user and password in Elasticsearch and provide the necessary credentials to ES, or to restore anonymous user access in Elasticsearch and use anonymous access from Mastodon.

In the second case, please would it be possible to document how to implement the anonymous access?

In the first case, Mastodon does not seem to have ES configuration parameters such as 'ES_USER' and 'ES_PASSWORD' (plus maybe 'ES_REALM' - a bit like a Redis namespace).

At least none are documented in the Mastodon documentation (although inserting 'ES_USER' does not cause any kind of error message or failure when starting Mastodon).

I'd really like to get full-text searching working in Burnout Café. In the past, quite a while back, with a much-earlier version of Elasticsearch, I've had it working in a previous Mastodon instance.

(P.S. I'd be happy to help document this in the Mastodon github Elasticsearch guide, if we manage to get this issue resolved).

[renchap]

Mastodon does not support Elasticsearch 8 (due to one of our dependency not supporting it, see #26399).

The list of supported version for our stack can be found in the release notes: https://github.com/mastodon/mastodon/releases/tag/v4.2.0-beta1

You are right that the documentation needs to be updated for ES (and also to include ES_PRESET from #26483), feel free to open a PR in our documentation repository to improve it (and ping me in it so I can have a look).

The variables you are looking for are listed in .env.production.sample: ES_USER and ES_PASS

[renchap]

As we need this for the new beta, I updated the documentation with more info: mastodon/documentation#1279

[DalekDave]

Renaud, thank you so much for the very responsive help you've given me over the past couple of days. I'll continue this conversation in the documentation ticket you opened.