Skip to content

Set Up Elasticsearch with MongoDB (feature elasticsearch)

Jump to bottom
Irfan edited this page Apr 10, 2022 · 1 revision

⚠ DO NOT use this guide. This is for the feature/elasticsearch branch only which is currently under development.

How To Set Up Elasticsearch with MongoDB For Jikan REST API

This guide is specifically for whatever the hosted version of the Jikan REST API uses.

ℹ This guide is as-is and will not provide any additional support on alternative distros or Elasticsearch versions. If you feel that any important context is needed, a PR would be great.

Usage Version
Elasticsearch 8.1
Distro Ubuntu 20

Why Elasticsearch?

Jikan v4 uses an internal database, MongoDB, for querying indexed entries. This helps reduce connections to MyAnimeList and keeps the service in a healthy state. However, a self-managed MongoDB instance (not MongoDB atlas) does not come with an advanced Fulltext search engine with much-needed features like the Lucene engine.

tl;dr search sucks.

Supported Endpoints

Only the following search endpoints support Elasticsearch.

Endpoint Support
Anime ✅ Fully Supported
Manga ✅ Fully Supported
People ✅ Fully Supported
Characters ✅ Fully Supported
Users ✅ Fully Supported

How It Works

The collection entries from the supported endpoints are ingested by Elasticsearch via Monstache

1. Install Elasticsearch

  1. Install Elasticsearch

Alternatively, for a quick reference; here's what I did:

  1. Install Java prerequisites
apt update && apt upgrade && apt dist-upgrade
apt install default-jre
java -version
apt install default-jdk
javac -version
  1. Install Elasticsearch
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
sudo apt-get install apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install elasticsearch

At this point, you will receive a generated Elasticsearch password in a message that looks like this.

  1. Restart stuff and enable for it to startup on boot
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
  1. Confirm it all works
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200
  1. Copy Elasticsearch certificates and enable them
cp /etc/elasticsearch/certs/http_ca.crt /usr/local/share/ca-certificates
cp /etc/elasticsearch/certs/http.p12 /usr/local/share/ca-certificates
cp /etc/elasticsearch/certs/transport.p12 /usr/local/share/ca-certificates
sudo update-ca-certificates
  1. Restart the service
sudo systemctl restart elasticsearch.service
  1. (optional) If your Elasticsearch instance and MongoDB server are on the same server, you can skip this. If not, then you might want to expose Elasticsearch.

Change the following values in /etc/elasticsearch/elasticsearch.yml

network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: [0.0.0.0]
  1. Enable the ingest-attachement plugin on Elasticsearch
cd /usr/share/elasticsearch
bin/elasticsearch-plugin install ingest-attachment
sudo systemctl restart elasticsearch.service

2. Install Monstache

Monstache is "a go daemon that syncs MongoDB to Elasticsearch in realtime.". You can use whatever connector you like, this is the one I picked out to use for Jikan.

  1. Download the (latest release zip file](https://github.com/rwynn/monstache/releases)

👉 Currently using Monstache v6.7.7

cd ~
wget https://github.com/rwynn/monstache/releases/download/v6.7.7/monstache-2d437b2.zip
apt install unzip -y
unzip monstache-2d437b2.zip
cd build/linux-amd64

  1. Copy the provided Monstache config.toml to your Monstache folder.

  2. Run it to ingest the initial content and to troubleshoot any errors:

monstache -f config.toml
  1. Configure config.toml as per your needs or keep the default jikan behavior. Remember to change the following to whatever your configs are:
mongo-url = "mongodb://someuser:password@localhost:40001"
elasticsearch-urls = ["https://es1:9200", "https://es2:9200"]
elasticsearch-user = "someuser"
elasticsearch-password = "somepassword"

For advanced configuration, view Monstache docs.

3. Install Supervisor

Supervisord will manage Monstache to continuously run in the background as a service.

⚠ WIP Docs

4. Enable Elasticsearch in Jikan

  1. Set ELASTICSEARCH=true in the .env file
  2. Configure Elasticsearch .env for Jikan

✨ Enjoy better search ✨

The supported endpoints will automatically switch over from the default MongoDB controller to the Elasticsearch controller.

And if you want to temporarily switch back to MongoDB (or vice versa), simply toggle ELASTICSEARCH=false in .env and restart the HTTP server so the environment variable changes so the other controller is used instead.

Clone this wiki locally