New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Elasticsearch to Mesh Docker image #348

Closed
afilimonov opened this Issue Mar 16, 2018 · 7 comments

Comments

Projects
None yet
2 participants
@afilimonov

afilimonov commented Mar 16, 2018

Gentics Mesh Version, operating system, or hardware.

  • v0.17.3

Operating System

  • Linux
  • Docker

JVM

  • Oracle JDK 1.8.0_100
  • Open JDK 1.8.0_100

Problem

Embedded Elasticsearch installation gets extracted from Mesh jar on start if it it not there. If Docker container gets relocated to a different node say by Kubernetes scheduler or Docker Swarm it will force extraction again and the index data is lost.

It is possible to mitigate this by configuring ElasticSearch to use Docker volume for data and configs but it seems that extraction step is redundant.

Reproducer

  1. Start Mesh Docker container with data volume for Mesh data
  2. Stop and remove container
  3. Start new Mesh container with existing Mesh data volume.
  4. It forces Elasticserch reinstall and index data is gone.
  5. You will have to reindex to use Mesh UI.

Expected behaviour and actual behaviour

Elasticsearch instalation is part of Mesh Docker image with default configuration using /config for ES confs and /data for index

@Jotschi

This comment has been minimized.

Member

Jotschi commented Mar 16, 2018

@afilimonov I recommend to run ES in a dedicated container. You can easily set the env:

Another option would be to add another volume for /mesh/elasticsearch. In that case the ES and the data gets preserved and no issue arises.

@afilimonov

This comment has been minimized.

afilimonov commented Mar 16, 2018

Yes dedicated ES is preferred option and that is what we use. However for quick starts or personal dev environment having all in one Docker image is really handy. Mounting/mesh/elasticsearch does work it seems to be un-Dockerish solution. I'd prefere to have an immutable Docker image with everything that is needed included. Also ES doesn't really like to star from a mounted NFS drive you have to bump up ES startup timeout to make it work.

For embedded ES we use custom startup parameters that point to a mounted elasticsearch.yml with index data on the mounted data volume.

@Jotschi

This comment has been minimized.

Member

Jotschi commented Mar 16, 2018

@afilimonov Okay. Dev environment is a good point. I'll see what I can do about the mountpoint. Maybe I can setup the data folder for the embedded ES at a different location and provide a volume (e.g. /searchindex) or something similar.

Keep in mind that Elasticsearch really does not like NFS. We had a lot of trouble with NFS in the past.
See https://getmesh.io/docs/beta/performance.html#_search_index

@afilimonov

This comment has been minimized.

afilimonov commented Mar 16, 2018

How about this in the Dockerfile:

RuN unzip mesh*.jar elasticsearch*.zip && unzip elasticsearch*.zip && rm -f elasticsearch*.zip

/mesh/elasticsearch/data can be mounted if needed

@Jotschi

This comment has been minimized.

Member

Jotschi commented Mar 16, 2018

@afilimonov I like this approach (untested):

ENV ESVERSION 6.1.2
ADD https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ESVERSION.zip /es.zip

ENV MESH_EMBEDDED_ELASTICSEARCH_DIR /elasticsearch
RUN apk --update add unzip \
    && unzip es.zip \
    && mv elasticsearch* elasticsearch \
    && rm es.zip

VOLUME /elasticsearch

That way the ES can quickly be updated.
The MESH_EMBEDDED_ELASTICSEARCH_DIR is not yet supported but I already have an open issue to include such setting.

Jotschi added a commit that referenced this issue Mar 17, 2018

@Jotschi

This comment has been minimized.

Member

Jotschi commented Mar 17, 2018

@afilimonov I added the volume /elasticsearch/data - The change will be released with the next release. Thanks for the feedback.

@Jotschi Jotschi self-assigned this Mar 17, 2018

@Jotschi Jotschi added the f/docker label Mar 17, 2018

@Jotschi

This comment has been minimized.

Member

Jotschi commented Apr 6, 2018

Change has been released.

@Jotschi Jotschi closed this Apr 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment