Add support for Elasticsearch 5.x

[veger]

In order to be able to install a Elasticsearch cluster using version 5.0.0, I had to add the following v5.x support (next to the other PRs I submitted today and yesterday):

• Use log4j2.properties instead of logging.yml - https://www.elastic.co/guide/en/elasticsearch/reference/5.0/settings.html#logging
• Add jvm.options - https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html#es-java-opts
• Use bootstrap.memory_lock instead of bootstrap.mlockall - https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-configuration-memory.html#mlockall
• Get rid of work directory - Remove working directory elasticsearch#10672
• Executable does not accept -D anymore, replace with -E and -p - --help of executable
• Executable does not accept -- anymore, replace with -E - https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_settings_changes.html#_removed_using_double_dashes_to_configure_elasticsearch
• Get rid of 5.x default (file)s - Similar to other package defaults

⚠️ I am absolutely no Elasticsearch expert (or a experiences user), so I added my references to the list of changes to show why I made the modifications

Additional remarks:

• I tried to keep ES < 5.x compatible, hence the version checks. If support for < 5.x is not required, I can remove the checks
• I do not know about (the/Ansible) role tests, so they probably break... I would appreciate it, if someone is willing to help, or point me in the right direction!

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

[veger]

I misunderstood how version_compare worked, resulting in issues with EX 2.X (when version is larger than 2.0, like 2.3.1).

This is fixed now

[houdejun214]

+1

[houdejun214]

When this can be merged? we need it

[veger]

I do not know, but you can always checkout my branch and us that (same we are doing ATM... ;) )

[jakommo]

@veger Thanks for this PR, very much appreciated.
Before merging this I'd like to run some local tests (as the build test only do 1.x and 2.x), but I haven't had time so far.
I start the build test now anyways as it should at least show if the backwards compatibility is still there.

I tried to keep ES < 5.x compatible, hence the version checks. If support for < 5.x is not required, I can remove the checks

Yes, we need to keep backwards compatibility. 1.x will be EOL in January 2017 but 2.x will stay for quite some time

I do not know about (the/Ansible) role tests, so they probably break... I would appreciate it, if someone is willing to help, or point me in the right direction!

@gingerwizard any resources/tips you could provide?

[jakommo]

Jenkins, test it!

[jakommo]

Ok, good the existing test are all green, so backwards compatibility seems to be fine.

I'm just not sure if we should merge this in master as it could give the wrong impression that it could be used to install 5.0 already.

Just from top of my head and a quick test, these things are missing:

• Repo url changed between 1.x/2.x and 5.x
• ES 5.x requires Java 8 (e.g. not shipped with the default Ubuntu 14.04 repos)
• Enforce values needed for the bootstrap checks if >=5

It should be easy to add once this PR is in, I just fear that some users will see this merged in master and think it's 5.0 ready yet.

@gingerwizard @jpcarey @veger what do you guys think?
Maybe have a branch for 5.0 and merge it to master when everything is in?
I'm also fine with merging it into master if other options are too complicated.

[veger]

Repo url changed between 1.x/2.x and 5.x

I currently override the URL in my play, but I am happy to move it into this PR

ES 5.x requires Java 8 (e.g. not shipped with the default Ubuntu 14.04 repos)

(I use Ubuntu 16.04... ;) )

Easy solution: Add a note in the README.

Hard(er) solution: Add Java 8 repo/ppa if required for OS/version/etc.

Enforce values needed for the bootstrap checks if >=5

I do not know about 'bootstrap checks'... So I cannot help with this.

Maybe have a branch for 5.0 and merge it to master when everything is in?
I'm also fine with merging it into master if other options are too complicated.

I suppose the maintainer(s) of this repo should decide what is best. I am fine with both.

[GaneshbabuRamamoorthy]

Hi @veger

I had a little doubt in setting up ES 5.0 and I am using your git branch (fix_es5_issues) to start elasticsearch and I am getting the below error,

fatal: [10.247.0.5]: FAILED! => {"changed": true, "cmd": ["sudo", "service", "master_node_elasticsearch", "start"], "delta": "0:00:00.054053", "end": "2016-11-14 05:09:33.621534", "failed": true, "rc": 7, "start": "2016-11-14 05:09:33.567481", "stderr": "", "stdout": "MAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set", "stdout_lines": ["MAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set"], "warnings": ["Consider using 'become', 'become_method', and 'become_user' rather than running sudo"]}

Though I have given the es_heap_size value in my playbook but still getting the error.

I have attached the my playbook for your reference and somewhere I am doing mistake & pls correct me rdaas-elasticsearch.zip

Note:- We are in developing stage of setting up ES 5.0.0

[veger]

You did not specify which OS you are using. I only tested it with Ubuntu 16.04 (using systemd service manager).

The error originates from either the Debian or Redhat init.d service file. So I guess that is untested...

2 Choices (I am no ES expert so I do not know which is better):

1. The templates/elasticsearch.j2 file contains a line to set ES_HEAP_SIZE, but it is only added for ES < 5.x as I understood that this is obsolete for ES 5.x. It can be re-enabled again for all ES versions
2. Remove to check from templates/init/debian/elasticsearch.j2 and templates/init/redhat/elasticsearch.j2 when ES 5.x is used.

For now I would say, try both and see whether either or both are working for you. (Please report your findings ;) )

It would be great if a ES expert could give his opinion which one is better! So the PR can be updated!

[Jazznight]

@veger, base on the document of jvm.options.

The default location of this file is config/jvm.options (when installing from the tar or zip distributions) and /etc/elasticsearch/jvm.options

In case when set es_instance_name variable will move this file into {es_config_dir}/{es_instance_name} so that all the java options will not be read by elasticsearch wrapper by default as mention in the document.

[veger]

@Jazznight so you are basically saying that jvm.options must be installed in es_conf_dir (which should default to /etc/elasticsrearch) instead of conf_dir (which defaults to /etc/elasticsearch/)

Now that I think about it, it might be possible that I create the jvm.options myself later (in my playbook)... I'll check when I am back at work tomorrow!

[Jazznight]

Hi @veger ,

I'm setting multiple node in one host so that I'll have multiple instances in es_conf_dir . Base on the default setting i thought conf_dir will point to es_conf_dir/es_instance_name.

I think conf_dir is good place for the jvm.options since I can define jvm.options for different node but I don't think the current elasticsearch wrapper support it.

[GaneshbabuRamamoorthy]

Hi @veger

As per your 2 choices I have tried both the settings but its unable to start ES either I am getting one of the error during the playbook run,

fatal: [10.247.0.5]: FAILED! => {"changed": true, "cmd": ["service", "master_node_elasticsearch", "start"], "delta": "0:00:00.050567", "end": "2016-11-17 01:26:21.900953", "failed": true, "rc": 7, "start": "2016-11-17 01:26:21.850386", "stderr": "", "stdout": "/usr/share/elasticsearch/bin/elasticsearch\nMAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set", "stdout_lines": ["/usr/share/elasticsearch/bin/elasticsearch", "MAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set"], "warnings": ["Consider using service module rather than running service"]}

TASK [elasticsearch : Start elasticsearch service] *****************************
task path: /home/siddharth/myansible/playbook/ansible-elasticsearch-5.0/tasks/elasticsearch-service.yml:21
basic
changed: [10.247.0.5] => {"changed": true, "cmd": ["service", "master_node_elasticsearch", "start"], "delta": "0:00:00.192677", "end": "2016-11-17 01:04:45.200027", "rc": 0, "start": "2016-11-17 01:04:45.007350", "stderr": "Error: Could not find or load main class g", "stdout": "/usr/share/elasticsearch/bin/elasticsearch\nStarting elasticsearch: [ OK ]", "stdout_lines": ["/usr/share/elasticsearch/bin/elasticsearch", "Starting elasticsearch: [ OK ]"], "warnings": ["Consider using service module rather than running service"]}

Let me know your suggestions.

Hi @Jazznight

I am trying the same by setting up multiple node in the same host. so I defined the conf_dir path as /etc/elasticsearch and so jvm.options file will copied to the same.

But I am getting the below error when its trying to up the ES service

fatal: [10.247.0.5]: FAILED! => {"changed": true, "cmd": ["service", "master_node_elasticsearch", "start"], "delta": "0:00:00.050567", "end": "2016-11-17 01:26:21.900953", "failed": true, "rc": 7, "start": "2016-11-17 01:26:21.850386", "stderr": "", "stdout": "/usr/share/elasticsearch/bin/elasticsearch\nMAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set", "stdout_lines": ["/usr/share/elasticsearch/bin/elasticsearch", "MAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set"], "warnings": ["Consider using service module rather than running service"]}

When I debug the elasticsearch file under bin folder & I could see its expecting jvm options file in any of the below path

if [ -z "$ES_JVM_OPTIONS" ]; then
for jvm_options in "$ES_HOME"/config/jvm.options
/etc/elasticsearch/jvm.options; do
if [ -r "$jvm_options" ]; then
ES_JVM_OPTIONS=$jvm_options
break
fi
done
fi

Let me know your thoughts..

[Jazznight]

@GaneshbabuRamamoorthy Looks like elasticsearch wrapper still not found the jvm.options

base on the veger:fix_es5_issues it will remove /etc/elasticsearch/jvm.options in the last step so that i don't think it will work if you define conf_dir

[GaneshbabuRamamoorthy]

@Jazznight Ya I think so its not working even I changed the conf_dir to this path "$ES_HOME"/config/jvm.options* but still elasticsearch service failed to start up.

@veger @jakommo Pls let me know your thoughts.

[Jazznight]

@GaneshbabuRamamoorthy FYI, I just modified and made another fork. It's working for my case which have multi-node in same host deployment.

[veger]

It has been a while since I made these changes, and I have been making loads of Ansible roles last couple of weeks. So things a bit blurred... :/

I do know it is working for Ubuntu 16.04/systemd, you guys still have not said which OS/service manager/Elasticsearch version you are targeting.

The role installs jvm.options in es_conf which is the conf dir for a specific node (/etc/elasticsearch/<node name>) and explicitly removes the package jvm.options in /etc/elasticsearch/.

Using -Edefault.path.conf=$CONF_DIR the config directory is given at startup (fo bothr systemd and initd (Debian & Redhat)), but I only checked systemd.
Maybe something is wrong here? (e.g. the config directory is not properly provided to ES, hence using the explicitly removed files in /etc/elasticsearch are required)

Again, please provide these details (maybe even include some of the generated files), so we can get a better view what is going wrong..?

[GaneshbabuRamamoorthy]

Hi @veger @Jazznight

I forked ansible elasticsearch project in which I am trying to setup ES 5.0 in Linux machine. ES5.0 setup

use the rdaas-elasticsearch.yml (playbook) and check you able to up the ES 5.x cluster with that settings.

Please kindly let me your thoughts and it would be very helpful.

Note:- @Jazznight I forked your project with your recent changes but still unable to start the ES cluster

Regards,
Ganeshbabu R

[GaneshbabuRamamoorthy]

Hi @veger

Did you get any chance to try and up ES 5.x using my playbook?

I am trying to setup in ES 5.x (Red Hat Enterprise Linux Server release 6.7 (Santiago))

[veger]

Thanks for telling which OS you use, it helped with finding the/a issue.

I have added a new commit which disabled the check.
ElasticSearch 5.X uses the java options Xms and Xmx:

I use in my playbook:

    es_java_opts:
    - "-Xms{{ es_heap_size }}"
    - "-Xmx{{ es_heap_size }}"

But maybe it needs to be included in the role?

(I did not try your playbook, as I do not have access to a RedHat machine ATM)

[GaneshbabuRamamoorthy]

Hi @veger @jakommo

I tried the same settings in my playbook and after completion of playbook I could see elasticsearch_started message but ES is not up & running

changed: [10.7.147.22] => {"changed": true, "cmd": ["service", "master_node_elasticsearch", "start"], "delta": "0:00:01.113344", "end": "2016-11-22 06:20:09.870126", "rc": 0, "start": "2016-11-22 06:20:08.756782", "stderr": "", "stdout": "Starting elasticsearch: [ OK ]", "stdout_lines": ["Starting elasticsearch: [ OK ]"], "warnings": ["Consider using service module rather than running service"]}

I am unable to figure out the issue but I suspect ES is not able to find the log4j properties because after ES installation I manually tried to start the ES (./elasticsearch) in the bin folder (usr/share/elasticsearch/bin) and I am getting the error,

Warning: Ignoring JAVA_OPTS= -Xms16g -Xmx16g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8
Please pass JVM parameters via ES_JAVA_OPTS instead
log4j:WARN No appenders could be found for logger (bootstrap).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.io.FileNotFoundException: /usr/share/elasticsearch/lib/elasticsearch-1.7.2.jar (No such file or directory)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:219)
        at java.util.zip.ZipFile.<init>(ZipFile.java:149)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:103)
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:174)
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:87)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:175)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:267)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
        at org.elasticsearch.cli.Command.main(Command.java:53)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)

Do I need to do any configuration change related to log4j properties?

Attached my playbook with the latest changes..
ansible-elasticsearch_latest.zip

[veger]

@GaneshbabuRamamoorthy

I am not a support forum, but I am merely trying to get a PR accepted (which is not going to happen at this rate, since the maintainer probably does not like to read through all of this mess, at least I would not if I would have been the maintainer). Please do not dump a complete project but provide a minimal example, I am not going to read all files (let alone install VM(s) and run your playbook) in order to try and figure out what is happening (especially since I suspect that this has nothing to do with this PR, but more about not reading ES/Ansible/YML/etc documentation). I am no expert in ES and/or Ansible and/or Redhat, and I simply do not have the time to read the Internet in order to find out what is happening with your setup.

That said:

When looking at your provided log, I see a big exception telling

Exception in thread "main" java.io.FileNotFoundException: /usr/share/elasticsearch/lib/elasticsearch-1.7.2.jar (No such file or directory)

Did you check if this file indeed is not available? If so, it has nothing to do with this PR as it does not change installation locations (using apt/debian package, the binaries are installed in that location and stay there).

About log4j.properties (which are 'merely' a warning): I have not idea what you are doing/using due to lack of details. Are you still using you own branch, or the one of this PR? In your own branch the location of log4j.properties is changed... Maybe this is the reason that they cannot be found?

Again, please try to find a more suitable setting to get help to install ElasticSearch using Ansible, this PR is polluted already as it is...

[pdaugavietis]

I'm going to be starting on ES on RHEL7 now, and would like to use 5.0 versions. I can also help with this if needed from that perspective. Can we perhaps merge this into a 5.0 branch for now, and work off that? Or just work off veger's branch?

[jakommo]

@pdaugavietis Yes, I just created a 5.x branch, but I can't change it on an existing PR.
@veger mind creating a new PR against the 5.x branch?

[veger]

Added 2 PRs #206 and #207, both fixes for 5.x. I think this is all for 5.x support, if you are missing other stuff, please check my (open) branches and request a PR for the 5.x branch as well!

Hopefully, we'll be able to confirm this is all working in different situations and get the support merged back into the master branch! :)

[jakommo]

closing in favor of #206

[GaneshbabuRamamoorthy]

Hi @pdaugavietis

Check out the branch v5.x
https://github.com/GaneshbabuRamamoorthy/ansible-elasticsearch/tree/v5.x

We able to setup ES 5.0 cluster in Redhat with the help of @veger PR

Hope you able to setup the same !!