Install hangs on [Wait for elasticsearch to startup] Ubuntu 16

[runtman]

It seems the installation of elasticsearch via this ansible playbook hands on [Wait for elasticsearch to startup].

If I was to take a stab in the dark, I believe it relates to systemd missing the startup.

bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-iolgltkwkkyiyzdtbcvhnhnejczxkbup; /usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1494493145.89-115538618345757/wait_for.py; rm -rf "/home/ubuntu/.ansible/tmp/ansible-tmp-1494493145.89-115538618345757/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"'' <54.77.108.183> (0, '\r\n{"msg": "Timeout when waiting for localhost:9200", "failed": true, "elapsed": 300, "invocation": {"module_args": {"active_connection_states": ["ESTABLISHED", "SYN_SENT", "SYN_RECV", "FIN_WAIT1", "FIN_WAIT2", "TIME_WAIT"], "host": "localhost", "port": 9200, "delay": 5, "state": "started", "sleep": 1, "timeout": 300, "exclude_hosts": null, "search_regex": null, "path": null, "connect_timeout": 1}}}\r\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /home/anthony/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 15368\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 54.77.108.183 closed.\r\n') fatal: [54.77.108.183]: FAILED! => { "changed": false, "elapsed": 300, "failed": true, "invocation": { "module_args": { "active_connection_states": [ "ESTABLISHED", "SYN_SENT", "SYN_RECV", "FIN_WAIT1", "FIN_WAIT2", "TIME_WAIT" ], "connect_timeout": 1, "delay": 5, "exclude_hosts": null, "host": "localhost", "path": null, "port": 9200, "search_regex": null, "sleep": 1, "state": "started", "timeout": 300 } }, "msg": "Timeout when waiting for localhost:9200" }

[runtman]

I added this dirty hack to get around it, in main.yml
- name: Debian - Ensure it's enabled on startup lineinfile: path: /etc/default/{{ es_instance_name }}_elasticsearch regexp: '^START_DAEMON=' line: 'START_DAEMON=true' notify: restart elasticsearch

[runtman]

[bpartridge]

While that didn't work for me, a simpler version that simply ensured it was started in the first place before waiting worked for me, to insert at the same location.

- name: Debian - Ensure it's enabled on startup
  service: name={{instance_init_script | basename}} state=started enabled=yes

I actually think that the absence of this is a bug. It seems we notify a restart of elasticsearch on a change of config. But if there's a misconfiguration or some step fails AFTER the change was made, a rerun of the script may not change the (now properly uploaded) config, and so we'd wait without the service up. Since state=started is a noop if it's already up, this seems like a decent sanity check.

[masterjus]

both solutions doesn't work for me :(

[ericdowell]

Using Ansible 2.3.1, Vagrant 1.9.5, Virtualbox 5.1.22 and attempting to run in the bento/ubuntu-16.04 vagrant box this issue still occurs.

Neither of the solutions listed work.

It would be great if there was a fix for this.

@jakommo @gingerwizard

[wassim]

@masterjus @ericdowell Try to decrease memory to half your server's memory size (es_heap_size). Worked for me.

{ role: elasticsearch, es_instance_name: "node1", es_heap_size: "256m", tags: [elasticsearch] }

[ericdowell]

@wassim that did the trick. Thanks a lot!

[gingerwizard]

Reproducible and linked to #315 i think. Investiating,

[barryib]

I think that there 2 mains problems:

1. Why Elasticsearch is not starting?
Elasticsearch won't start if there are some "human" misconfigurations. There are many reasons for that, but reading logs from /var/log/elasticsearch or journarctl help a lot. Generally speaking, bad heap size is a very common mistake.

2. The role will not trigger handlers if config_updated hasn't changed
When Ansible try to restart the first time (here) and hang (here) because of Java misconfiguration, you can't fix this with ansible-elasticsearch. Right now, you have to:

• fix it and restart your nodes manually
• or make changes on elasticsaerch.yml to let Ansible restart your nodes

When you read the elasticsearch-config.yml, you notice that you have handlers on lot of tasks, but only one with config_updated registered.

In the hanler, Elasticsearch will restart only if all of these following statement are True

• es_restart_on_change is set to True
• es_start_service is set to True
• plugin_installed.changed OR config_updated.changed OR debian_elasticsearch_install_from_repo.changed OR redhat_elasticsearch_install_from_repo.changed OR elasticsearch_install_from_package.changed

It means that, even you fix your java or default file configuration, you will always hang because ansible will not trigger Elasticsaerch restart to apply your changes.

I don't know if I'm really clear, but I think that we don't need to need to register config_updated or to take decision on plugin_installed.changed OR config_updated.changed OR debian_elasticsearch_install_from_repo.changed OR redhat_elasticsearch_install_from_repo.changed OR elasticsearch_install_from_package.changed. We have to let ansible trigger handler when files change.

cc @gingerwizard

[runtman]

Hey Barryib, you are absolutely correct and if the process fails to start due to misconfiguration, you can't just pop up and try to use autocomplete on the service name as it won't recognise it.

I wanted to implement SearchGuard with my set up, so I hacked the living crap out of this role but at one point to try and move away from the dreaded failed service (then the rest of the role doing nothing) I added in a manual "service blah restart". It's a dirty hack I know.

The reason I am responding here is because I feel you are in the correct direction in order of fixing this through my experience when modifying the role.

[gingerwizard]

This logic was added to minimise restarts. The result is unfortunately i don't think we restart when we need to. Annoyingly this is really difficult to reproduce with a test as its effectively the result of a user mistake - which we then need to make sure we can get out of.

We did add this https://github.com/elastic/ansible-elasticsearch/blob/master/tasks/main.yml#L47
to ensure ES was started.

If we remove those conditions @barryib highlights and the var config_updated it has a side effect. We don't currently notify the handler if other places e.g. here - so we won't restart ES on an upgrade. We will therefore need to notify in these tasks as well.

I think the https://github.com/elastic/ansible-elasticsearch/blob/master/tasks/main.yml#L47 entry here should also be idempotent and is necessary in case of a failure between the config applying and the handler firing.

If my logic is right i can make a PR and test this. @barryib