Handler queued by change of an ES templates file fails to find the file when run #238
Comments
|
@welchwilmerck im curious are you running multiple concurrent threads when running this playbook? My concern is the handler is fired for the first host and this is interfering with the other hosts which don't have the file yet. |
|
Can you also attach your playbook? |
|
Maybe related #166 |
|
@welchwilmerck im going to rework the template code as i believe this is reproducible. |
|
I've changed my playbook considerably and updated to the latest commit of this role since I saw this issue and I can't reproduce ATM. Getting stopped at some other error (below). I'll work through the other errors, but didn't want your attention to this issue to go unacknowledged. For reference, no multiple threads. I'm running just one playbook with the default strategy. task path: /Users/welchwil/gendb_cluster_devops/ansible/roles/elasticsearch/tasks/elasticsearch-config.yml:5 |
|
We're going to re-do the template handing - i see potential issues with it and am suprised its worked for anyone. Are you installing ES via the tar or rpm/deb? The former create the elasticsearch user. If not, you'll need to either:
|
|
fce5eac Merge pull request #269 from gingerwizard/master Currently, install from deb, option 2. Got that part figured out. Now, seeing a different problem in roles/elasticsearch/tasks/elasticsearch-templates.yml. Will continue to troubleshoot. First run, fails on first host and playbook halts immediately. No PLAY RECAP. next run: |
|
I found the problem - an ongoing issue in ansible regarding using include: to have multiple tasks in one handler - ansible/ansible#15915 This silently fails to parse in the version of ansible I'm using (ansible 2.3.0): Renamed issue accordingly. |
welchwilmerck
changed the title
welchwilmerck
changed the title
|
Here's the ansible issue with handler blocks - ansible/ansible#20603. A handler block gets added to the list only if it's the very first handler in handlers/main.yml. To work around, move load-templates to the top of handlers/main.yml and now can reproduce the original issue: To repeat, just go to the target machine and rm /etc/elasticsearch/templates/basic.json so that the handler triggers again. |
welchwilmerck
changed the title
|
I see the bug, now. In the handler, you're using lookup to pull in the contents of a file as the body of a PUT to ES. Unfortunately, lookups run on the local machine, not the remote machine. As I don't have ES installed locally, I get outright 'file not found'. Somebody with a local install with matching template names will PUT their local files, not the ones declared in Ansible. Further, the PUT is a cluster operation, not a per host operation, so the PUT of the template to the API host should run just once for the entire playbook. I can see adapting the idea in http://stackoverflow.com/questions/41556574/run-an-ansible-handler-only-once-for-the-entire-playbook. As the final play only needs to run on localhost, collect the set of changed templates instead of changed hosts. |
|
Here's my implementation of 'one-time play for cluster operations':
ansible-playbook ... -e es_templates_fileglob="{{playbook_dir + '/files/templates/*.json'}}"
- file: path=/etc/elasticsearch/templates state=directory owner={{ es_user }} group={{ es_group }}
- name: Copy example templates to cluster machines unless overridden
copy: src=templates dest=/etc/elasticsearch/ owner={{ es_user }} group={{ es_group }}
notify: load-templates
when: es_templates_fileglob is not defined
- name: Copy locally defined templates instead of examples
copy: src={{ item }} dest=/etc/elasticsearch/templates owner={{ es_user }} group={{ es_group }}
notify: load-templates
with_fileglob: "{{ es_templates_fileglob }}"
when: es_templates_fileglob is defined
- name: Ensure elasticsearch is started
service: name={{instance_init_script | basename}} state=started enabled=yes
- name: Wait for elasticsearch to startup
wait_for: host={{es_api_host}} port={{es_api_port}} delay=10
- name: Flag that something changed in templates
add_host:
hostname: localhost
es_changed_templates: true
---
- name: Set flag for some template changed just as in handlers/elasticsearch-templates.yml
hosts: "{{ es_cluster_name }}"
tasks:
- add_host:
hostname: localhost
es_changed_templates: true
- name: end of management of ES nodes -- run cluster operations one-time
hosts: "{{ es_cluster_name }}"
connection: local
gather_facts: false
# Just in case they're not set elsewhere in vars or group_vars.
# Some have defaults that are set in the elasticsearch role, which isn't activated in this play.
vars:
es_api_host: "0.0.0.0"
es_api_port: 9200
es_enable_xpack: true
es_xpack_features:
- security
tasks:
# ln -s roles/elasticsearch/filter_plugins/ to make filename visible
- name: Install templates without auth
uri:
url: "http://{{es_api_host}}:{{es_api_port}}/_template/{{(item | filename)}}"
method: PUT
status_code: 200
body_format: json
body: "{{ lookup('file', item) }}"
when: (not es_enable_xpack or not es_xpack_features is defined or "security" not in es_xpack_features)
and hostvars['localhost']['es_changed_templates'] is defined and hostvars['localhost']['es_changed_templates']
with_fileglob: "{{ es_templates_fileglob }}"
delegate_to: localhost
- name: Install templates with auth
uri:
url: "http://{{es_api_host}}:{{es_api_port}}/_template/{{ item | filename }}"
method: PUT
status_code: 200
user: "{{es_api_basic_auth_username}}"
password: "{{es_api_basic_auth_password}}"
force_basic_auth: yes
body_format: json
body: "{{ lookup('file', item) }}"
when: es_enable_xpack and es_xpack_features is defined and "security" in es_xpack_features
and hostvars['localhost']['es_changed_templates'] is defined and hostvars['localhost']['es_changed_templates']
with_fileglob: "{{ es_templates_fileglob }}"
delegate_to: localhost
run_once: true
... |
|
The PR #300 should fix this issue. |
|
Closed by #358 |
|
Please, could you help me. I have these commands in ansible: #change setStartEnv.sh to Webcenter sites finds your group.
And it displays the error: TASK [strings : insert Line in setStartEnv to node cluster get group of Rules] *********************************************************************************************************** fatal: [172.16.20.24]: FAILED! => {"failed": true, "msg": "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /app/install/base-cluster/hostname.txt"} PLAY RECAP ******************************************************************************************************************************************************************************* how can I do to run successfully ? |
|
http://docs.ansible.com/ansible/latest/playbooks_lookups.html - Lookups occur on the local computer, not on the remote computer.
Your task is looking for /app/install/base-cluster/hostname.txt on the control machine – the one you’re using to run the playbook, not the remote machine you want to change.
To change remote files, look at ‘File Modules’ like lineinfile or patch.
From: flavioleandro <notifications@github.com>
Reply-To: elastic/ansible-elasticsearch <reply@reply.github.com>
Date: Wednesday, 14 February,2018 at 09:16
To: elastic/ansible-elasticsearch <ansible-elasticsearch@noreply.github.com>
Cc: "Welch, Bill" <william.welch@merck.com>, Mention <mention@noreply.github.com>
Subject: Re: [elastic/ansible-elasticsearch] Handler queued by change of an ES templates file fails to find the file when run (#238)
EXTERNAL EMAIL – Use caution with any links or file attachments.
Please, could you help me.
I have these commands in ansible:
#change setStartEnv.sh to Webcenter sites finds your group.
* name: insert Line in setStartEnv to node cluster get group of Rules
command: "{{item}}"
with_items:
* sed -i "s/satellite_server1/{{ lookup('file', '{{install_dir}}/base-cluster/hostname.txt') }}/g" /app/oracle/domains/SIT_SAT/bin/setStartupEnv.sh
And it displays the error:
TASK [strings : insert Line in setStartEnv to node cluster get group of Rules] ***********************************************************************************************************
[WARNING]: Unable to find '/app/install/base-cluster/hostname.txt' in expected paths.
fatal: [172.16.20.24]: FAILED! => {"failed": true, "msg": "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /app/install/base-cluster/hostname.txt"}
PLAY RECAP *******************************************************************************************************************************************************************************
how can I do to run successfully ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#238 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AR1r7YZqJu7hI3Ltw1LltGIMT0dWjquaks5tUuqjgaJpZM4LoM-L>.
Notice: This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (2000 Galloping Hill Road, Kenilworth,
New Jersey, USA 07033), and/or its affiliates Direct contact information
for affiliates is available at
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from
your system.
|
|
ok, could you help me mount it with the 'File Modules' with remote lineinfile
file in server remoto: Associate server with a startup groupif [ "${STARTUP_GROUP}" = "" ] ; then Thanks, |
The timestamp on basic.json on the problem instance is several minutes prior to the handler activating. Have to run playbook several times (once for each ec2 instance, maybe) before handler sees basic.json for every instance.
Ansible 2.2.0.0
local: mac os
remote: ubuntu 14
ansible-elasticsearch: f8753ff
The text was updated successfully, but these errors were encountered: