failing lookup affects copy/template/file even when not part of the task

[evoracer]

Issue Type:

Bug

Ansible Version:

ansible 1.8.2

Environment:

debian 7 / gentoo

Summary:

The copy and template modules throws an error with a wrong file.
In ansible 1.7.x everything is fine and ansible copies the jail.conf.
In ansible 1.8.2 i cannot copy any file while i have a lookup defined in group vars. Even if the defined lookup is not used in any playbook

Steps To Reproduce:

group_vars/all

ssh_users:

• name: evo

  key: "{{ lookup('file', 'evo.pub') }}"

tasks/main.yml

• name: copy fail2ban config

  copy: src=jail.conf dest=/etc/fail2ban/jail.conf

Expected Results:

ansible 1.7.x

TASK: [common-j | copy fail2ban config] ************************************
changed: [evosrv.test]

Actual Results:

ansible 1.8.2

TASK: [common-j | copy fail2ban config] ************************************
fatal: [evosrv.test] => could not locate file in lookup: evo.pub

[evoncken]

Appears I'm having the exact same issue. Same playbook used to work, now aborts early in "ansible-playbook". CentOS 6.6 X86_64 with all updates applied.

ESTABLISH CONNECTION FOR USER: root
fatal: [c3p0] => could not locate file in lookup: ssh_pubkeys/root.pub

Version history:
ansible-1.8.2-1.el6.noarch << ansible-1.8.1-1.el6.noarch << ansible-1.7.2-2.el6.noarch

I'm 99% sure that the same Ansible playbooks worked with 1.8.1.
Cannot "yum downgrade", unfortunately, to be 100% sure.

[rickmb]

I have a similar issue, with a completely unrelated lookup in group_vars/all suddenly blowing up whilst doing a synchronize. Downgrading to 1.8.1 fixes the issue. (This has worked fine in many earlier versions.)

[evoncken]

Did some experimenting:
commit e938f55 - FAIL
commit 2d17d18 - OK

When checking out earlier commits, playbooks run just fine.

[bcoca]

@evoncken so the commit just pushes up an existing error that was swallowed before, trying to avoid the lookup running at all when not used in the template

[evoracer]

still the same error in 1.8.4

[bcoca]

@evoracer why did yclose the issue? reopening as this still happens

[evoracer]

I am sorry. I pushed the wrong button. :/

[bcoca]

it happens, thankfully its harder to launch nukes than to close a bug (i hope)

[bcoca]

tested v2 and this is solved there

[abadger]

So, @bcoca and I (mainly bcoca :-) took a look at this for a few hours today. Unfortunately, the code that's causing this is a mess. In fact, it's one of the sections of code that made us decide to create v2. The strings that we get from the yaml playbooks need to be templated by jinja2 using the other available variables. As ansible works through tasks, we potentially define more variables (or overwrite previous ones with new values). In the v1 codebase, adding these variables and templating are both being performed multiple times at multiple points in the code on the same variables. This leads to the situation where modifying how a variable is templated with which variables at one point in the code can have unfortunate side effects on other points in the code.

This issue seems to be the result of just such a sequence of fixes for 1.8.x. Fixing this in v1 is hard as the fixes will almost certainly cause new bugs to appear and may cause slightly different versions of old bugs to reappear. The v2 code base cleans up these areas via a VariableManager class that manages the order and templating of variables so fixing these things and understanding the ramifications of the fixes are much better there. We'll continue to explore the code for this but it may be that other bugs are more important to keep fixed than this one so a fix for this may not be available until v2.

Sorry to be the bearer of discouraging news :-(

[bcoca]

so after going back and forth for a good while i think i was able to find the least invasive change that fixes this and doesn't break other stuff. please test PR above, it passes my test case but I would like to pass the original cases also.

[rickmb]

I apologize for the hit-and-run comment, but the release of 1.9.1 triggered me try to upgrade, and I still had the problem with 1.9.1. So I tried the devel branch (which I'm presuming uses v2) from source, and that still failed for me:

TASK: [common | Common | Auto upgrades configuration] *************************
fatal: [ec2-54-171-242-225.eu-west-1.compute.amazonaws.com] => Failed to template {{ lookup('file', '../../../shared/ydeploy.pub' ) }}: could not locate file in lookup: ../../../shared/ydeploy.pub

The lookup in question is used to define a variable in group_vars/all, and the task on which it fails is a simply copy task which has no relation to the variable. The whole thing is a convoluted legacy construct which I've been able to refactor to something more sane that doesn't trigger this error on either release.

This is just FYI, if I'm the only one still reporting this issue it's probably just a very, very rare edge case.

[patrickeasters]

I'm having the same issue. I've tested with 1.9.0.1, 1.9.1, and the current 2.0 dev branch.

I have a list of SSH users and keys defined in global_vars with .pub files in the files directory of the task. It's able to do the lookups and add the keys just fine. It's the subsequent template task that fails.

# Add SSH keys
- name: add ssh user keys
  authorized_key: user={{ item.name }} key="{{ item.key }}"
  with_items: ssh_users

# Disable root SSH login and force public key auth
- name: update sshd config
  template: src=sshd_config.j2 dest=/etc/ssh/sshd_config

[hyperfocus1338]

I still have this issue on 1.9.1: ansible/ansible-modules-core#1263

[bcoca]

unrelated as your issue happens when you reference it directly

[mikesparr]

Same issue on 1.9.1 after upgrading from 1.4. Is there any fix or workaround? Thanks in advance.

[rickmb]

I worked around it by merging the two roles that used the variables and making those variables local to the role instead of group vars. Seems to prevent other tasks from tripping over them.

[mikesparr]

So that means maintaining in two or more places, right? I'll try that for now until we can update to v2 and hopefully revert. Thanks!

[psi-4ward]

No workaround for this yet?

[mikesparr]

It worked for me by removing the lookup in the vars file and moving to the task file (old code commented out)

vars/main.yml

users:
  - name: user1
    key: keys/user1.pub #"{{ lookup('file', 'keys/user1.pub') }}"
  - name: user2
    key: keys/user2.pub #"{{ lookup('file', 'keys/user2.pub') }}"

tasks/main.yml

- name: Install public keys
  authorized_key: user={{ item['name'] }} 
                  key="{{ lookup('file', item['key']) }}"
  with_items: users

Keys stored in files/keys folder respectively. So the playbook organized like this:

playbooks

• roles
  • common
    • files/keys/
      • user1.pub
      • user2.pub
    • tasks/main.yml
    • vars/main.yml

[psi-4ward]

works perfect! Thanks @mikesparr !

[tallandtree]

I still have this issue with ansible 2.1.0.0 with group_vars. Will this ever be solved?

[gthieleb]

I also have this issue with ansible 2.1.2.0. Is the file lookup currently broken?

$ sudo ls /home/django1/.ssh/id_rsa.pub -la
-rw-r--r-- 1 django1 django1 410 Nov  1 12:28 /home/django1/.ssh/id_rsa.pub


fatal: [localhost]: FAILED! => {"failed": true, "msg": "the file_name 'fatal: [localhost]: FAILED! => {"failed": true, "msg": "the file_name '/home/django1/.ssh/id_rsa.pub' does not exist, or is not readable"}
' does not exist, or is not readable"}