variables in hostvars are not templated

[ktosiek]

Issue Type: Bug Report

Ansible Version: ansible 1.7 (devel d0dd4ac)

Environment: N/A

Summary:

Values comming from hostvars should go through similar variable expansion process as all the others.

Steps To Reproduce:

$ ansible-playbook -i vars_inv hostvars_example.yml

vars_inv:

localhost ansible_connection=local varA="{{ some_var }}"
otherhost ansible_connection=local varA="{{ varB }}" varB="{{ other_var }}"

hostvars_example.yml:

- hosts: all
  gather_facts: no
  vars:
    some_var: "some value"
    other_var: "other value"
  tasks:
  - debug: "msg='localhost: {{ hostvars['localhost'].varA }}'"
  - debug: "msg='otherhost: {{ hostvars['otherhost'].varA }}'"

Expected Results:

TASK: [debug msg='localhost: {{hostvars['localhost'].varA}}'] ***************** 
ok: [localhost] => {
    "msg": "localhost: some value"
}
ok: [otherhost] => {
    "msg": "localhost: some value"
}

TASK: [debug msg='otherhost: {{hostvars['otherhost'].varA}}'] ***************** 
ok: [localhost] => {
    "msg": "otherhost: other value"
}
ok: [otherhost] => {
    "msg": "otherhost: other value"
}

Actual Results:

TASK: [debug msg='localhost: {{hostvars['localhost'].varA}}'] ***************** 
ok: [localhost] => {
    "msg": "localhost: {{some_var}}"
}
ok: [otherhost] => {
    "msg": "localhost: {{some_var}}"
}

TASK: [debug msg='otherhost: {{hostvars['otherhost'].varA}}'] ***************** 
ok: [localhost] => {
    "msg": "otherhost: {{varB}}"
}
ok: [otherhost] => {
    "msg": "otherhost: {{varB}}"
}

Other notes:

The expected result above was created with help of the patch below.
I'm afraid it's not usable as is, that's why I'm submitting this as an issue and not a PR - if you think otherwise I'll happily resubmit it as one

diff --git a/lib/ansible/utils/template.py b/lib/ansible/utils/template.py
index 7848dd1..47772c6 100644
--- a/lib/ansible/utils/template.py
+++ b/lib/ansible/utils/template.py
@@ -139,6 +139,42 @@ def template(basedir, varname, vars, lookup_fatal=True, depth=0, expand_lists=Tr
             return varname


+class _jinja2_host_vars(object):
+    '''Wrapper around HostVars to make sure they are templated properly'''
+    def __init__(self, hostvars, vars):
+        self.vars = vars
+        self.hostvars = hostvars
+
+    def __contains__(self, k):
+        return k in self.hostvars
+
+    def __getitem__(self, hostname):
+        return _jinja2_vars(
+            self.vars.basedir,
+            chained_dicts(self.hostvars[hostname], self.vars),
+            self.vars.globals,
+            self.vars.fail_on_undefined)
+
+
+class chained_dicts(object):
+    def __init__(self, *dicts):
+        self.dicts = dicts
+
+    def __contains__(self, k):
+        return any(k in d for d in self.dicts)
+
+    def __getitem__(self, k):
+        for d in self.dicts[:-1]:
+            try:
+                return d[k]
+            except KeyError:
+                pass
+        return self.dicts[-1][k]
+
+    def keys(self):
+        return sorted({k for d in self.dicts for k in d.keys()})
+
+
 class _jinja2_vars(object):
     '''
     Helper class to template all variable content before jinja2 sees it.
@@ -179,7 +215,7 @@ class _jinja2_vars(object):
         var = self.vars[varname]
         # HostVars is special, return it as-is
         if isinstance(var, dict) and type(var) != dict:
-            return var
+            return _jinja2_host_vars(var, self)
         else:
             return template(self.basedir, var, self.vars, fail_on_undefined=self.fail_on_undefined)

[jimi-c]

Inventory variables are for simple variables, if you want to do more complex things like this you should put the variables in host_vars/group_vars, where this will work as expected. If you have any further questions regarding this, please let us know.

[ktosiek]

This issue is not about inventory variables.
It doesn't matter if I put them in host_vars, group_vars or in the inventory (and I did check with host_vars just now) - it's about the way they are accessed.

My point is that the magic variable named "hostvars" should act like any other dictionary - it should run the values through jinja2 templating. At the moment it doesn't.
I think I understand why it gets special treatment - forcing all variables for all hosts each time something in hostvars is accessed wouldn't scale too well. But I think it shouldn't be THAT special, as it renders preparing data for collection in one place pretty hard.

[dorkmatt]

Would love to see this addressed, as I ran into the same problem (where I populated host_vars/group_vars files with complex yaml). Again, variables are interpolated with templates but not via the hostvars[] call.

[hughsaunders]

I have also come across this problem, simple example here: https://gist.github.com/hughsaunders/517b7a1867cb5e87e7ad

I think it would be great if accessing a variable directly always gave the same result as accessing it via hostvars. Would you mind having another look @jimi-c ?

[jimi-c]

We'll take another look at this.

[jimi-c]

@hughsaunders / @ktosiek I've just pushed the above commit to my personal repo, if either of you'd like to give it a try. My approach was to template the variables directly in the HostVars class when fetching the item (where it is cached too) so hopefully that will reduce the impact of templating. This also includes some refactoring of the runner code where we build the injectable variables for the task.

[hughsaunders]

@jimi-c Thanks so much for working on this, sadly your patch didn't work for me, see inline comment jimi-c@128cb3b#commitcomment-7766169

[jimi-c]

@hughsaunders I've updated the commit per your note and re-pushed, if you'd like to retest it now. Beyond that typo, was everything else working for you?

[hughsaunders]

@jimi-c 7914989 works for me, thanks!

[jimi-c]

Closing This Ticket

Hi!

We believe the above commit should resolve this problem for you. This will also be included in the next major release.

If you continue seeing any problems related to this issue, or if you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

• https://groups.google.com/forum/#!forum/ansible-project - for user questions, tips, and tricks
• https://groups.google.com/forum/#!forum/ansible-devel - for strategy, future planning, and questions about writing code

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular issue is resolved.

Thank you!

[hughsaunders]

There was a regression shortly after the fix for this issue was landed. Hostvars are expanded correctly in 4a9cf3f, but expansion is broken from e9229cf. @jimi-c any chance we can get this issue reopened?

Example:

In the examples, note the abbreviated SHAs in [].

Working, y is defined as {{x}} which is correctly exapanded to value_of_x:

cb 0 (.venv) [(4a9cf3f...) 4a9cf3f] ~/git/ansible/test-playbook
$ ansible-playbook -i inventory test.yml

PLAY [localhost] **************************************************************

GATHERING FACTS ***************************************************************
ok: [localhost]

TASK: [debug ] ****************************************************************
ok: [localhost] => {
    "hostvars['localhost']['y']": "value_of_x"
}

PLAY RECAP ********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

Not working, {{x}} is not expanded:

cb 0 (.venv) [(e9229cf...) e9229cf] ~/git/ansible/test-playbook
$ ansible-playbook -i inventory test.yml

PLAY [localhost] **************************************************************

GATHERING FACTS ***************************************************************
ok: [localhost]

TASK: [debug ] ****************************************************************
ok: [localhost] => {
    "hostvars['localhost']['y']": "{{x}}"
}

PLAY RECAP ********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

inventory/group_vars/g1.yml:

---
x: "value_of_x"
y: "{{x}}"

playbook:

---
- hosts: localhost
  tasks:
    - debug:
        var: hostvars['localhost']['y']

[jimi-c]

@hughsaunders if you're still seeing this, could you please open a new issue (if you have not done so already)? Thanks!

[hughsaunders]

continuing this discussion in #8213

[jimi-c]

Closing This Ticket

Hi!

We believe the above commit should resolve this problem for you. This will also be included in the next major release.

If you continue seeing any problems related to this issue, or if you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

• https://groups.google.com/forum/#!forum/ansible-project - for user questions, tips, and tricks
• https://groups.google.com/forum/#!forum/ansible-devel - for strategy, future planning, and questions about writing code

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular issue is resolved.

Thank you!

[michaelw]

still seeing the same behavior: hostvars are not template expanded:

PLAY [all] *********************************************************************

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "localhost: {{ some_var }}"
}
ok: [otherhost] => {
    "msg": "localhost: {{ some_var }}"
}

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "otherhost: {{ varB }}"
}
ok: [otherhost] => {
    "msg": "otherhost: {{ varB }}"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0   
otherhost                  : ok=2    changed=0    unreachable=0    failed=0   

Playbook run took 0 days, 0 hours, 0 minutes, 0 seconds
[60726]% ansible --version
ansible 2.1.0.0 (instart/master fa59d7fa45) last updated 2016/06/26 09:45:25 (GMT -700)

[jimi-c]

@michaelw this fix was included in 2.1.1.