"KeyError: 'undefined variable: 0'" when using {% import %} and {% include %}

[candlerb]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• template expansion

ANSIBLE VERSION

ansible 2.2.1.0
  config file =
  configured module search path = Default w/o overrides

CONFIGURATION

N/A

OS / ENVIRONMENT

Ubuntu 16.04. ansible installed via pip.

$ pip list --format=legacy
ansible (2.2.1.0)
cffi (1.9.1)
configobj (5.0.6)
crit (0.0.1)
cryptography (1.7.1)
dynagen (0.9.0)
enum34 (1.1.6)
idna (2.2)
ipaddr (2.1.11)
ipaddress (1.0.18)
Jinja2 (2.9.4)
MarkupSafe (0.23)
paramiko (2.1.1)
pip (9.0.1)
protobuf (3.1.0.post1)
pyasn1 (0.1.9)
pycparser (2.17)
pycrypto (2.6.1)
python-apt (1.1.0b1)
PyYAML (3.12)
setuptools (33.1.1)
six (1.10.0)
wheel (0.29.0)

SUMMARY

When one template imports another one, and then includes another template, it barfs with "KeyError: 'undefined variable: 0'"

STEPS TO REPRODUCE

# test.yml
- hosts: localhost
  tasks:
    - template:
        src: foo
        dest: /tmp/foo

# templates/foo
{% import 'qux' as qux with context %}
hello world
{{ qux.wibble }}
{% include 'bar' %}

# templates/bar
Goodbye

# templates/qux
{% set wibble = "WIBBLE" %}

EXPECTED RESULTS

/tmp/foo to be created with:

hello world
WIBBLE
Goodbye

ACTUAL RESULTS

TASK [template] ****************************************************************
task path: /home/ubuntu/bug/test.yml:3
fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "dest": "/tmp/foo",
            "src": "foo"
        },
        "module_name": "template"
    },
    "msg": "KeyError: 'undefined variable: 0'"
}

ADDITIONAL INFO

If you change templates/foo to this, it works fine:

hello world
{% include 'bar' %}

And so does this:

{% import 'qux' as qux with context %}
hello world
{{ qux.wibble }}

So it appears to be something to do with the combination of import and include which is causing it to barf.

However if you remove the "with context", like this:

{% import 'qux' as qux  %}
hello world
{{ qux.wibble }}

then it barfs in a different way:

TASK [template] ****************************************************************
task path: /home/ubuntu/bug/test.yml:3
fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "dest": "/tmp/foo",
            "src": "foo"
        },
        "module_name": "template"
    },
    "msg": "AttributeError: 'NoneType' object has no attribute 'add_locals'"
}

So there is a potentially separate issue, that 'import ... with context' works, but 'import' by itself does not, in Ansible.

The difference is that in the latter case, jinja2 will cache the import with the template (since it doesn't depend on any of the dynamic context in that particular expansion of the template)

Finally: this appears not to be a bug with jinja2 itself. Using jinja2 API directly runs the test case with no problem:

#!/usr/bin/python
from jinja2 import Environment, FileSystemLoader
env = Environment(
    loader=FileSystemLoader('templates'),
)
template = env.get_template('foo')
print template.render()

gives:

$ python test.py

hello world
WIBBLE
Goodbye

[sivel]

I believe this may be a duplicate of #20063

I would recommend trying out the ansible devel release to see if that resolves your problem.

You may also try downgrading to jinja2<2.9 as 2.9 is known to be incompatible.

[candlerb]

I believe this may be a duplicate of #20063

Doesn't sound like the same thing to me ("nested templates inside of loops defined in a parent template do not receive the local scope"). I'm not doing any loops here.

I would recommend trying out the ansible devel release to see if that resolves your problem.

In a fresh 16.04 environment:

# pip install git+https://github.com/ansible/ansible.git

Set up the same test, it fails in the same way

You may also try downgrading to jinja2<2.9 as 2.9 is known to be incompatible.

# pip install Jinja2==2.8.1

OK, that works.

So, perhaps should ansible declare this python version dependency explicitly, until the issue is sorted?

[sivel]

We had actually pinned jinja2, and found that on a number of systems, that jinja2<2.9 is not available via package managers, so it has been unpinned for now.

cc @jimi-c

[djoudi5]

Hello

Thanks for suggestion, its works also for me, with only changing Jinja2 version 2.9.4 -> 2.8.1. My case was like following:

• Problem: upstream_ip var wasn't found
• jenkins_upstream.j2 file:

location ^~ /jenkins {
        proxy_pass http://{{ upstream_ip  }}:{{ ports.jenkins }}/jenkins;
}
- vhost.conf.j2 file:
    {% set upstream_ip = ansible_default_ipv4['address'] %}
    {% for appli in applis_list %}
        {% include appli+'_upstream.j2' %}
    {% endfor %}

• Error:

{'msg': "KeyError: 'undefined variable: 0'", 'failed': True}

Regards,

[jimi-c]

Just noting that with Jinja2 > 2.8.1, this breaks all the way back to the 1.x versions, so it's not necessarily because of anything new we've done.

[SimonBin]

same issue when importing 2x, jinja 2.8 downgrade can work around it

my templates just suddenly stopped working after I updated my system regularly

757a05a doesn't help

[michalmedvecky]

kill me now

[williamh]

Hello,

my distro ansible package maintainer cites this bug as a reason he is forcing us to downgrade jinja when we install ansible-2.3.1.0 [1]., so I want to ask here, is there any idea of an ETA to get this fixed? Downgrading everyone's jinja when they install ansible doesn't seem to be an optimal solution to me.

[1] http://bugs.gentoo.org/show_bug.cgi?id=622472

[prometheanfire]

this should also be tagged affects_2.3 as that's the version we have

[anross]

Don't upgrade to fedora 26 or you'll hit this issue.

Comes with: python2-jinja2-2.9.6-1
And dnf downgrade says: Package python2-jinja2 of lowest version already installed, cannot downgrade it.

Probably not the best place, but workaround if you're on Fedora 26 to install the fc25 version of jinja2:

1. Copy the existing fedora-updates repo and call it (say) fedora-25-updates
2. Replace $releasever with 25
3. sudo dnf install python2-jinja2-2.8.1-1.fc25 --enablerepo=fedora-25-updates

[edit] dnf updates may want to include --exclude=python2-jinja2

[williamh]

Can you please give us an update on this bug? Do you have any ideas regarding what is going on? Have you been able to reproduce this, or do you know when it could possibly be fixed?
Thanks for your time and consideration. :-)

[prometheanfire]

looks like this wasn't fixed in 2.3.2.0 either...

[williamh]

Can someone from upstream please take a look at this and give us an idea of what is going on and when it will be fixed?

Thanks much for your time.

[abadger]

Note: the affects_2.3 is to mark that this affects 2.3; it's not a guarantee that it will be fixed in 2.3. We'll have to find out what the fix is before we can decide whether it can be backported or not.

[zmedico]

pallets/jinja@d67f0fd changed the behavior of get_all so that it returns a "private" dictionary:

diff --git a/jinja2/runtime.py b/jinja2/runtime.py
index 43f25063..95268e5a 100644
--- a/jinja2/runtime.py
+++ b/jinja2/runtime.py
@@ -170,9 +170,14 @@ def get_exported(self):
         return dict((k, self.vars[k]) for k in self.exported_vars)
 
     def get_all(self):
-        """Return a copy of the complete context as dict including the
-        exported variables.
+        """Return the complete context as dict including the exported
+        variables.  For optimizations reasons this might not return an
+        actual copy so be careful with using it.
         """
+        if not self.vars:
+            return self.parent
+        if not self.parent:
+            return self.vars
         return dict(self.parent, **self.vars)
 
     @internalcode

That could cause problems with any code that isn't careful with that "private" dictionary. For example, the jinja2.debug.get_jinja_locals function modifies the dictionary returned from ctx.get_all():

https://github.com/pallets/jinja/blob/d67f0fd4cc2a4af08f51f4466150d49da7798729/jinja2/debug.py#L198

[zmedico]

The get_all calls inside jinja2.compiler look suspicious:

https://github.com/pallets/jinja/blob/d67f0fd4cc2a4af08f51f4466150d49da7798729/jinja2/compiler.py

It appears to be creating new contexts that hold references to the "private" dictionary returned from get_all!

[alikins]

I think the problem on ansible sides is related to ansible.template.vars.AnsibleJ2Vars.

It isn't 'dict-like' enough for the way jinja creates the context. At one point it uses an instance
of AnsibleJ2Vars as an arg to 'dict()' and the dict constructor ends up thinking it is not a map/dict
but a sequence and eventually ends up trying to access vars[0] which throws a KeyError which
is where the 'KeyErrror: undefined variable: 0' originates from.

Adding an iter to AnsibleJ2Vars seems to get this working at first glance

[mrunge]

manually applying the patch zmedico@db69669 fixes the issues I had since updating to jinja 2.9

Thank you @zmedico !

[candlerb]

Thank you to @alikins and @zmedico for doing the analysis on this!

[jlec]

@zmedico Once again you made things work again. Thanks.

[alikins]

related: #27494

and possibly related: #20063

[alikins]

for reference, stack traces for the 'fails1.sh' and 'fails2.sh' repros, after letting the 'template' action re raise templating exceptions

diff

diff --git lib/ansible/plugins/action/template.py lib/ansible/plugins/action/template.py
index 33aa580..54034e9 100644
--- lib/ansible/plugins/action/template.py
+++ lib/ansible/plugins/action/template.py
@@ -154,6 +154,7 @@ class ActionModule(ActionBase):
             resultant = self._templar.do_template(template_data, preserve_trailing_newlines=True, escape_backslashes=False)
             self._templar.set_available_variables(old_vars)
         except Exception as e:
+            raise
             result['failed'] = True
             result['msg'] = "%s: %s" % (type(e).__name__, to_text(e))
             return result

jinja-2.9
fails1.sh

The full traceback is:
Traceback (most recent call last):
  File "/home/adrian/src/ansible/lib/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/home/adrian/src/ansible/lib/ansible/executor/task_executor.py", line 526, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/adrian/src/ansible/lib/ansible/plugins/action/template.py", line 154, in run
    resultant = self._templar.do_template(template_data, preserve_trailing_newlines=True, escape_backslashes=False)
  File "/home/adrian/src/ansible/lib/ansible/template/__init__.py", line 672, in do_template
    res = j2_concat(rf)
  File "<template>", line 17, in root
  File "/home/adrian/src/jinja-2.9/jinja2/runtime.py", line 231, in get_all
    return dict(self.parent, **self.vars)
  File "/home/adrian/src/ansible/lib/ansible/template/vars.py", line 82, in __getitem__
    raise KeyError("undefined variable: %s" % varname)
KeyError: 'undefined variable: 0'

fatal: [localhost]: FAILED! => {
    "failed": true, 
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}

fails2.sh

The full traceback is:
Traceback (most recent call last):
  File "/home/adrian/src/ansible/lib/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/home/adrian/src/ansible/lib/ansible/executor/task_executor.py", line 526, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/adrian/src/ansible/lib/ansible/plugins/action/template.py", line 154, in run
    resultant = self._templar.do_template(template_data, preserve_trailing_newlines=True, escape_backslashes=False)
  File "/home/adrian/src/ansible/lib/ansible/template/__init__.py", line 672, in do_template
    res = j2_concat(rf)
  File "<template>", line 11, in root
  File "/home/adrian/src/jinja-2.9/jinja2/environment.py", line 1089, in _get_default_module
    self._module = rv = self.make_module()
  File "/home/adrian/src/jinja-2.9/jinja2/environment.py", line 1073, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/home/adrian/src/ansible/lib/ansible/template/template.py", line 36, in new_context
    return self.environment.context_class(self.environment, vars.add_locals(locals), self.name, self.blocks)
AttributeError: 'NoneType' object has no attribute 'add_locals'

fatal: [localhost]: FAILED! => {
    "failed": true, 
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}

[alikins]

some intg tests the reproduce this at devel...alikins:jinja_29_intg_tests_20494

[alikins]

and some unit tests at devel...alikins:jinja_29_20494_unit_tests

(separate branches so they can more easily be merged/picked into @zmedico branch)