v2.0.0-0.6-rc1 and v2.0.0-0.5.beta3 - Iterating over a set_fact list of dictionaries is broken

[Etherdaemon]

Issue Type:

Bug Report

Ansible Version
ansible ((HEAD detached at v2.0.0-0.6.rc1)) $ ansible --version
ansible 2.0.0 (detached HEAD f222539) last updated 2015/11/23 10:17:38 (GMT +1000)
lib/ansible/modules/core: (devel 0e043f8) last updated 2015/11/23 10:13:05 (GMT +1000)
lib/ansible/modules/extras: (devel 0298f47) last updated 2015/11/23 10:12:09 (GMT +1000)

Running Ansible from source - v2.0.0-0.6.rc1

Environment:
OSX 10.11.1
Running Ansible from source - v2.0.0-0.6.rc1
Running boto (2.38.0) and boto3 (1.1.4)
Python 2.7.10

Summary:

When iterating over a set_fact list of dictionaries using with_items, an error of TypeError: unhashable type: 'dict' is received. This works in v2.0.0-0.4.beta2 but appears to have broken since v2.0.0-0.5.beta3. It looks like a change in ansible/lib/ansible/executor/task_executor.py may have had this affect.

Steps To Reproduce:

- name: Setting fact
  set_fact:
    testing_dictionary:
      - name: number_one
        value: one
      - name: number_two
        value: two

- name: Debugging
  debug:
    var: "{{ item }}"
  with_items: "{{ testing_dictionary | default({}) }}"

Expected Results:

TASK [allinone : Setting fact] *************************************************
ok: [127.0.0.1] => {"ansible_facts": {"testing_dictionary": [{"name": "number_one", "value": "one"}, {"name": "number_two", "value": "two"}]}, "changed": false, "invocation": {"module_args": {"testing_dictionary": [{"name": "number_one", "value": "one"}, {"name": "number_two", "value": "two"}]}, "module_name": "set_fact"}}

TASK [allinone : Debugging] ****************************************************
ok: [127.0.0.1] => (item={u'name': u'number_one', u'value': u'one'}) => {
"invocation": {
    "module_args": {
        "var": "{{ item }}"
    },
    "module_name": "debug"
    },
    "item": {
        "name": "number_one",
        "value": "one"
    },
    "{{ item }}": {
        "name": "number_one",
        "value": "one"
    }
}
ok: [127.0.0.1] => (item={u'name': u'number_two', u'value': u'two'}) => {
    "invocation": {
        "module_args": {
        "var": "{{ item }}"
    },
    "module_name": "debug"
    },
    "item": {
    "name": "number_two",
    "value": "two"
    },
   "{{ item }}": {
        "name": "number_two",
        "value": "two"
    }
}

Actual Results:

TASK [allinone : Setting fact] *************************************************
task path: testing.yml:65
ok: [127.0.0.1] => {"ansible_facts": {"testing_dictionary": [{"name": "number_one", "value": "one"}, {"name": "number_two", "value": "two"}]}, "changed": false, "invocation": {"module_args": {"testing_dictionary": [{"name": "number_one", "value": "one"}, {"name": "number_two", "value": "two"}]}, "module_name": "set_fact"}}

TASK [allinone : Debugging] ****************************************************
task path: testing.yml:73
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/Users/Etherdaemon/ansible/lib/ansible/executor/process/worker.py", line 134, in run
shared_loader_obj,
  File "/Users/Etherdaemon/ansible/lib/ansible/executor/task_executor.py", line 90, in run
item_results = self._run_loop(items)
  File "/Users/Etherdaemon/ansible/lib/ansible/executor/task_executor.py", line 228, in _run_loop
res = self._execute(variables=task_vars)
  File "/Users/Etherdaemon/ansible/lib/ansible/executor/task_executor.py", line 387, in _execute
result = self._handler.run(task_vars=variables)
  File "/Users/Etherdaemon/ansible/lib/ansible/plugins/action/debug.py", line 46, in run
result[self._task.args['var']] = results
TypeError: unhashable type: 'dict'

fatal: [127.0.0.1]: FAILED! => {"failed": true, "stdout": ""}

[Etherdaemon]

Finally found the issue:

It appears the following lines had been removed for some reason - not sure why though:

Under function:

def _execute(self, variables=None):

Above self._task.post_validate(templar=templar)

prev_var = None
if self._task.action == 'debug' and 'var' in self._task.args:
    prev_var = self._task.args.pop('var')

Below self._task.post_validate(templar=templar):

if prev_var is not None:
        self._task.args['var'] = prev_var

[Yannig]

Hi @Etherdaemon

I was looking after your problem and you can fix this issue by modifying the debug action using the following patch:

diff --git a/lib/ansible/plugins/action/debug.py b/lib/ansible/plugins/action/debug.py
index 5a5a805..223ebe1 100644
--- a/lib/ansible/plugins/action/debug.py
+++ b/lib/ansible/plugins/action/debug.py
@@ -43,7 +43,7 @@ class ActionModule(ActionBase):
             results = self._templar.template(self._task.args['var'], convert_bare=True)
             if results == self._task.args['var']:
                 results = "VARIABLE IS NOT DEFINED!"
-            result[self._task.args['var']] = results
+            result[str(self._task.args['var'])] = results
         else:
             result['msg'] = 'here we are'

With this patch, ansible produce the following output:

TASK [Debugging] ***************************************************************
ok: [localhost] => (item={u'name': u'number_one', u'value': u'one'}) => {
    "item": {
        "name": "number_one",
        "value": "one"
    },
    "{u'name': u'number_one', u'value': u'one'}": "VARIABLE IS NOT DEFINED!"
}
ok: [localhost] => (item={u'name': u'number_two', u'value': u'two'}) => {
    "item": {
        "name": "number_two",
        "value": "two"
    },
    "{u'name': u'number_two', u'value': u'two'}": "VARIABLE IS NOT DEFINED!"
}

@jimi-c : what do you think about?

[Yannig]

With ansible 1.9.2, the current playbook result in the following exception:

ansible-playbook test-set_fact.yml

PLAY [localhost] ************************************************************** 

GATHERING FACTS *************************************************************** 
ok: [localhost]

TASK: [Setting fact] ********************************************************** 
ok: [localhost]

TASK: [Debugging] ************************************************************* 
fatal: [localhost] => Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 582, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 811, in _executor_internal
    complex_args=complex_args
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 1032, in _executor_internal_inner
    result = handler.run(conn, tmp, module_name, module_args, inject, complex_args)
  File "/usr/lib/python2.7/dist-packages/ansible/runner/action_plugins/debug.py", line 53, in run
    elif 'var' in args and not utils.LOOKUP_REGEX.search(args['var']):
TypeError: expected string or buffer


FATAL: all hosts have already failed -- aborting

[Etherdaemon]

@Yannig I'm not sure that solves my original problem. My original problem is that I was able to iterate over a dictionary of list of dictionaries. Your proposed patch appears to return VARIABLE_NOT_DEFINED as it's not accessing the list correctly.

The behaviour in the Expected Results in my bug report is what you get in v2.0.0-0.4.beta2 and if I updated the task_executor.py to pop those 6 lines that were removed.

[Yannig]

OK. And what about the following output:

TASK [Debugging] ***************************************************************
ok: [localhost] => (item={u'name': u'number_one', u'value': u'one'}) => {
    "item": {
        "name": "number_one", 
        "value": "one"
    }
}
ok: [localhost] => (item={u'name': u'number_two', u'value': u'two'}) => {
    "item": {
        "name": "number_two", 
        "value": "two"
    }
}

This time, I'm using, the following patch:

diff --git a/lib/ansible/plugins/action/debug.py b/lib/ansible/plugins/action/debug.py
index 5a5a805..1d9af30 100644
--- a/lib/ansible/plugins/action/debug.py
+++ b/lib/ansible/plugins/action/debug.py
@@ -25,7 +25,6 @@ class ActionModule(ActionBase):
     ''' Print statements during execution '''

     TRANSFERS_FILES = False
-
     def run(self, tmp=None, task_vars=None):
         if task_vars is None:
             task_vars = dict()
@@ -40,10 +39,14 @@ class ActionModule(ActionBase):
                 result['msg'] = self._task.args['msg']
         # FIXME: move the LOOKUP_REGEX somewhere else
         elif 'var' in self._task.args: # and not utils.LOOKUP_REGEX.search(self._task.args['var']):
-            results = self._templar.template(self._task.args['var'], convert_bare=True)
-            if results == self._task.args['var']:
-                results = "VARIABLE IS NOT DEFINED!"
-            result[self._task.args['var']] = results
+            if isinstance(self._task.args['var'], dict):
+                # Do not display dict as we have already an item field in current output
+                pass
+            else:
+                results = self._templar.template(self._task.args['var'], convert_bare=True)
+                if results == self._task.args['var']:
+                    results = "VARIABLE IS NOT DEFINED!"
+                result[self._task.args['var']] = results
         else:
             result['msg'] = 'here we are'

[Etherdaemon]

@Yannig Yep - that looks good and appears to address both #13252 #13254. Will you submit the patch and I'll cancel my PR?

[Yannig]

@Etherdaemon of course :)

[Etherdaemon]

@Yannig Thank you very much for your help.