include + with_items not working as expected

[ejrowley]

Hi, we are trying to make use of with_items with include but have found that variables are not being expanded inside the include. I have constructed a simple test case:

inventory

localhost

host_vars/localhost

---
test_items:
  - name: test1
    property: 1
  - name: test2
    property: 2

main.yml

---
- hosts: localhost
  gather_facts: no
  vars:
    testing: $test_items
  tasks:
  - name: debug in main
    debug: msg={{ item.name }}
    with_items: ${testing} 

  - include: tasks/debug.yml item=$item
    with_items: ${testing} 

task/debug.yml

---
- name: debug in the task
  debug: msg={{ item.name }}

running the playbook results in:

% ansible-playbook -i inventory -c local main.yml -v                                                                                                                                                                   

PLAY [localhost] **************************************************************

TASK: [debug in main] *********************************************************
ok: [localhost] => (item={'property': 1, 'name': 'test1'}) => {"item": {"name": "test1", "property": 1}, "msg": "test1"}
ok: [localhost] => (item={'property': 2, 'name': 'test2'}) => {"item": {"name": "test2", "property": 2}, "msg": "test2"}

TASK: [debug in the task] *****************************************************
ok: [localhost] => (item=$test_items) => {"item": "$test_items", "msg": "{{item.name}}"}

PLAY RECAP ********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

the debug in main expands the variable correctly but in the include it is just the literal string.

[mpdehaan]

Please show me where test_items was defined and the output of "ansible --version" please.

Edit: ok, I see you have said host_vars, still need version info.

If you are using 1.2, just do this:

with_items: testing

and see if that sorts you out...

[bcoca]

FYI, I use this currently and don't even have to do item=$item at the end
of the include, it already gets passed in

Brian Coca
Stultorum infinitus est numerus
0110000101110010011001010110111000100111011101000010000001111001011011110111010100100000011100110110110101100001011100100111010000100001
Pedo mellon a minno

[mpdehaan]

Ah, this is incude + with_items.

Correct, don't pass this in.

This is a generally confusing thing and there's a reason we're moving to (keep this working but) not really document it so much. It is better if the with_items is used inside the task includes as then you can use inventory scoped variables and are not tempted to feed them to the include (which won't work)

So yes, remove the item line as @bcoca says and you should be good to go.

[ejrowley]

Hi,
Thanks for you help, I am on version 1.2 at the moment.

I did try removing item=$item from the include line which did not make a difference, I also changing with_items: line to just testing but that did not help either.

The only way to make it work was to move the with_items within the include as you suggest. The only downside is I will need to repeat the with_items for each task in the include (as I understand it) which is not very DRY.

Thanks again for your help.

[mpdehaan]

Yeah, while it may not seem ideal, with_items inside is better because it
avoids folks trying to loop over inventory variables in task or collide
using items, and is ultimately better in the long run.

(It is kind of awkward to use that to generate what feels like a nested
block.)

On Sat, Apr 20, 2013 at 10:24 AM, ejrowley notifications@github.com wrote:

Hi,
Thanks for you help, I am on version 1.2 at the moment.

I did try removing item=$item from the include line which did not make a
difference, I also changing with_items: line to just testing but that did
not help either.

The only way to make it work was to move the with_items within the include
as you suggest. The only downside is I will need to repeat the with_items
for each task in the include (as I understand it) which is not very DRY.

Thanks again for your help.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2726#issuecomment-16704806
.

[DazWorrall]

So for example, given a role looking something like this (consider it pseudo code):

- name: create user
  user: name={{username}} password={{password}}

- name: create docroot
  file: path=/var/www/{{hostname}} owner={{username}} ensure=directory

- name: create config
  ....

- name: add to monitoring
  ....

And some host vars like so:

host1:

    {
        "websites": [
            {"username": "user1", "password: "1234", "hostname": "www.foo.com"},
            {"username": "user2", "password: "1234", "hostname": "www.bar.com"}
        ],
    }

host2:

    {
        "websites": [
            {"username": "user3", "password: "1234", "hostname": "www.baz.com"}
        ],
    }

Can you wrap the role up in something to have it run those tasks n times with the appropriate vars, where n is the length of that list?

It would be nice if this worked:

- include: website.yml site={{item}}
  with_items: {{websites}}

Even if the role had to expect a dict:

- name: create user
  user: name={{site.username}} password={{site.password}}

<etc>

[mpdehaan]

Yeah include + with_items is not something folks should use.

We should not have documented it.

While I understand you have that particular structure in your variables
now, it's better to use with_items inside the task.

It doesn't work with inventory variables, so this is why you want to put
loops in your tasks/roles instead.

On Sat, Apr 20, 2013 at 4:50 PM, Darren Worrall notifications@github.comwrote:

So for example, given a role looking something like this (consider it
pseudo code):

• name: create user
  user: name={{username}} password={{password}}
• name: create docroot
  file: path=/var/www/{{hostname}} owner={{username}} ensure=directory
• name: create config
  ....
• name: add to monitoring
  ....

And some host vars like so:

host1:

{
    "websites": [
        {"username": "user1", "password: "1234", "hostname": "www.foo.com"},            {"username": "user2", "password: "1234", "hostname": "www.bar.com"}
    ],
}

host2:

{
    "websites": [
        {"username": "user3", "password: "1234", "hostname": "www.baz.com"}
    ],
}

Can you wrap the role up in something to have it run those tasks n times
with the appropriate vars, where n is the length of that list?

It would be nice if this worked:

• name: create user
  user: name={{site.username}} password={{site.password}}
  

Wrapped up with:

• include: website.yml site={{item}}
  with_items: {{websites}}

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2726#issuecomment-16711075
.

[DazWorrall]

Right, I see what you're getting at, you're saying do this:

inventory

host1
host2

foo.yml

---
- hosts: ["host1", "host2"]
  gather_facts: no
  tasks:
  - include: test.yml

test.yml

---
- name: print out
  debug: msg="Output {{ item.a }} and {{ item.b }}"
  with_items: $example

host_vars

daz$ ls host_vars/
host1.yml host2.yml
daz$ cat host_vars/*
---
  example:
    - a: 1
      b: 2
---
  example:
    - a: 3
      b: 4

Which does work:
ansible-playbook -c local -i inventory foo.yml -vv

PLAY [host1;host2] ************************************************************

TASK: [print out] *************************************************************
ok: [host1] => (item={'a': 1, 'b': 2}) => {"item": {"a": 1, "b": 2}, "msg": "Output 1 and 2"}
ok: [host2] => (item={'a': 3, 'b': 4}) => {"item": {"a": 3, "b": 4}, "msg": "Output 3 and 4"}

PLAY RECAP ********************************************************************
host1                      : ok=1    changed=0    unreachable=0    failed=0
host2                      : ok=1    changed=0    unreachable=0    failed=0

So the with_items is duplicated for every task but that's acceptable. Thanks for clarifying :)

[mpdehaan]

I like parameterized roles in 1.2 in this case, BTW:

roles:
   - { role: website, asdf: foo, xyz: 1234 }
   - { role: website, asdf: bar, xyz: 5678 }

Won't work if you have lots of per-host variance, but you could do different plays per host if you wanted

[DazWorrall]

Yeah thats definitely removing a lot of would be duplication for a lot of things, but this is about efficiently handling a few things on a lot of hosts with a lot of variance, as you say.

One limitation of this appears to be things like when_failed and potentially notify. What would happen if one of the items in the list failed?

----
- name: check thing exists
  command: check_for_thing {{item}}
  with_items: $list_of_things
  register: result

- name: handle missing thing
  command: handle_missing_thing {{item}}
  with_items: $list_of_things
  when_failed: result

Would we have to create a custom module here to deal with idempotence?

Sorry to keep poking sticks, I appreciate your responses - particularly at the weekend!

[mpdehaan]

I'm confused as to why idempotence and custom modules are brought up here.

I'd suggest simplifying the question and also moving this to the mailing
list.

On Sat, Apr 20, 2013 at 5:49 PM, Darren Worrall notifications@github.comwrote:

Yeah thats definitely removing a lot of would be duplication for a lot of
things, but this is about efficiently handling a few things on a lot of
hosts with a lot of variance, as you say.

One limitation of this appears to be things like when_failed and
potentially notify. What would happen if one of the items in the list
failed?

----- name: check thing exists
command: check_for_thing {{item}}
with_items: $list_of_things
register: result

• name: handle missing thing
  command: handle_missing_thing {{item}}
  with_items: $list_of_things
  when_failed: result

Would we have to create a custom module here to deal with idempotence?

Sorry to keep poking sticks, I appreciate your responses - particularly at
the weekend!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2726#issuecomment-16711972
.

[ibotty]

@mpdehaan: can one include roles with with_items though?

as in

• hosts: webservers
  roles:
  • { role=webserver site={{item}} }
    with_items: sites

with sites a variable in host_vars?

(or is it at least possible to include the roles with an inline list?)

[hyphen-jb]

short answer: no.
I would like to do something similar, but mpdehaan best explained the problem here: #3512 (comment)
Even though he's talking about includes, ansible's constraints prevent roles from using with_items in that manner.

Parameterized roles that can be multi-instantiated using with_items (from host vars) would be amazingly powerful. It would also change the number of tasks from host to host. Ansible can skip tasks using when:, but there still needs to be the same number of tasks on all hosts even if all of them are not executed.

I only responded to a closed issue because I really wanted roles using with_items: but it took a long time to find the why of it not being usable.

If you still want more history, read #2647 and the role documentation regarding when: http://www.ansibleworks.com/docs/playbooks_roles.html
Look at how when: is applied to roles, then imagine applying with_items: in the same manner.