New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature request: looping over blocks #13262

Closed
smiller171 opened this Issue Nov 23, 2015 · 154 comments

Comments

Projects
None yet
@smiller171
Contributor

smiller171 commented Nov 23, 2015

Issue Type:

Feature Idea

Component Name:

blocks

Ansible Version:

Ansible 2.0.0_rc-1

Ansible Configuration:

NA

Environment:

Ubuntu 15.10

Summary of Decision:

We're open to implementing this but want it to go through the proposal process. Please see: #13262 (comment) for details.

Summary:

There are a number of use-cases where it would be valuable to be able to loop over a block of tasks, such that a few tasks are done in order, and that specific block of tasks are looped over for some set of values. It seems that the new block functionality could lend itself well to this if you were to enable looping over blocks.

Steps To Reproduce:
- hosts: localhost
  connection: local
  tasks:
  - block:
    - debug: msg="task 1 loop {{item}}"
    - debug: msg="task 2 loop {{item}}"
    with_items:
    - "1"
    - "2"
Expected Results:
PLAY ***************************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debug msg=task 1 loop {{item}}] ******************************************
ok: [localhost] => {
    "changed": false, 
    "msg": "task 1 loop 1"
}

TASK [debug msg=task 2 loop {{item}}] ******************************************
ok: [localhost] => {
    "changed": false, 
    "msg": "task 2 loop 1"
}

TASK [debug msg=task 1 loop {{item}}] ******************************************
ok: [localhost] => {
    "changed": false, 
    "msg": "task 1 loop 2"
}

TASK [debug msg=task 2 loop {{item}}] ******************************************
ok: [localhost] => {
    "changed": false, 
    "msg": "task 2 loop 2"
}

PLAY RECAP *********************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0
Actual Results:
ERROR! 'with_items' is not a valid attribute for a Block

The error appears to have been in '/root/test.yml': line 5, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  tasks:
  - block:
    ^ here

@bcoca bcoca closed this Nov 23, 2015

@smiller171

This comment has been minimized.

Contributor

smiller171 commented Nov 23, 2015

@bcoca no, I absolutely understand the functionality here. Currently you can only loop over a certain task, but I want to be able to loop over a set of tasks in series. I need to run (task 1 then task 2)x3 instead of (task 1)x3 then (task2)x3

Currently the only way to do this is to create a script that does tasks 1 and 2, or perhaps call a playbook from within a playbook.

Looping over roles would allow for the same behavior.

@bcoca

This comment has been minimized.

Member

bcoca commented Nov 23, 2015

you can loop over includes, sorry about the response, I misread this as a bug report

@bcoca bcoca added the feature_idea label Nov 23, 2015

@bcoca bcoca reopened this Nov 23, 2015

@smiller171

This comment has been minimized.

Contributor

smiller171 commented Nov 23, 2015

@bcoca I hadn't seen the docs on looping over includes, which seems the better solution than what I have suggested, as my method could quickly result in code that is more difficult to follow.

@messiahUA

This comment has been minimized.

messiahUA commented Dec 10, 2015

I want this, because includes are very slow (even after recent fix) and clutter output with full hosts lists and are processed even if skipped (tags).

@RuBiCK

This comment was marked as off-topic.

RuBiCK commented Jan 22, 2016

+1 for this feature.

@hewo

This comment was marked as off-topic.

hewo commented Jan 22, 2016

+1

3 similar comments
@srgvg

This comment was marked as off-topic.

Member

srgvg commented Jan 22, 2016

+1

@sergevs

This comment was marked as off-topic.

sergevs commented Jan 24, 2016

+1

@Lowess

This comment was marked as off-topic.

Contributor

Lowess commented Jan 26, 2016

+1

@lystor

This comment was marked as off-topic.

lystor commented Jan 27, 2016

+1024

@aturetta

This comment was marked as off-topic.

aturetta commented Jan 29, 2016

+1

first thing I tried after installing 2.0, disappointed it doesn't work...

@tyl0r

This comment was marked as off-topic.

tyl0r commented Jan 29, 2016

+1 (feature would be very helpful)

@arthur-c

This comment was marked as off-topic.

Contributor

arthur-c commented Jan 29, 2016

+1
But it should be able to manage tasks with "with_items" so "item" might not be the good keyword here (speaking of the first message).
Looping over an include means you need a new file. Looping over a block is way more cool !

@piotrminkina

This comment was marked as off-topic.

piotrminkina commented Jan 31, 2016

+1, but to prevent naming collisions with item from loops, loop from block could be name block_item.

@adul3

This comment was marked as off-topic.

adul3 commented Feb 1, 2016

+1 very interesting feature to work with 👍

@apenney

This comment was marked as off-topic.

apenney commented Feb 2, 2016

I hate to +1, but I was very sad to discover you can't do this. As @piotrminkina mentions, block_item would be a good name for the block item.

@nikhilo

This comment was marked as off-topic.

nikhilo commented Feb 3, 2016

+1 for the feature. block_item sounds like a good keyword in this case

@arthurtsang

This comment was marked as off-topic.

arthurtsang commented Feb 18, 2016

+1 this would be a really useful feature

@ssbarnea

This comment was marked as off-topic.

Contributor

ssbarnea commented Feb 24, 2016

👍

@dejlek

This comment was marked as off-topic.

dejlek commented Feb 24, 2016

👍 I see few possible use-cases for this feature. - It would be really nice to have it.

@steveholden

This comment was marked as off-topic.

steveholden commented Feb 24, 2016

+1. "Do the same thing a number of times with different data" seems like a classic devops use case

@soar

This comment was marked as off-topic.

soar commented Feb 29, 2016

+1
This will be useful

@ffinfo

This comment was marked as off-topic.

ffinfo commented Mar 7, 2016

+1
This would be exactly what I need right now only then with with_dict ;)

@k15r

This comment was marked as off-topic.

k15r commented Mar 8, 2016

+1
For all types of loops!

@plombardi89

This comment was marked as off-topic.

plombardi89 commented Mar 9, 2016

+1

2 similar comments
@kevensen

This comment was marked as off-topic.

Contributor

kevensen commented Mar 13, 2016

+1

@GeorgeZhai

This comment was marked as off-topic.

GeorgeZhai commented Mar 13, 2016

+1

@calfonso

This comment has been minimized.

Contributor

calfonso commented Oct 9, 2017

The support for looping over includes is the recommended way to achieve this. Take a look at dynamic includes for more detail.

If you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular issue is resolved.

Thank you!

@calfonso calfonso closed this Oct 9, 2017

@smiller171

This comment has been minimized.

Contributor

smiller171 commented Oct 9, 2017

@calfonso looping over includes forces you to break out of the current logic and context-switch when writing or updating playbooks, which isn't always ideal. We are aware of the ability to loop overs includes, but would prefer to be able to loop over blocks if we choose. Closing your most popular (by far) feature request as wontfix is pretty disappointing.

@calston

This comment was marked as off-topic.

calston commented Oct 10, 2017

This is as ridiculous as removing all the loops from Python and telling people they have to use cyclic imports. Clearly this project is destined for failure

@smiller171

This comment was marked as off-topic.

Contributor

smiller171 commented Oct 10, 2017

@calston No, I really doubt that one of the most popular configuration management tools, backed by RedHat, and the only one I know of that's idempotent by design is going to fail any time soon, even if they aren't willing to look at this issue. I think they are making a mistake here, but it's still an incredible tool that's very much worth using.

@calfonso

This comment was marked as resolved.

Contributor

calfonso commented Oct 10, 2017

@smiller171 this shouldn't have been closed, it should have been marked as waiting on contributor.

@smiller171

This comment was marked as resolved.

Contributor

smiller171 commented Oct 10, 2017

@calfonso ansibot came in behind you and removed that tag.

@abadger

This comment has been minimized.

Member

abadger commented Oct 10, 2017

@calfonso yesterday we decided that this was not a feature that we wanted inside of Ansible. So according to our discussion yesterday, closing this ticket is proper. I think since there's some flip-flopping we desperately need to have a discussion today about the pros and cons of accepting this feature and once we have them there, lay all of the pros and cons out in this ticket, not just the "final" decision as to whether this is a feature we want if someone is willing to do the work or if it is counter to what we want Ansible to do.

People will be disappointed if we decide that the feature is not something that we will put in but they will be able to at least understand why we think the alternative is sufficient and what the cost of putting the new feature in is. And if we decide that we would take this as a PR then future committers will be able to refer to the ratoinale here as to why they should take the feature instead of reject it as well.

@smiller171

This comment has been minimized.

Contributor

smiller171 commented Oct 10, 2017

@abadger If you do choose to close this permanently I think that rationale should be explained in-depth attached to my original comment so that it's easy to access for anyone finding this issue later.

@abadger

This comment has been minimized.

Member

abadger commented Oct 11, 2017

We talked about this extensively these last few days and came to the conclusion that we are open to accepting this feature if someone in the community would care to implement it. However, we're going to leave this ticket closed as this is a complex enough problem that we'd like it to go through the Proposal Process instead. To make this into a proposal, the person who wants to implement it should open an issue on https://github.com/ansible/proposals/issues (there's a template to fill out, you can look at past proposals for guidance), put it on the IRC meeting agenda ( https://github.com/ansible/community/issues?q=is%3Aopen%20label%3Ameeting_agenda%20label%3Acore ) and then plan on attending the IRC meetings to discuss how to implement it and get reviews of finished (or half-finished) code.

Do note, there are many pitfalls in implementing this. To get possible implementers started in the right direction, here are some problems and hints that we've identified in our discussions since we started considering this feature:

  • Looping over includes already serves the same function as looping over blocks. The syntax is different but the functionality is the same.
  • A naive approach to looping over blocks (by turning them into dynamic includes behind the scenes) would be possible but it would bring with it all the assumptions that plague dynamic includes without being obvious to the user why a block acted in a different way when it was inside of a loop versus outside.
  • Similarly, statically expanding the loop would entail similar shortcomings found when doing static includes (early evaluation of variables, no inventory variables, etc.).
  • So a non-naive way would have to be sought out for looping over blocks. This probably entails a reworking of core architecture as neither looping over dynamic nor looping over static includes offers a model that we can apply here. Any implementation needs to make sure that blocks inside and outside of loops can:
    • Reference the same set of variables with the same results
    • Needs to consider rescue/always semantics ...are they also looped?
    • Any_errors_fatal, serial, free vs linear strategies and other interactions.

One of the difficulties in implementing features like this is that we are trying to walk a fine line between automation and not becoming a programming language. We want people to be able to declare what their machines look like, using little bits of programming as shortcuts, to make the declarations clearer, and as an escape hatch when declarative forms are not sufficient. We do not want to create an environment where playbooks are mostly imperative code which the next sysadmin has to puzzle through and interpret. Even within the core team we go back and forth on this kind of feature, struggling to balance the needs of functionality, code maintenance, playbook maintenance, overlap with existing features, etc.

Thank you for understanding and thanks to whoever eventually wants to take a stab at implementing this!

@abadger abadger closed this Oct 11, 2017

@abadger

This comment was marked as resolved.

Member

abadger commented Oct 11, 2017

@smiller171 I've added a summary of details to your inital post that links to my in-depth comment. Good idea, adding it to the top post.

openstack-gerrit pushed a commit to openstack/openstack-ansible-lxc_container_create that referenced this issue Nov 2, 2017

Correct mac generation block
The block/rescue we were using in the mac generation task did not work
as expected. Because we use an iterator on the task and we can't iterate
over the entire block the task would fail when mac address lookups
within a running container didn't already exist but needed to be
created. This resulted in the task failing for a single host and being
removed from the inventory instead of running a rescue for only the
missing network.

The use of block/rescue has been removed. If the feature to loop over a
block is ever implemented [0] we can revisit how this action is done.

[0] - ansible/ansible#13262
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: Ie4bc3b130874047a5cbd36b98ed86a731ae5c317

@ansibot ansibot added feature and removed feature_idea labels Mar 2, 2018

@berney

This comment was marked as off-topic.

berney commented Mar 13, 2018

So why I think it's balls that Ansible (still) doesn't have this feature, looping over includes is an OK way, hopefully a contributor will come along.

There is another way that I haven't seen discussed.

Edit: OP has pointed out this doesn't really help, I misunderstood things. the rest of my original post is below 💩


- name: Install Certificate and Key                                                                                                                                                                                  
  block:                                                                                                                                                                                                               
    - command: mv /root/{{item}} /some/where/                                                                                                                                                                   
      args:                                                                                                                                                                                                                
        removes: /root/{{item}}                                                                                                                                                                                          
      with_items: &cert_files                                                                                                                                                                                              
        - cert.pem                                                                                                                                                                                                         
        - key.pem                                                                                                                                                                                                      
    - file:                                                                                                                                                                                                                
        path: /some/where/{{item}}                                                                                                                                                                                
        owner: root                                                                                                                                                                                                        
        group: root                                                                                                                                                                                                        
        mode: 0600                                                                                                                                                                                                         
        state: file                                                                                                                                                                                                      
      with_items: *cert_files                                                                                                                                                                                        
      become: true                                                                                                                                                                                                       
  tags:                                                                                                                                                                                                                
    - cert                   

This uses one of YAML's little known neat feature, anchors. This page has a pretty good demo of it http://blog.daemonl.com/2016/02/yaml.html. The TL/DR is put a &something on the part you want to repeat, and then a *something where you want to repeat it. The page I linked to shows some more anchor tricks.

If I just want to repeat something once and I'm too lazy to make a separate task file (just so I can loop), I can use anchors instead. If you had many sub-tasks that you wanted to do, you'll probably want to loop over an include instead, but if it's just a small number you might prefer the convenience of keeping it all in the one file.

@smiller171

This comment was marked as resolved.

Contributor

smiller171 commented Mar 13, 2018

@berney That doesn't help with the problem presented here. All you're doing with anchors is defining a var on the fly vs defining it with a vars file or an argument. It will still execute your command task twice and then your file task twice, where the intent of this ticket is to allow running the command task, then the file task, then running them both again. (task1x2, task2x2) vs (task1, task2)x2

@staylorx

This comment was marked as off-topic.

staylorx commented Mar 13, 2018

@barney thanks for the reminder about anchors. I see them in puppet all the time then straight out forget to use them in Ansible.

@smiller171 , I'd be interested if the distribution property of multiplication holds in Ansible? I'm not being glib. I can see that it wouldn't in a single-threaded program where the order strictly matters, but does (task1x2, task2x2) = (task1, task2)x2? Are they the same?

@berney

This comment was marked as off-topic.

berney commented Mar 13, 2018

@staylorx it's different, as smiller171 pointed out to me. If you have two tasks (A & B) and two items (1 & 2) you will get A1, A2, B1, B2 with my way, and A1, B1, A2, B2, with way OP wants. (presumably looping of includes works this way). Also smiller171 pointed out that what I've done is the equivalent of defining a variable (a variable can be a list). So the YAML anchors don't help solve this problem. I misunderstood the problem.

@smiller171

This comment was marked as off-topic.

Contributor

smiller171 commented Mar 14, 2018

@staylorx in most playbooks the two situations should create the same result, but there are some situations where you may want to do several tasks on a single item, and then move onto the next item.

Admittedly, in the few situations where this can come up it may be better to write a custom module to handle all of the things you want to do as a single task for better idempotency.

@mrkvost

This comment was marked as off-topic.

mrkvost commented May 4, 2018

+1 would be nice and useful

@smiller171

This comment was marked as off-topic.

Contributor

smiller171 commented May 4, 2018

@mrkvost please in the future just click the thumbs up on the first comment. There are a lot of people watching this thread and notifying them all for a +1 comment isn't helpful.

@BenJaziaSadok

This comment was marked as off-topic.

BenJaziaSadok commented May 23, 2018

Hello, does this feature implemented?

@smiller171

This comment has been minimized.

Contributor

smiller171 commented May 23, 2018

@BenJaziaSadok No. see: #13262 (comment) for details.

Looping over includes is the best way to handle this for now.

@ansible ansible locked as resolved and limited conversation to collaborators May 23, 2018

@dagwieers

This comment has been minimized.

Member

dagwieers commented Sep 28, 2018

I cleaned up this thread to keep the most relevant information visible. At least it makes an easier read for anyone who wants to understand the whats and whys related to this subject. I am tempted to open this issue, on the basis that if anyone wants to implement this, they can. But given the sheer volume of people, I fear this may get out of hand again.

There's also a new feature request #46203 for supporting until-loops on includes (which are currently not supported) and has some rationale why loops on blocks are unlikely to be featured anytime soon.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.