Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Allow until-loops on blocks or includes #46203

Open
dagwieers opened this issue Sep 27, 2018 · 25 comments · May be fixed by #62151
Open

Feature: Allow until-loops on blocks or includes #46203

dagwieers opened this issue Sep 27, 2018 · 25 comments · May be fixed by #62151

Comments

@dagwieers
Copy link
Member

@dagwieers dagwieers commented Sep 27, 2018

SUMMARY

It would be quite useful if you can loop over more than one single tasks.

For instance if you have to poll a remote system for some progress and at the same time you want to push this progress to another backend, you could be doing:

- hosts: localhost
  tasks:
  - name: Start a long-running task
    uri:
      url: https://some-service/v1/put/new_job
      body: { foo: bar }
    register: new_job

  - until: job_status.json.message in ['Finished', 'Failed']
    block:
    - name: Get job status
      uri:
        url: https://some-service/v1/get/new_job
      register: job_status

    - name: Report job status to web service
      uri:
        url: https://backend-system/v1/post/job_status
        body: '{{ job_status.json }}'

There are many uses to this.

ISSUE TYPE
  • Feature Idea
COMPONENT NAME

Core

@mkrizek

This comment has been minimized.

Copy link
Contributor

@mkrizek mkrizek commented Sep 27, 2018

@dagwieers

This comment has been minimized.

Copy link
Member Author

@dagwieers dagwieers commented Sep 27, 2018

@mkrizek Hmm, I searched for various combination of keywords, and that one did not stick out :-(

That said, I tried using until: with block:, include: and include_tasks, but the first one fails, and the 2 others only run the included file once.

- hosts: localhost
  tasks:
  - include: taskboot.yml
    until: 5|random == 5

But apparently looping only works when using loop: ?

- hosts: localhost
  tasks:
  - include: taskboot.yml
    loop: [ 1, 2, 3, 4, 5 ]
@dagwieers

This comment has been minimized.

Copy link
Member Author

@dagwieers dagwieers commented Sep 27, 2018

Whatever I try, using until: does not work with include: and include_tasks:.

- hosts: localhost
  tasks:
  - include: taskbook.yml
    until: false
    retries: 5
    delay: 1
@mkrizek

This comment has been minimized.

Copy link
Contributor

@mkrizek mkrizek commented Sep 27, 2018

Yeah, until is not a valid argument for includes, see #46177.

@dagwieers dagwieers changed the title Feature: Allow loops on blocks Feature: Allow until-loops on blocks or includes Sep 27, 2018
@dagwieers

This comment has been minimized.

Copy link
Member Author

@dagwieers dagwieers commented Sep 27, 2018

@mkrizek In other words, what we need is not possible, neither on blocks or on includes.

So I will keep this one open, but changed the title.

@bcoca bcoca removed the needs_triage label Sep 27, 2018
@sivel

This comment has been minimized.

Copy link
Member

@sivel sivel commented Sep 27, 2018

We do have an open proposal to "taskify" includes, which would allow things like until to work on them.

ansible/proposals#136

I, however, do not believe that blocks should be extended to support this feature.

@dagwieers

This comment has been minimized.

Copy link
Member Author

@dagwieers dagwieers commented Sep 27, 2018

@sivel And what would be the reason for not extending the functionality to blocks ? As it would be a natural thing if it would work. (i.e. being able to loop every construction within a play)

@bcoca

This comment has been minimized.

Copy link
Member

@bcoca bcoca commented Sep 27, 2018

blocks are currently 'static' groupings, enabling loops on them (not just having tasks inherit them) would require making them dynamic ... as we saw with include: this has many consequences that are not immediately apparent.

@sivel

This comment has been minimized.

Copy link
Member

@sivel sivel commented Sep 27, 2018

To extend what @bcoca mentions, doing so would require us to deprecate block and replace with something like block_dynamic and block_static.

Also, every user of ansible utilizes blocks, whether explicit, or our internal implicit use of them. They are a fundamental building block of how tasks are represented and executed. Changing such an integral feature is sure to lead to unforeseen issues.

@dagwieers

This comment has been minimized.

Copy link
Member Author

@dagwieers dagwieers commented Sep 28, 2018

In any case, the documentation does not give any detail, or even does not discuss what is supposed to work and what not. There's no real distinction between "loops" and until-loops, not sure how we can make this more clear overal. The expectation is that what works for "loops" also works for until-loops.

@bcoca

This comment has been minimized.

Copy link
Member

@bcoca bcoca commented Sep 28, 2018

i would do both, allow until/retry loops to work with includes and then clearly document how they work ... so we have something to point at when it does not meet some people's expectations

@sivel

This comment has been minimized.

Copy link
Member

@sivel sivel commented Sep 28, 2018

Just to provide a small amount of detail about how includes work, is that dynamic includes are more of an internal trigger, as opposed to something that wraps execution.

As such, the task_executor short circuits early on an include, indicating to the strategy that it should read a file and insert task blocks into the TQM, that will later be processed by the task_executor.

Due to this, there is no tracking of state as a roll up to the parent include. So an until loop, which would rely on some version of a failed when/success scenario, would only refer to whether or not the strategy was told to do as detailed above. In which case, it should always succeed.

In any case, the mode of operation is that we short circuit far before an until conditions are inspected. If we just "made it work" right now, it definitely wouldn't do what a person expects. To do what people expect, would require ansible/proposals#136 to be implemented.

@bcoca

This comment has been minimized.

Copy link
Member

@bcoca bcoca commented Sep 28, 2018

the until in this case would have to rely on vars set or registered from the included tasks as the registration of the include itself would be useless ... it would still 'work' just not how most other cases do.

@dreamcat4

This comment has been minimized.

Copy link
Contributor

@dreamcat4 dreamcat4 commented Sep 28, 2018

@dagwieers The other thread is locked now. But this recent suggestion (@sivel comment above) is brand new:

#46203 (comment)

And you are saying we might want to leave open the possibility of someone else coming along later, to do a PR for looping over blocks off their own backs. Then at least we could make it crystal clear to them, as to make it as a separate and new block_dynamic:, and not touching the traditional static block: intact? Would that not make more sense to everybody ? Can we all agree upon that ahead of time? Because I agree with this idea. For all the same reasons - it's going to help prevent breaking other existing stuff which we rely on. Whilst still allowing the possibility of someone to come along, try making a PR for actually implementing it. Should we really want them to be making the best possible job and such. Then we should at least be clearly specifying this. If we already know that ahead of time. Which seems to be the case now?

@mkrizek

This comment has been minimized.

Copy link
Contributor

@mkrizek mkrizek commented Oct 31, 2018

Linking this here #16621

@deatheros

This comment was marked as off-topic.

Copy link

@deatheros deatheros commented Nov 5, 2018

+1 For implementing this feature

@mkrizek mkrizek mentioned this issue Nov 9, 2018
@ramon-garcia

This comment has been minimized.

Copy link

@ramon-garcia ramon-garcia commented Nov 30, 2018

It is natural that a block should be repeatable. Otherwise, it is very counterintuitive. It confuses.

And there seems to be no way to write a playbook with loop with more than one statement. No programming language is so limited.

@ramon-garcia

This comment has been minimized.

Copy link

@ramon-garcia ramon-garcia commented Nov 30, 2018

It looks like in Ansible, if one needs to do something complex, one should write an action plugin. This is what we are going to do.

Here one has many examples:
https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/action

Best regards

@Freeze

This comment has been minimized.

Copy link

@Freeze Freeze commented Mar 26, 2019

This is absolutely a shortcoming. Ansible has been perfect for most of my needs so far, but it seems like expanding the capabilities of blocks would make Ansible a lot better solution.

@lucasbasquerotto

This comment has been minimized.

Copy link

@lucasbasquerotto lucasbasquerotto commented Mar 26, 2019

I tried to use until in a include_tasks and it didn't worked.

What I did as a workaround was to create a yml file that include the task (loop.yml) and call itself recursively (recursive.yml) while the condition is still not satisfied.

recursive.yml:

---

- name: 'checking {{ watch_job }} status (recursive)'
  include_tasks: 'loop.yml'

- name: 'count ({{ watch_count | int + 1 }})'
  set_fact:
    watch_count: '{{ watch_count | int + 1 }}'

- name: 'retries ({{ (watch_timeout | int / watch_poll | int) | int }})'
  set_fact:
    watch_retries: '{{ (watch_timeout | int / watch_poll | int) | int }}'

- name: 'timeout ({{ watch_timeout }} seconds)'
  fail: 
    msg: "Timeout of {{ watch_timeout }} seconds exceeded ({{ watch_retries }} retries)"
  when: (not watch_status.finished) and (watch_count | int > watch_retries | int)

- name: 'wait for {{ watch_poll }} seconds'
  wait_for:
    timeout: '{{ watch_poll | int }}'
  when: not watch_status.finished

- name: 'call itself recursively'
  include_tasks: 'recursive.yml'
  when: not watch_status.finished

In the above file, I included a timeout in the case of taking too long (this is in a role that shows the output of what is running in the hosts).

Not the ideal solution, but worked for me and was relatively easy to change using until to do the above.

@efazenda

This comment has been minimized.

Copy link

@efazenda efazenda commented May 7, 2019

+1 for this feature !

@woopstar

This comment has been minimized.

Copy link

@woopstar woopstar commented May 7, 2019

+1

@wahab-icp

This comment has been minimized.

Copy link

@wahab-icp wahab-icp commented May 27, 2019

+1 for until loops on blocks

@matanbaru

This comment has been minimized.

Copy link

@matanbaru matanbaru commented May 28, 2019

I have been searching a way to do until (infinitely) for a success on all modules on the block and I managed to do this with include_tasks with rescue

I could not use regular until because the IP is changing over time and had to modify it on the run

wait_until_success.yml

- name: 'Wait until success'
  block:
    - name: Get server updated ip
      uri:
        url: https://localhost/ip
        return_content: yes
        status_code: 200
      register: ip

    - name: ssh to the server
      wait_for:
        host: "{{ ip }}"
        port: 22
        timeout: 30
        state: started
  rescue:
    - debug:
        msg: "Failed to connect - Retrying..."
    - include_tasks: wait_until_success.yml
@Ovski4

This comment has been minimized.

Copy link

@Ovski4 Ovski4 commented Nov 20, 2019

Same as @matanbaru with a way to fail after multiple retries

- name: 'Wait until success'
  block:
    - name: Set the retry count
      set_fact:
        retry_count: "{{ 0 if retry_count is undefined else retry_count|int + 1 }}"

    - name: Get server updated ip
      uri:
        url: https://localhost/ip
        return_content: yes
        status_code: 200
      register: ip

    - name: ssh to the server
      wait_for:
        host: "{{ ip }}"
        port: 22
        timeout: 30
        state: started
  rescue:
    - fail:
        msg: Ended after 5 retries
      when: retry_count|int == 5

    - debug:
        msg: "Failed to connect - Retrying..."

    - include_tasks: wait_until_success.yml
@bcoca bcoca linked a pull request that will close this issue Jan 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.