Skip to content
This repository has been archived by the owner on Oct 30, 2018. It is now read-only.

speed up async module execution by removing some sleeps #4432

Closed
wants to merge 1 commit into from
Closed

speed up async module execution by removing some sleeps #4432

wants to merge 1 commit into from

Conversation

caseylucas
Copy link

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

async_wrapper

ANSIBLE VERSION
ansible 2.2.0 (faster_async b3c686f4c5) last updated 2016/08/15 09:45:02 (GMT -500)
  lib/ansible/modules/core: (devel badde29ecf) last updated 2016/08/15 15:51:28 (GMT -500)
  lib/ansible/modules/extras: (detached HEAD 61d5fe148c) last updated 2016/08/15 09:44:17 (GMT -500)
  config file = /Users/clucas/work/ansible_async/ansible.cfg
  configured module search path = Default w/o overrides
SUMMARY

Before this change, when running async tasks in the "fire and forget, check on it later" variation described in the docs, there is a mandatory one second delay for each async task that is started. With this change, the one second delay (per tasks) is eliminated.

The one second delay is normally not noticed unless you are running with many async tasks. However, if you kick off many async tasks you will definitely notice the delay - especially if only sometimes the async tasks themselves are long-running.

Example before / after execution:

Playbook (faster_async.yml).

---
- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: start async tasks
      shell: date
      register: async_start_results
      with_sequence: count=5
      async: 30
      poll: 0

    - name: wait for async tasks
      async_status:
        jid: "{{ item.ansible_job_id }}"
      with_items: "{{ async_start_results.results }}"
      when: "{{ 'ansible_job_id' in item }}"
      until: async_wait_results.finished
      retries: 10
      register: async_wait_results

    - name: print start time
      debug:
        msg: "{{ async_wait_results.results | map(attribute='start') | list }}"
    - name: print stdout
      debug:
        msg: "{{ async_wait_results.results | map(attribute='stdout') | list }}"
...

Before. Notice approximate one second intervals:

ansible-playbook -i hosts.yml faster_async.yml

PLAY [localhost] ***************************************************************

TASK [start async tasks] *******************************************************
ok: [localhost] => (item=1)
ok: [localhost] => (item=2)
ok: [localhost] => (item=3)
ok: [localhost] => (item=4)
ok: [localhost] => (item=5)

TASK [wait for async tasks] ****************************************************
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'949308387845.43504', u'started': 1, '_ansible_item_result': True, 'item': u'1', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/949308387845.43504'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'712452982902.43526', u'started': 1, '_ansible_item_result': True, 'item': u'2', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/712452982902.43526'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'30716518596.43548', u'started': 1, '_ansible_item_result': True, 'item': u'3', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/30716518596.43548'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'155211893401.43570', u'started': 1, '_ansible_item_result': True, 'item': u'4', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/155211893401.43570'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'795675467115.43592', u'started': 1, '_ansible_item_result': True, 'item': u'5', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/795675467115.43592'})

TASK [print start time] ********************************************************
ok: [localhost] => {
    "msg": [
        "2016-08-15 16:07:43.088828",
        "2016-08-15 16:07:44.230531",
        "2016-08-15 16:07:45.427850",
        "2016-08-15 16:07:46.552987",
        "2016-08-15 16:07:47.730321"
    ]
}

TASK [print stdout] ************************************************************
ok: [localhost] => {
    "msg": [
        "Mon Aug 15 16:07:43 CDT 2016",
        "Mon Aug 15 16:07:44 CDT 2016",
        "Mon Aug 15 16:07:45 CDT 2016",
        "Mon Aug 15 16:07:46 CDT 2016",
        "Mon Aug 15 16:07:47 CDT 2016"
    ]
}

After. Notice lack of one second delays.

ansible-playbook -i hosts.yml faster_async.yml

PLAY [localhost] ***************************************************************

TASK [start async tasks] *******************************************************
ok: [localhost] => (item=1)
ok: [localhost] => (item=2)
ok: [localhost] => (item=3)
ok: [localhost] => (item=4)
ok: [localhost] => (item=5)

TASK [wait for async tasks] ****************************************************
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'94284541132.43736', u'started': 1, '_ansible_item_result': True, 'item': u'1', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/94284541132.43736'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'199926782622.43757', u'started': 1, '_ansible_item_result': True, 'item': u'2', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/199926782622.43757'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'350992208013.43778', u'started': 1, '_ansible_item_result': True, 'item': u'3', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/350992208013.43778'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'772387848892.43800', u'started': 1, '_ansible_item_result': True, 'item': u'4', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/772387848892.43800'})
changed: [localhost] => (item={'_ansible_no_log': False, u'ansible_job_id': u'44966404972.43822', u'started': 1, '_ansible_item_result': True, 'item': u'5', u'finished': 0, u'results_file': u'/Users/clucas/.ansible_async/44966404972.43822'})

TASK [print start time] ********************************************************
ok: [localhost] => {
    "msg": [
        "2016-08-15 16:10:37.174516",
        "2016-08-15 16:10:37.327496",
        "2016-08-15 16:10:37.477879",
        "2016-08-15 16:10:37.633585",
        "2016-08-15 16:10:37.776106"
    ]
}

TASK [print stdout] ************************************************************
ok: [localhost] => {
    "msg": [
        "Mon Aug 15 16:10:37 CDT 2016",
        "Mon Aug 15 16:10:37 CDT 2016",
        "Mon Aug 15 16:10:37 CDT 2016",
        "Mon Aug 15 16:10:37 CDT 2016",
        "Mon Aug 15 16:10:37 CDT 2016"
    ]
}

@gregdek
Copy link
Contributor

gregdek commented Aug 15, 2016

Thanks @caseylucas for this PR. This PR requires revisions, either because it fails to build or by reviewer request. Please make the suggested revisions. When you are done, please comment with text 'ready_for_review' and we will put this PR back into review.

[This message brought to you by your friendly Ansibull-bot.]

@caseylucas
Copy link
Author

ready_for_review

@gregdek
Copy link
Contributor

gregdek commented Aug 17, 2016

Thanks @caseylucas for this PR. This module is maintained by the Ansible core team, so it can take a while for patches to be reviewed. Thanks for your patience.

Core team: please review according to guidelines (http://docs.ansible.com/ansible/developing_modules.html#module-checklist) and comment with 'needs_revision' or merge as appropriate.

[This message brought to you by your friendly Ansibull-bot.]

@caseylucas
speed up async module execution by removing some sleeps
e1f42a4 
also, handle special case where status of waitpid under
FreeBSD is unreliable
@caseylucas caseylucas mentioned this pull request Sep 1, 2016
Closed
@gregdek
Copy link
Contributor

gregdek commented Sep 3, 2016

Thanks @caseylucas for this PR. Unfortunately, it is not mergeable in its current state due to merge conflicts. Please rebase your PR. When you are done, please comment with text 'ready_for_review' and we will put this PR back into review.

For help on how to do this cleanly please see http://docs.ansible.com/ansible/community.html#contributing-code-features-or-bugfixes

[This message brought to you by your friendly Ansibull-bot.]

@gregdek
Copy link
Contributor

gregdek commented Sep 18, 2016

@caseylucas A friendly reminder: this pull request has been marked as needing your action. If you still believe that this PR applies, and you intend to address the issues with this PR, just let us know in the PR itself and we will keep it open pending your changes. When you do address the issues, please respond with ready_for_review in your comment, so that we can notify the maintainer.

[This message brought to you by your friendly Ansibull-bot.]

@gregdek
Copy link
Contributor

gregdek commented Oct 4, 2016

@caseylucas Another friendly reminder: this pull request has been marked as needing your action. If you still believe that this PR applies, and you intend to address the issues with this PR, just let us know in the PR itself and we will keep it open. If you have addressed the issues and believe it's ready for review, please comment with the text "ready_for_review". If we don't hear from you within another 14 days, we will close this pull request.

[This message brought to you by your friendly Ansibull-bot.]

@gregdek
Copy link
Contributor

gregdek commented Oct 19, 2016

@caseylucas A friendly reminder: this pull request has been marked as needing your action. If you still believe that this PR applies, and you intend to address the issues with this PR, just let us know in the PR itself and we will keep it open pending your changes. When you do address the issues, please respond with ready_for_review in your comment, so that we can notify the maintainer.

[This message brought to you by your friendly Ansibull-bot.]

@gregdek
Copy link
Contributor

gregdek commented Nov 8, 2016

@caseylucas Another friendly reminder: this pull request has been marked as needing your action. If you still believe that this PR applies, and you intend to address the issues with this PR, just let us know in the PR itself and we will keep it open. If you have addressed the issues and believe it's ready for review, please comment with the text "ready_for_review". If we don't hear from you within another 14 days, we will close this pull request.

[This message brought to you by your friendly Ansibull-bot.]

@gregdek
Copy link
Contributor

gregdek commented Nov 23, 2016

@caseylucas A friendly reminder: this pull request has been marked as needing your action. If you still believe that this PR applies, and you intend to address the issues with this PR, just let us know in the PR itself and we will keep it open pending your changes. When you do address the issues, please respond with ready_for_review in your comment, so that we can notify the maintainer.

[This message brought to you by your friendly Ansibull-bot.]

@ansibot
Copy link

ansibot commented Dec 7, 2016

This repository has been locked. All new issues and pull requests should be filed in https://github.com/ansible/ansible

Please read through the repomerge page in the dev guide. The guide contains links to tools which automatically move your issue or pull request to the ansible/ansible repo.

1 similar comment
@ansibot
Copy link

ansibot commented Apr 11, 2017

This repository has been locked. All new issues and pull requests should be filed in https://github.com/ansible/ansible

Please read through the repomerge page in the dev guide. The guide contains links to tools which automatically move your issue or pull request to the ansible/ansible repo.

@ansibot ansibot closed this Apr 11, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
feature_pull_request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@caseylucas @gregdek @ansibot @sivel