Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lookups in inventory are eagerly evaluated in 2.1.1.0, greatly impacting performance #17024

Closed
kadaan opened this issue Aug 9, 2016 · 3 comments
Closed

Lookups in inventory are eagerly evaluated in 2.1.1.0, greatly impacting performance #17024

kadaan opened this issue Aug 9, 2016 · 3 comments
Assignees
@jimi-c @bcoca @nitzmahone
Labels
affects_2.1 This issue/PR affects Ansible v2.1 affects_2.2 This issue/PR affects Ansible v2.2 bug This issue/PR relates to a bug. P2 Priority 2 - Issue Blocks Release performance

Comments

@kadaan
Copy link

kadaan commented Aug 9, 2016

ISSUE TYPE
  • Bug Report
COMPONENT NAME

Lookup Plugin

ANSIBLE VERSION
ansible 2.1.1.0
  config file = /Users/kadaan/Source/ansible_2.1.1.0_lookup_regression/ansible.cfg
  configured module search path = Default w/o overrides
CONFIGURATION
OS / ENVIRONMENT
SUMMARY

The timing of when lookups defined in an inventory are resolved has changed between 2.1.0.0 and 2.1.1.0. In 2.1.0.0, the lookups are not resolved until they are used. In 2.1.1.0, the lookups are eagerly resolved.

STEPS TO REPRODUCE
  1. Clone the repository: https://github.com/kadaan/ansible_2.1.1.0_lookup_regression
  2. Run: ./ansible_2.1.0.0.sh
  3. Run: ./ansible_2.1.1.0.sh
EXPECTED RESULTS

Lookups are evaluated lazily.

Using /Users/kadaan/Source/ansible_2.1.1.0_lookup_regression/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: test.yml *************************************************************
1 plays in playbooks/test.yml

PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debug] *******************************************************************
task path: /Users/kadaan/Source/ansible_2.1.1.0_lookup_regression/playbooks/test.yml:5
DEBUG: Lookup Initializing
DEBUG: Lookup - lookup1
ok: [localhost] => {
    "msg": []
}

TASK [debug] *******************************************************************
task path: /Users/kadaan/Source/ansible_2.1.1.0_lookup_regression/playbooks/test.yml:6
DEBUG: Lookup Initializing
DEBUG: Lookup - lookup2
ok: [localhost] => {
    "msg": []
}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0

real    0m4.204s
user    0m1.394s
sys 0m1.239s
ACTUAL RESULTS

Lookups are evaluated eagerly

Using /Users/kadaan/Source/ansible_2.1.1.0_lookup_regression/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: test.yml *************************************************************
1 plays in playbooks/test.yml

PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupQ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupP
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupS
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupR
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupU
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupT
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupW
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupV
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupY
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupX
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupZ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupA
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupC
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupB
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupD
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupG
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupF
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupI
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupH
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupK
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupJ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupM
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupL
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupO
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupN
ok: [localhost]

TASK [debug] *******************************************************************
task path: /Users/kadaan/Source/ansible_2.1.1.0_lookup_regression/playbooks/test.yml:5
DEBUG: Lookup Initializing
DEBUG: Lookup - lookup1
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupQ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupP
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupS
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupR
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupU
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupT
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupW
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupV
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupY
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupX
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupZ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupA
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupC
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupD
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupG
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupF
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupI
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupH
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupK
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupJ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupM
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupL
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupO
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupN
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupB
ok: [localhost] => {
    "msg": []
}

TASK [debug] *******************************************************************
task path: /Users/kadaan/Source/ansible_2.1.1.0_lookup_regression/playbooks/test.yml:6
DEBUG: Lookup Initializing
DEBUG: Lookup - lookup2
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupQ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupP
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupS
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupR
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupU
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupT
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupW
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupV
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupY
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupX
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupZ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupA
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupC
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupD
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupG
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupF
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupI
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupH
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupK
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupJ
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupM
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupL
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupO
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupN
DEBUG: Lookup Initializing
DEBUG: Lookup - lookupB
ok: [localhost] => {
    "msg": []
}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0


real    1m19.533s
user    0m7.208s
sys 0m5.953s
@Daryes
Copy link

Daryes commented Aug 19, 2016 •
edited

I'm having the same behavior with a task rebooting the servers after an update, like this :

serial : 10%

 tasks:
  - include: roles/linux/common-reboot/tasks/main.yml
    when: reboot_required is defined and reboot_required.stat.exists == true

Before, I had a single [skipping: ...] for each server, but with v2.1.1.0, each action in the reboot task is visible.
Also, with the serial restriction, the play stops after the first batch,

@ansibot ansibot added triage affects_2.1 This issue/PR affects Ansible v2.1 labels Sep 7, 2016
@brickyard
Copy link

6b286ee#diff-76ffc0551f8cf3d6255500316568e60bR412

self._connection.set_host_overrides(host=self._host, hostvars=variables.get('hostvars', {}).get(self._host.name, {}))

This line causes the change of behavior.

Any comment from @nitzmahone on whether this was intended?

@nitzmahone nitzmahone added P2 Priority 2 - Issue Blocks Release affects_2.2 This issue/PR affects Ansible v2.2 labels Sep 13, 2016
jimi-c added a commit that referenced this issue Sep 21, 2016
@jimi-c
Create a raw lookup for hostvars that does not template the data
139d99d 
When using hostvars to get extra connection-specific vars for connection
plugins, use this raw lookup to avoid prematurely templating all of the
hostvar data (triggering unnecessary lookups).

Fixes #17024
@jimi-c jimi-c closed this as completed in 23651b6 Sep 21, 2016
jimi-c added a commit that referenced this issue Sep 21, 2016
@jimi-c
Create a raw lookup for hostvars that does not template the data
01439aa 
When using hostvars to get extra connection-specific vars for connection
plugins, use this raw lookup to avoid prematurely templating all of the
hostvar data (triggering unnecessary lookups).

Fixes #17024

(cherry picked from commit ac5ddf4aa092e12f9e1c85c6b74aa30b7ef0a382)
@jimi-c
Copy link
Member

jimi-c commented Sep 21, 2016

Closing This Ticket

Hi!

We believe the above commit should resolve this problem for you. This will be included in the 2.2.0 and 2.1.2 releases.

If you continue seeing any problems related to this issue, or if you have any further questions, please let us know by stopping by one of the two mailing lists, as appropriate:

Because this project is very active, we're unlikely to see comments made on closed tickets, but the mailing list is a great way to ask questions, or post if you don't think this particular issue is resolved.

Thank you!

sereinity pushed a commit to sereinity-forks/ansible that referenced this issue Jan 25, 2017
@jimi-c
Create a raw lookup for hostvars that does not template the data
d27e288 
When using hostvars to get extra connection-specific vars for connection
plugins, use this raw lookup to avoid prematurely templating all of the
hostvar data (triggering unnecessary lookups).

Fixes ansible#17024
@ansibot ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 7, 2018
@ansible ansible locked and limited conversation to collaborators Apr 25, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jimi-c jimi-c

@bcoca bcoca

@nitzmahone nitzmahone

Labels
affects_2.1 This issue/PR affects Ansible v2.1 affects_2.2 This issue/PR affects Ansible v2.2 bug This issue/PR relates to a bug. P2 Priority 2 - Issue Blocks Release performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jimi-c @brickyard @bcoca @kadaan @ansibot @nitzmahone @Daryes