Tasks with include_vars take ages :( #13221
Labels
bug
This issue/PR relates to a bug.
needs_info
This issue requires further information. Please answer any outstanding questions.
Milestone
Comments
|
@zentavr can you provide any details about how many vars are in these various files? |
jimi-c
added
bug_report
needs_info
|
@jimi-c - do you want me to attach files? |
|
@zentavr sure, if there's nothing sensitive in them. A gist is also fine. |
|
@jimi-c - there are sensitive values inside, so i need to figure how to the in the best way. |
|
@zentavr if you just do a |
|
@jimi-c - here it is: https://gist.github.com/zentavr/d3727b53bde29df02250 |
|
I'd upgraded to the 2.0.0-0.6.rc1 and the problem is still there |
|
@zentavr possibly, though we have very considerably closed the gap on timings now, that issue is being held open as a catch-all until we determine if other types of slowness (like yours) is related or not. Since you're still seeing this, I'd say not. |
|
@zentavr it does. |
|
@jimi-c I'd set |
|
Agreed, ansible v2 is a little bit slow here |
|
With this playbook, Ansible 2 spent a lot of time in self._tqm._hostvars_manager.hostvars().set_variable_manager(self._variable_manager)I think a good way to reduce this overhead could be to update existing objects instead of copying them each time. I'll take a look to see if I can improve something around here. |
|
There are issues with updating the common objects, specially when connection data is set at task level as this might now affect all subsequent tasks. |
|
Well, forget the previous fix, I'm working on a better way of handling this by avoiding copy. Here is the resulting patch: diff --git a/lib/ansible/executor/task_queue_manager.py b/lib/ansible/executor/task_queue_manager.py
index 0f0dba0..0f8f16e 100644
--- a/lib/ansible/executor/task_queue_manager.py
+++ b/lib/ansible/executor/task_queue_manager.py
@@ -202,6 +202,7 @@ class TaskQueueManager:
# to do this with a proper BaseProxy/DictProxy derivative
exposed=(
'set_variable_manager', 'set_inventory', '__contains__', '__delitem__',
+ 'set_nonpersistent_facts', 'set_host_facts', 'set_host_variable',
'__getitem__', '__len__', '__setitem__', 'clear', 'copy', 'get', 'has_key',
'items', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values'
),
diff --git a/lib/ansible/plugins/strategy/__init__.py b/lib/ansible/plugins/strategy/__init__.py
index 405a8c7..1de7897 100644
--- a/lib/ansible/plugins/strategy/__init__.py
+++ b/lib/ansible/plugins/strategy/__init__.py
@@ -275,7 +275,7 @@ class StrategyBase:
var_value = wrap_var(result[3])
self._variable_manager.set_nonpersistent_facts(host, {var_name: var_value})
- self._tqm._hostvars_manager.hostvars().set_variable_manager(self._variable_manager)
+ self._tqm._hostvars_manager.hostvars().set_nonpersistent_facts(host, {var_name: var_value})
elif result[0] in ('set_host_var', 'set_host_facts'):
host = result[1]
@@ -300,13 +300,15 @@ class StrategyBase:
var_value = result[5]
self._variable_manager.set_host_variable(target_host, var_name, var_value)
+ self._tqm._hostvars_manager.hostvars().set_host_variable(target_host, var_name, var_value)
elif result[0] == 'set_host_facts':
facts = result[4]
if task.action == 'set_fact':
self._variable_manager.set_nonpersistent_facts(target_host, facts)
+ self._tqm._hostvars_manager.hostvars().set_nonpersistent_facts(target_host, facts)
else:
self._variable_manager.set_host_facts(target_host, facts)
- self._tqm._hostvars_manager.hostvars().set_variable_manager(self._variable_manager)
+ self._tqm._hostvars_manager.hostvars().set_host_facts(target_host, facts)
else:
raise AnsibleError("unknown result message received: %s" % result[0])
diff --git a/lib/ansible/vars/hostvars.py b/lib/ansible/vars/hostvars.py
index 130bc3f..1c9377e 100644
--- a/lib/ansible/vars/hostvars.py
+++ b/lib/ansible/vars/hostvars.py
@@ -79,6 +79,15 @@ class HostVars(collections.Mapping):
self._cached_result[sha1_hash] = result
return result
+ def set_host_variable(self, host, varname, value):
+ self._variable_manager.set_host_variable(host, varname, value)
+
+ def set_nonpersistent_facts(self, host, facts):
+ self._variable_manager.set_nonpersistent_facts(host, facts)
+
+ def set_host_facts(self, host, facts):
+ self.set_host_facts(host, facts)
+
def __contains__(self, host_name):
return self._find_host(host_name) is not NoneBasicaly, we do not try to copy the whole variables each time a small modification is made. Here, we simply update both variable manager. With this new patch, Ansible take about 10s vs 4 minutes without. @jimi-c what do you think about? |
|
@Yannig I think that's a good solution, send a PR. |
Yannig
added a commit
to Yannig/yannig-ansible-playbooks
that referenced
this issue
Nov 28, 2015
|
You can also retrieve a playbook exemple at the following location: https://github.com/Yannig/ansible-issue-low-speed/blob/master/big-vars-files/site_provision.yml To launch it, you need the compute group with hosts within it. Here's an example: [compute]
compute-[0:19]And finaly, then following variables: |
jimi-c
added a commit
that referenced
this issue
Nov 30, 2015
|
Everything works except this: The problem was gone after I commented the settings variable in my ansible.cfg. |
|
the play was (line no. 4 is that include): |
|
Whoops, that's probably my fault. I added in the role_uuid variable and hadn't tested it with compression enabled, so should be an easy fix. |
|
@jimi-c - works great for me with defined |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Issue Type: Bug Report
Ansible Version: 2.0.0 devel@d75e707
Ansible Configuration:
Environment: Mac OS X El Capitan 10.11.1
Summary
I have a playbook, which starts like this:
game_envandaws_envcome from a command-line (-eparameter).In order to execute this step i need to wait little less than an hour. Ansible 1.9.4 executes that piece of cake in ~30 seconds. I have up to 70 machines in my inventory.
Expected Results: ~30 seconds
The text was updated successfully, but these errors were encountered: