Please sign in to comment.
Defer task_vars creation to WorkerProcess when possible.
Ansible calls VariableManager.get_vars() prior to forking each task in order to template the task's title for the "v2_playbook_on_task_start" plugin callback, and to template the "run_once" playbook setting if it exists. This wraps get_vars() in a caching factory, taking care to avoid calling it when neither the task title nor run_once contain a template, avoiding a major bottleneck in the top-level process when possible. The factory produces the same dict instance on each call, so the dict's lifetime and any changes made to it propagate identically to how things worked previously. For DebOps common.yml, 90% of tasks can avoid calling get_vars() in the top-level process, delaying it until TaskExecutor starts in the worker, effectively parallelizing get_vars() for most tasks. The net effect is that in a 10 target run against Ansible HEAD on an 8 vCPU controller, common.yml runtime drops from 4m22s to 2m40s (-38.9%). This will be even more pronounced with larger runs -- the top-level process spends about one third of its time asleep, which leaves plenty of room to be kept busy with more targets. process/worker.py: Receive a factory function rather than a concrete dict. Invoke the factory during run(), i.e. post-fork. strategy/__init__.py: * Add get_vars_factory() to produce the factory. * add_tqm_variables() is needlessly inflexible requiring a dict as input, change it to get_tqm_variables() * _queue_task() receives the factory rather than a dict. strategy/free.py: strategy/linear.py: * Use factory rather than task_vars dict. * Restructure "run_once" and callback logic to only invoke the factory if necessary. template/__init__.py: * Rename _clean_regex to better reflect what it matches. * Add is_template_uncached() to avoid invoking Jinja2 compiler in the majority of cases. This can't become default since some code relies on is_template() to ensure a compiled template is cached pre-fork in the top-level process.
- Loading branch information...
Showing with 124 additions and 58 deletions.