Skip to content
Permalink
Browse files

Speed up PluginLoader.all().

Replace glob.glob() with inline logic, reduces all() runtime 76%,
function calls 22%, overall runtime 19%.

Tested aginst dummy load comprised of the 12 disabled steps of
debops.auth with an inventory of 80 hosts, which stresses variable
processing and task setup. Before:

         9552218 function calls (9358883 primitive calls) in 39.030 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   39.190   39.190 ansible-playbook:3(<module>)
        1    0.007    0.007   38.088   38.088 ansible-playbook:21(<module>)
        1    0.000    0.000   37.379   37.379 playbook.py:80(run)
    ...
    33976    0.931    0.000    8.807    0.000 loader.py:423(all)
     3044    0.334    0.000    6.246    0.002 __init__.py:295(_get_filters)
   112337    0.534    0.000    6.174    0.000 glob.py:18(glob)

After:

         7441746 function calls (7248413 primitive calls) in 31.373 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   31.514   31.514 ansible-playbook:3(<module>)
        1    0.006    0.006   30.467   30.467 ansible-playbook:21(<module>)
        1    0.000    0.000   29.735   29.735 playbook.py:80(run)
    ...
     3204    0.065    0.000    2.054    0.001 loader.py:546(all)
  • Loading branch information...
dw committed May 16, 2018
1 parent 82f6f08 commit 05e3f87738a639affabcf803545c4091a1db30e6
Showing with 7 additions and 4 deletions.
  1. +7 −4 lib/ansible/plugins/loader.py
@@ -463,11 +463,14 @@ def all(self, *args, **kwargs):
if path_only and class_only:
raise AnsibleError('Do not set both path_only and class_only when calling PluginLoader.all()')

all_matches = []
found_in_cache = True

for i in self._get_paths():
all_matches.extend(glob.glob(os.path.join(i, "*.py")))
all_matches = [
os.path.join(dirpath, name)
for dirpath in self._get_paths()
if os.path.isdir(dirpath)
for name in os.listdir(dirpath)
if name[0] != '.' and name.endswith('.py')
]

loaded_modules = set()
for path in sorted(all_matches, key=os.path.basename):

0 comments on commit 05e3f87

Please sign in to comment.
You can’t perform that action at this time.