Prototype vars and inventory dump performance tweaks

[nitzmahone]

SUMMARY

Demo elimination of unnecessary disk I/O that slows down host/group vars resolution and inventory dumps.

fixes #79664 (poorly; these changes are not production-ready but just show the issues and suggest fixes)

Running ansible-inventory with these changes against this inventory script with 100 groups of 500 hosts each (50k hosts total) reduces the inventory dump wall clock time by ~ an order of magnitude:

devel:

(ansible-dev39) [mdavis@mdavis-p1 TEMP_slowinv]$ time ansible-inventory -i ./inv.py --list --output /dev/null

real    0m50.638s
user    0m43.346s
sys     0m6.845s

this branch:

(ansible-dev39) [mdavis@mdavis-p1 TEMP_slowinv]$ time ansible-inventory -i ./inv.py --list --output /dev/null

real    0m5.495s
user    0m5.151s
sys     0m0.337s

Some notes about the rationale for the changes:

PluginLoader.all():

• makes heavy use of globbing to find builtin and legacy plugins
• does not cache path traversal info or missing paths, only source for loaded plugins
• does not cache config defs for loaded plugins

host_group_vars vars plugin:

• overuses os.path.realpath(), which is incredibly expensive; normalize paths earlier, or maybe not at all- penalty for duplicates is likely much less than the aggregate cost of the calls
• does not cache missing paths/files

utils/path.py basedir() helper:

• normalize with abspath() earlier or cache the normalized result; abspath calls are very expensive and occurring on every get_vars call

utils/vars.py _validate_mutable_mappings() helper:

• this gets called millions of times during combine_vars; normalize things earlier or catch the immutability problems at the call sites- the isinstance calls are surprisingly expensive when called so frequently

vars/clean.py clean_facts():

• excessive (and incomplete for collections!) use of PluginLoader.all() with connection plugins to get var prefixes to clean
• cache the needed dynamic info

vars/plugins.py get_vars_from_path():

• excessive use of PluginLoader.all() for vars plugins (cache findings)
• excessive creation of vars plugins instances; builtin host_group_vars is basically stateless. Either cache instances or cache vars_plugins configs- the plugin setup and config is what's killing us when this is called millions of times.

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

ansible-inventory, vars plugins

[bcoca]

plugins now have option_definitions which is basically a cache savvy access to their config defs (which are cached in the ConfigManager), but only works for 'configurable' plugins currently (want to add modules to this to eventually reflect argspec). The issue is initializing automatically on plugin first load (or plugin class load?) and use the cache from then on.

[ansibot]

The test ansible-test sanity --test pep8 [explain] failed with 4 errors:

lib/ansible/plugins/loader.py:933:5: E303: too many blank lines (3)
lib/ansible/plugins/vars/host_group_vars.py:68:1: E302: expected 2 blank lines, found 1
lib/ansible/plugins/vars/host_group_vars.py:95:21: E265: block comment should start with '# '
lib/ansible/vars/plugins.py:45:1: E302: expected 2 blank lines, found 1

The test ansible-test sanity --test pylint [explain] failed with 1 error:

lib/ansible/utils/vars.py:72:4: unreachable: Unreachable code

click here for bot help