Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

slow inventory build #79664

Open
1 task done
mdidomenico4 opened this issue Jan 4, 2023 · 11 comments · May be fixed by #79687
Open
1 task done

slow inventory build #79664

mdidomenico4 opened this issue Jan 4, 2023 · 11 comments · May be fixed by #79687
Labels
affects_2.14 affects_2.15 affects_2.16 bug This issue/PR relates to a bug. has_pr This issue has an associated PR. inventory Inventory category verified This issue has been verified/reproduced by maintainer

Comments

@mdidomenico4
Copy link

mdidomenico4 commented Jan 4, 2023 •
edited by bcoca

Summary

in order to build my inventory file i have a directory of files, each file is a group and inside each file is a list of hostnames. there are no variables or other associated things, just hostnames and groupnames. there are 108 group files and 58k hostname entries (many duplicated) in all the group files, ~18k unique entries

when i run a python script to pull in these group files and organize them into a json dictionary in the format expected by ansible-inventory it takes less then a second, (fs cache depending)

$ time ./inv.py --list | head
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped",
            "nodeg1",
            "nodeg2",
            "nodeg3",
...snipped...
real	0m0.146s
user	0m0.126s
sys	0m0.021s

when i run this through ansible-inventory the time ballons to 30seconds

$ time ansible-inventory -i ./inv.py --list | head
{
    "_meta": {
        "hostvars": {}
    },
    "nodeg1": {
        "hosts": [
            "host1",
            "host2",
            "host3",
            "host4",
...snipped...
real	0m27.851s
user	0m23.119s
sys	0m4.586s

i also ran the inventory command through python performance tool, but i don't know enough about the underpinnings to see if it says anything useful. (output attached)

this previous bug report seems to indicate that there may have been a similar problem in the past that was fixed

#30534

i'm not sure if there's a performance issue or if i'm doing something wrong. any suggestions would be helpful.

Issue Type

Bug Report

Component Name

inventory

Ansible Version

$ ansible --version
ansible 2.9.27
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/u/mdidome1/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Oct 20 2022, 09:31:56) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)]

Configuration

$ ansible-config dump --only-changed
<no changes from the rhel dist config>

OS / Environment

$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterprise
Description: Red Hat Enterprise Linux release 8.7 (Ootpa)
Release: 8.7
Codename: Ootpa

Steps to Reproduce

see summary

Expected Results

see summary

Actual Results

see summary

Code of Conduct

  • I agree to follow the Ansible Code of Conduct
@mdidomenico4
Copy link
Author 

ansible-inventory 2.9.27
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/u/mdidome1/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-inventory
  python version = 3.6.8 (default, Oct 20 2022, 09:31:56) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/mdidome1/inventory-testing/inv.py as it did not pass its verify_file() method
Parsed /home/mdidome1/inventory-testing/inv.py inventory source with script plugin
{
    "_meta": {
        "hostvars": {}
    },
...snipped...
         87803663 function calls (87619685 primitive calls) in 46.543 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    340/1    0.014    0.000   46.544   46.544 {built-in method builtins.exec}
        1    0.000    0.000   46.544   46.544 ansible-inventory:22()
        1    0.000    0.000   46.008   46.008 inventory.py:121(run)
        1    0.011    0.011   43.363   43.363 inventory.py:302(json_inventory)
    18589    0.059    0.000   43.269    0.002 inventory.py:228(_get_host_variables)
    18589    0.543    0.000   43.159    0.002 manager.py:143(get_vars)
   724971    2.104    0.000   24.245    0.000 loader.py:597(all)
   520508    0.581    0.000   17.074    0.000 glob.py:9(glob)
1245487/1245479    1.225    0.000   16.373    0.000 glob.py:39(_iglob)
    55767    0.241    0.000   14.527    0.000 manager.py:232(_plugins_inventory)
    55767    0.136    0.000   14.077    0.000 manager.py:246(_plugins_play)
    18589    0.908    0.000   11.503    0.001 clean.py:118(clean_facts)
   111534    0.098    0.000   11.436    0.000 manager.py:213(_get_plugin_vars)
   111534    0.872    0.000   11.339    0.000 host_group_vars.py:58(get_vars)
   520508    1.041    0.000   10.300    0.000 glob.py:79(_glob1)
   190359    0.205    0.000    6.698    0.000 posixpath.py:391(realpath)
    18589    0.013    0.000    5.966    0.000 manager.py:269(groups_plugins_inventory)
    18589    0.013    0.000    5.897    0.000 manager.py:273(groups_plugins_play)
  2825544    3.337    0.000    5.515    0.000 glob.py:114(_iterdir)
190364/190359    0.918    0.000    5.267    0.000 posixpath.py:400(_joinrealpath)
  2271707    3.053    0.000    4.792    0.000 posixpath.py:75(join)
    18589    0.015    0.000    4.369    0.000 manager.py:259(all_plugins_inventory)
    18589    0.014    0.000    4.125    0.000 manager.py:262(all_plugins_play)
   520508    1.305    0.000    3.598    0.000 fnmatch.py:48(filter)
   130123    0.315    0.000    2.801    0.000 path.py:98(basedir)
   339072    0.363    0.000    2.766    0.000 posixpath.py:376(abspath)
12413709/12413707    1.855    0.000    2.626    0.000 {built-in method builtins.isinstance}
        1    0.000    0.000    2.417    2.417 __init__.py:440(_play_prereqs)
        1    0.000    0.000    2.416    2.416 manager.py:139(__init__)
        1    0.000    0.000    2.416    2.416 manager.py:210(parse_sources)
        1    0.000    0.000    2.399    2.399 manager.py:233(parse_source)
   761444    0.489    0.000    2.342    0.000 posixpath.py:168(islink)
        1    0.011    0.011    2.302    2.302 script.py:84(parse)
  1933299    1.210    0.000    2.281    0.000 posixpath.py:144(basename)
  1208517    0.792    0.000    2.248    0.000 posixpath.py:121(splitext)
   111534    0.062    0.000    2.185    0.000 __init__.py:37(get_vars)
   520516    2.178    0.000    2.178    0.000 {built-in method posix.scandir}
  1561548    0.698    0.000    2.155    0.000 glob.py:145(has_magic)
      109    0.052    0.000    2.094    0.019 script.py:163(_parse_group)
    76636    0.225    0.000    2.031    0.000 data.py:189(add_host)
  5329588    1.238    0.000    1.889    0.000 posixpath.py:41(_get_sep)
   448364    0.402    0.000    1.806    0.000 vars.py:80(combine_vars)
   761444    1.786    0.000    1.786    0.000 {built-in method posix.lstat}
   339122    1.060    0.000    1.571    0.000 posixpath.py:338(normpath)
   520516    0.805    0.000    1.311    0.000 posixpath.py:104(split)
   186225    0.387    0.000    1.276    0.000 {built-in method builtins.sorted}
  1561559    1.274    0.000    1.274    0.000 {method 'search' of '_sre.SRE_Pattern' objects}
   448364    0.214    0.000    1.215    0.000 vars.py:56(_validate_mutable_mappings)
  1208517    0.895    0.000    1.200    0.000 genericpath.py:117(_splitext)
   396424    1.189    0.000    1.189    0.000 {built-in method posix.stat}
  2825544    0.626    0.000    0.943    0.000 glob.py:82()
    58047    0.126    0.000    0.908    0.000 group.py:225(add_host)
   185919    0.161    0.000    0.860    0.000 genericpath.py:39(isdir)
  7928255    0.817    0.000    0.817    0.000 {built-in method posix.fspath}
  2306875    0.804    0.000    0.804    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
  1050658    0.452    0.000    0.771    0.000 abc.py:180(__instancecheck__)
   529675    0.325    0.000    0.682    0.000 posixpath.py:64(isabs)
  4926626    0.653    0.000    0.653    0.000 {method 'rfind' of 'str' objects}
   116422    0.065    0.000    0.615    0.000 group.py:156(get_ancestors)
   190375    0.117    0.000    0.558    0.000 genericpath.py:16(exists)
   116638    0.378    0.000    0.553    0.000 group.py:117(_walk_relationship)
   356/13    0.002    0.000    0.540    0.042 :966(_find_and_load)
   356/13    0.002    0.000    0.540    0.042 :936(_find_and_load_unlocked)
   376444    0.331    0.000    0.539    0.000 _text.py:52(to_bytes)
   308/19    0.001    0.000    0.539    0.028 :672(exec_module)
   342/13    0.002    0.000    0.537    0.041 :651(_load_unlocked)
   462/19    0.000    0.000    0.536    0.028 :211(_call_with_frames_removed)
   520509    0.315    0.000    0.478    0.000 posixpath.py:52(normcase)
  2825544    0.464    0.000    0.464    0.000 glob.py:152(_ishidden)
   502047    0.136    0.000    0.457    0.000 re.py:231(compile)
   320837    0.264    0.000    0.453    0.000 _text.py:169(to_text)
    18589    0.066    0.000    0.444    0.000 manager.py:435(_get_magic_variables)
    57995    0.074    0.000    0.411    0.000 host.py:115(add_group)
    89/13    0.000    0.000    0.385    0.030 {built-in method builtins.__import__}
  1969157    0.383    0.000    0.383    0.000 {method 'startswith' of 'bytes' objects}
    58103    0.048    0.000    0.335    0.000 group.py:255(clear_hosts_cache)
   502166    0.215    0.000    0.322    0.000 re.py:286(_compile)
  1128026    0.317    0.000    0.317    0.000 _weakrefset.py:70(__contains__)
    18589    0.008    0.000    0.310    0.000 manager.py:265(groups_inventory)
    18589    0.042    0.000    0.302    0.000 helpers.py:29(get_group_vars)
        2    0.001    0.000    0.275    0.137 __init__.py:7()
    74365    0.154    0.000    0.256    0.000 posixpath.py:154(dirname)
  1703924    0.243    0.000    0.243    0.000 {method 'startswith' of 'str' objects}
        1    0.000    0.000    0.220    0.220 inventory.py:166(dump)
        2    0.001    0.000    0.220    0.110 __init__.py:183(dumps)
        2    0.004    0.002    0.219    0.110 encoder.py:182(encode)
  1455379    0.212    0.000    0.212    0.000 {method 'endswith' of 'str' objects}
  2223362    0.209    0.000    0.209    0.000 {method 'append' of 'list' objects}
       10    0.000    0.000    0.206    0.021 __init__.py:19()
   761397    0.203    0.000    0.203    0.000 {method 'partition' of 'bytes' objects}
   111551    0.202    0.000    0.202    0.000 {built-in method builtins.locals}
1301/1003    0.001    0.000    0.185    0.000 :997(_handle_fromlist)
        2    0.000    0.000    0.170    0.085 manager.py:19()
    18589    0.016    0.000    0.164    0.000 host.py:161(get_vars)
        1    0.000    0.000    0.163    0.163 json.py:66(iterencode)
  58322/1    0.030    0.000    0.163    0.163 json.py:18(_preprocess_unsafe_encode)
  329/110    0.000    0.000    0.163    0.001 json.py:29()
    37178    0.033    0.000    0.163    0.000 unsafe_proxy.py:112(wrap_var)
      108    0.009    0.000    0.162    0.001 json.py:27()
        1    0.000    0.000    0.145    0.145 constants.py:5()
        1    0.000    0.000    0.138    0.138 subprocess.py:823(communicate)
        1    0.000    0.000    0.138    0.138 subprocess.py:1486(_communicate)
       46    0.000    0.000    0.135    0.003 selectors.py:365(select)
       46    0.135    0.003    0.135    0.003 {method 'poll' of 'select.poll' objects}
   750065    0.122    0.000    0.126    0.000 {method 'add' of 'set' objects}
   376694    0.120    0.000    0.120    0.000 {method 'encode' of 'str' objects}
   520508    0.120    0.000    0.120    0.000 glob.py:22(iglob)
   817184    0.119    0.000    0.119    0.000 {method 'endswith' of 'bytes' objects}
   583724    0.117    0.000    0.117    0.000 {method 'rstrip' of 'str' objects}
    18589    0.037    0.000    0.115    0.000 clean.py:69(strip_internal_keys)
   320523    0.113    0.000    0.113    0.000 {method 'split' of 'bytes' objects}
       13    0.000    0.000    0.113    0.009 __init__.py:5()
   543561    0.109    0.000    0.109    0.000 {method 'copy' of 'dict' objects}
        1    0.000    0.000    0.108    0.108 loader.py:7()
      137    0.001    0.000    0.106    0.001 sre_compile.py:557(compile)
      218    0.000    0.000    0.105    0.000 group.py:261(get_hosts)
      216    0.055    0.000    0.105    0.000 group.py:267(_get_hosts)
    37178    0.078    0.000    0.105    0.000 data.py:243(set_variable)
        1    0.000    0.000    0.102    0.102 yaml.py:7()
   467804    0.102    0.000    0.102    0.000 {method 'update' of 'dict' objects}
        1    0.000    0.000    0.101    0.101 loader.py:19()
        1    0.000    0.000    0.101    0.101 constructor.py:19()
   111542    0.061    0.000    0.100    0.000 loader.py:533(_update_object)
   320596    0.097    0.000    0.097    0.000 {method 'decode' of 'bytes' objects}
    37178    0.018    0.000    0.097    0.000 helpers.py:25(sort_groups)
        1    0.000    0.000    0.096    0.096 dataloader.py:6()
        1    0.000    0.000    0.096    0.096 __init__.py:28()
    18635    0.019    0.000    0.095    0.000 genericpath.py:27(isfile)
        1    0.000    0.000    0.094    0.094 basic.py:5()
  349/346    0.001    0.000    0.093    0.000 :564(module_from_spec)
        1    0.000    0.000    0.093    0.093 manager.py:270(__init__)
   411221    0.093    0.000    0.093    0.000 {built-in method builtins.getattr}
   232736    0.064    0.000    0.088    0.000 group.py:143()
   501904    0.086    0.000    0.086    0.000 {method 'intersection' of 'set' objects}
    37178    0.070    0.000    0.085    0.000 unsafe_proxy.py:96(_wrap_dict)
        1    0.000    0.000    0.084    0.084 __init__.py:340(update_cache_if_changed)
        1    0.000    0.000    0.084    0.084 __init__.py:292(update_cache_if_changed)
        1    0.000    0.000    0.084    0.084 __init__.py:296(set_cache)
  58598/1    0.044    0.000    0.084    0.084 copy.py:132(deepcopy)
    113/1    0.000    0.000    0.084    0.084 copy.py:236(_deepcopy_dict)
    55769    0.083    0.000    0.083    0.000 {built-in method posix.getcwdb}
      109    0.016    0.000    0.083    0.001 copy.py:210(_deepcopy_list)
    36/35    0.000    0.000    0.083    0.002 :919(create_module)
    18589    0.010    0.000    0.082    0.000 manager.py:182(get_groups_dict)
    36/35    0.079    0.002    0.082    0.002 {built-in method _imp.create_dynamic}
        9    0.000    0.000    0.081    0.009 constructor.py:33(get_single_data)
    18589    0.027    0.000    0.081    0.000 host.py:86(__init__)
   320526    0.080    0.000    0.080    0.000 {method 'join' of 'bytes' objects}
        2    0.000    0.000    0.078    0.039 manager.py:299(_read_config_yaml_file)
        2    0.001    0.000    0.078    0.039 __init__.py:65(load)
        1    0.000    0.000    0.077    0.077 __init__.py:11(default_backend)
    18589    0.039    0.000    0.077    0.000 host.py:153(get_magic_vars)
        1    0.000    0.000    0.076    0.076 backend.py:5()
    18589    0.005    0.000    0.072    0.000 data.py:273(get_groups_dict)
   262867    0.072    0.000    0.072    0.000 display.py:237(debug)
   522205    0.072    0.000    0.072    0.000 {method 'extend' of 'list' objects}
629425/628887    0.070    0.000    0.071    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.070    0.070 journal.py:21()
   571088    0.067    0.000    0.067    0.000 {built-in method _stat.S_ISLNK}
      308    0.002    0.000    0.065    0.000 :743(get_code)
1086/1082    0.024    0.000    0.064    0.000 {built-in method builtins.__build_class__}
   618221    0.064    0.000    0.064    0.000 {method 'update' of 'set' objects}
        1    0.000    0.000    0.061    0.061 manager.py:348(get_hosts)
      137    0.000    0.000    0.058    0.000 sre_compile.py:542(_code)
       14    0.000    0.000    0.055    0.004 __init__.py:1()
    18589    0.046    0.000    0.054    0.000 vars.py:44(get_unique_id)
        1    0.000    0.000    0.054    0.054 __init__.py:75(_patch_async)
        1    0.000    0.000    0.053    0.053 asyncsupport.py:11()
    59305    0.007    0.000    0.051    0.000 encoder.py:412(_iterencode)
   229752    0.036    0.000    0.051    0.000 host.py:48(__hash__)
    18589    0.023    0.000    0.049    0.000 clean.py:22(module_response_deepcopy)
    18740    0.049    0.000    0.049    0.000 {built-in method posix.getcwd}
 1164/137    0.010    0.000    0.047    0.000 sre_compile.py:64(_compile)
      137    0.001    0.000    0.047    0.000 sre_parse.py:844(parse)
    18790    0.022    0.000    0.046    0.000 _collections_abc.py:657(get)
   390629    0.046    0.000    0.046    0.000 {method 'pop' of 'dict' objects}
    18589    0.045    0.000    0.045    0.000 inventory.py:247(_remove_internal)
      308    0.001    0.000    0.044    0.000 :485(_compile_bytecode)
118390/59305    0.016    0.000    0.044    0.000 encoder.py:333(_iterencode_dict)
  457/137    0.002    0.000    0.043    0.000 sre_parse.py:407(_parse_sub)
      308    0.043    0.000    0.043    0.000 {built-in method marshal.loads}
  644/171    0.015    0.000    0.042    0.000 sre_parse.py:470(_parse)
    18589    0.013    0.000    0.042    0.000 __init__.py:197(_populate_host_vars)
      351    0.003    0.000    0.042    0.000 :870(_find_spec)
130125/130124    0.036    0.000    0.042    0.000 loader.py:229(_get_paths)
   225661    0.041    0.000    0.041    0.000 {built-in method builtins.setattr}
        1    0.000    0.000    0.041    0.041 manager.py:405(_evaluate_patterns)
        1    0.000    0.000    0.041    0.041 addresses.py:19()
        1    0.000    0.000    0.041    0.041 manager.py:4()
        9    0.024    0.003    0.040    0.004 {method 'get_single_node' of '_yaml.CParser' objects}
        9    0.001    0.000    0.040    0.004 constructor.py:40(construct_document)
        1    0.000    0.000    0.040    0.040 manager.py:431(_match_one_pattern)
        1    0.000    0.000    0.040    0.040 manager.py:534(_enumerate_matches)
    37180    0.025    0.000    0.040    0.000 __init__.py:597(iteritems)
        1    0.000    0.000    0.040    0.040 environment.py:10()
   111534    0.036    0.000    0.036    0.000 __init__.py:33(__init__)
      347    0.001    0.000    0.036    0.000 :1149(find_spec)
    76584    0.025    0.000    0.036    0.000 group.py:281(get_vars)
      347    0.002    0.000    0.036    0.000 :1117(_get_spec)
   111542    0.033    0.000    0.033    0.000 loader.py:584(_display_plugin_load)
   185898    0.033    0.000    0.033    0.000 {built-in method _stat.S_ISDIR}
        1    0.000    0.000    0.033    0.033 base_events.py:14()
 1244/994    0.001    0.000    0.031    0.000 constructor.py:395(construct_yaml_map)
      634    0.007    0.000    0.031    0.000 :1233(find_spec)
        1    0.000    0.000    0.030    0.030 __init__.py:2()
      622    0.001    0.000    0.030    0.000 constructor.py:201(construct_mapping)
        3    0.000    0.000    0.030    0.010 base.py:5()
    58322    0.012    0.000    0.030    0.000 collections.py:76(is_sequence)
      175    0.000    0.000    0.028    0.000 utils.py:44(register_decorator)
    58427    0.018    0.000    0.027    0.000 encoder.py:277(_iterencode_list)
      622    0.004    0.000    0.027    0.000 constructor.py:118(construct_mapping)
   205655    0.027    0.000    0.027    0.000 {method 'get' of 'dict' objects}
        1    0.000    0.000    0.025    0.025 loader.py:2()
     6175    0.012    0.000    0.024    0.000 constructor.py:53(construct_object)
    18589    0.018    0.000    0.024    0.000 fact_cache.py:30(__getitem__)
      176    0.002    0.000    0.023    0.000 utils.py:102(verify_interface)
   115990    0.022    0.000    0.022    0.000 helpers.py:26()
    109/1    0.000    0.000    0.022    0.022 inventory.py:306(format_group)
   116314    0.021    0.000    0.021    0.000 {built-in method from_iterable}
     4343    0.008    0.000    0.020    0.000 enum.py:803(__and__)
   116422    0.020    0.000    0.020    0.000 {method 'difference_update' of 'set' objects}
    10397    0.006    0.000    0.020    0.000 enum.py:267(__call__)
    18589    0.008    0.000    0.020    0.000 manager.py:256(all_inventory)
      408    0.001    0.000    0.020    0.000 sre_compile.py:223(_compile_charset)
        1    0.000    0.000    0.019    0.019 helpers.py:46(deduplicate_list)
        1    0.007    0.007    0.019    0.019 helpers.py:51()
        1    0.000    0.000    0.018    0.018 nodes.py:14()
        1    0.000    0.000    0.018    0.018 extensions.py:5()
        2    0.000    0.000    0.018    0.009 __init__.py:4()
      408    0.015    0.000    0.017    0.000 sre_compile.py:250(_optimize_charset)
    58322    0.012    0.000    0.017    0.000 collections.py:59(is_string)
    58047    0.017    0.000    0.017    0.000 group.py:162(host_names)
    37488    0.017    0.000    0.017    0.000 {method 'split' of 'str' objects}
        1    0.013    0.013    0.017    0.017 data.py:102(reconcile_inventory)
41467/41441    0.016    0.000    0.016    0.000 {method 'join' of 'str' objects}
        1    0.000    0.000    0.016    0.016 manager.py:544(update_module_defaults_groups)
       25    0.001    0.000    0.016    0.001 __init__.py:357(namedtuple)
    18589    0.016    0.000    0.016    0.000 manager.py:211()
      416    0.000    0.000    0.015    0.000 inspect.py:3063(signature)
        1    0.000    0.000    0.015    0.015 utils.py:10()
      416    0.000    0.000    0.015    0.000 inspect.py:2811(from_callable)
   230038    0.015    0.000    0.015    0.000 {built-in method builtins.hash}
      416    0.002    0.000    0.015    0.000 inspect.py:2183(_signature_from_callable)
        1    0.000    0.000    0.014    0.014 process.py:44()
    18589    0.014    0.000    0.014    0.000 host.py:157()
    37178    0.014    0.000    0.014    0.000 host.py:144(set_variable)
    75496    0.014    0.000    0.014    0.000 {method 'items' of 'dict' objects}
    14848    0.006    0.000    0.013    0.000 sre_parse.py:253(get)
        1    0.000    0.000    0.013    0.013 manager.py:192(_fetch_inventory_plugins)
      8/7    0.000    0.000    0.013    0.002 loader.py:539(get)
        1    0.000    0.000    0.013    0.013 unix_events.py:1()
        1    0.001    0.001    0.012    0.012 manager.py:552(update_config_data)
        1    0.000    0.000    0.012    0.012 data.py:19()
      190    0.002    0.000    0.011    0.000 manager.py:428(get_config_value_and_origin)
        1    0.000    0.000    0.011    0.011 selector_events.py:5()
        1    0.000    0.000    0.011    0.011 group.py:17()
        2    0.000    0.000    0.011    0.005 vars.py:19()
     3409    0.004    0.000    0.011    0.000 :57(_path_join)
        1    0.000    0.000    0.011    0.011 defaults.py:10()
     1494    0.001    0.000    0.010    0.000 :75(_path_stat)
      416    0.004    0.000    0.010    0.000 inspect.py:2102(_signature_from_function)
        1    0.000    0.000    0.010    0.010 shutil.py:5()
     6173    0.007    0.000    0.010    0.000 resolver.py:143(resolve)
        1    0.000    0.000    0.010    0.010 __init__.py:15()
      137    0.001    0.000    0.010    0.000 sre_compile.py:482(_compile_info)
        1    0.000    0.000    0.010    0.010 general_name.py:5()
        1    0.000    0.000    0.010    0.010 ssl.py:91()
      349    0.002    0.000    0.009    0.000 :504(_init_module_attrs)
    18589    0.009    0.000    0.009    0.000 clean.py:170(namespace_facts)
        1    0.000    0.000    0.009    0.009 context.py:1()
        1    0.000    0.000    0.009    0.009 reader.py:18()
        1    0.000    0.000    0.009    0.009 reader.py:45(Reader)
     4360    0.002    0.000    0.009    0.000 constructor.py:387(construct_yaml_str)
      308    0.006    0.000    0.009    0.000 :830(get_data)
    55767    0.009    0.000    0.009    0.000 host.py:150(get_groups)
        1    0.000    0.000    0.009    0.009 dataloader.py:78(load)
        1    0.000    0.000    0.009    0.009 yaml.py:58(from_yaml)
        1    0.000    0.000    0.009    0.009 __init__.py:302(loads)
        1    0.000    0.000    0.009    0.009 decoder.py:334(decode)
5132/5123    0.003    0.000    0.009    0.000 {built-in method builtins.hasattr}
        1    0.008    0.008    0.009    0.009 decoder.py:345(raw_decode)
        1    0.000    0.000    0.009    0.009 platform.py:10()
        1    0.000    0.000    0.008    0.008 resolver.py:2()
       13    0.000    0.000    0.008    0.001 enum.py:604(_convert)
        1    0.000    0.000    0.008    0.008 inspect.py:27()
        1    0.000    0.000    0.008    0.008 environment.py:919(__new__)
    16615    0.008    0.000    0.008    0.000 sre_parse.py:232(__next)
    10384    0.007    0.000    0.008    0.000 enum.py:517(__new__)
        1    0.000    0.000    0.008    0.008 environment.py:874(from_string)
        1    0.000    0.000    0.008    0.008 environment.py:553(compile)
    37178    0.008    0.000    0.008    0.000 unsafe_proxy.py:97()
    37202    0.008    0.000    0.008    0.000 {built-in method builtins.iter}
     4598    0.004    0.000    0.007    0.000 constructor.py:159(construct_scalar)
        1    0.000    0.000    0.007    0.007 uuid.py:45()
       30    0.003    0.000    0.007    0.000 enum.py:124(__new__)
  367/201    0.003    0.000    0.007    0.000 abc.py:196(__subclasscheck__)
        1    0.000    0.000    0.007    0.007 environment.py:495(_parse)
 1864/890    0.001    0.000    0.007    0.000 {built-in method builtins.issubclass}
     1828    0.001    0.000    0.007    0.000 constructor.py:390(construct_yaml_seq)
    37193    0.007    0.000    0.007    0.000 {method 'keys' of 'dict' objects}
        1    0.000    0.000    0.007    0.007 bz2.py:5()
    18591    0.007    0.000    0.007    0.000 {method 'rfind' of 'bytes' objects}
    37179    0.007    0.000    0.007    0.000 dataloader.py:171(get_basedir)
        4    0.000    0.000    0.007    0.002 utils.py:5()
      616    0.003    0.000    0.006    0.000 :263(cache_from_source)
    18617    0.006    0.000    0.006    0.000 {method 'rstrip' of 'bytes' objects}
      124    0.001    0.000    0.006    0.000 abc.py:132(__new__)
        1    0.000    0.000    0.006    0.006 filters.py:10()
        1    0.000    0.000    0.006    0.006 _compat.py:12()
    18720    0.006    0.000    0.006    0.000 collections.py:19(__getitem__)
        1    0.000    0.000    0.006    0.006 binding.py:5()
        1    0.000    0.000    0.006    0.006 __init__.py:73(run)
        1    0.000    0.000    0.006    0.006 lexer.py:16()
        2    0.000    0.000    0.006    0.003 hashes.py:5()
      933    0.001    0.000    0.006    0.000 constructor.py:110(construct_sequence)
        2    0.000    0.000    0.006    0.003 ocsp.py:5()
       13    0.000    0.000    0.006    0.000 enum.py:366(_create_)
        1    0.000    0.000    0.006    0.006 __init__.py:352(parse)
      8/7    0.000    0.000    0.006    0.001 loader.py:366(find_plugin_with_name)
        1    0.000    0.000    0.006    0.006 reduction.py:10()
        2    0.000    0.000    0.006    0.003 subprocess.py:608(__init__)
      8/7    0.000    0.000    0.006    0.001 loader.py:416(_find_plugin_legacy)
        1    0.000    0.000    0.006    0.006 parser.py:37(__init__)
        1    0.000    0.000    0.006    0.006 environment.py:524(_tokenize)
        2    0.000    0.000    0.005    0.003 subprocess.py:1228(_execute_child)
        1    0.000    0.000    0.005    0.005 lexer.py:391(get_lexer)
        1    0.000    0.000    0.005    0.005 lexer.py:420(__init__)
      182    0.000    0.000    0.005    0.000 abc.py:151(register)
       10    0.000    0.000    0.005    0.001 lexer.py:422()
      652    0.001    0.000    0.005    0.000 :403(cached)
        6    0.000    0.000    0.005    0.001 __init__.py:100(__get__)
      711    0.005    0.000    0.005    0.000 {built-in method __new__ of type object at 0x7fe79c687e40}
        6    0.000    0.000    0.005    0.001 __init__.py:89(_import_module)
    18591    0.005    0.000    0.005    0.000 memory.py:37(contains)
        2    0.000    0.000    0.005    0.003 __init__.py:123(_resolve)
     3409    0.003    0.000    0.005    0.000 :59()
    58322    0.005    0.000    0.005    0.000 {built-in method _json.encode_basestring_ascii}
      4/3    0.000    0.000    0.005    0.002 loader.py:214(_get_package_paths)
      208    0.001    0.000    0.005    0.000 inspect.py:2854(__eq__)
        1    0.000    0.000    0.005    0.005 threading.py:1()
      933    0.001    0.000    0.005    0.000 constructor.py:115()
        1    0.000    0.000    0.005    0.005 constructor.py:2()
     4852    0.003    0.000    0.005    0.000 sre_parse.py:163(__getitem__)
        1    0.000    0.000    0.005    0.005 socket.py:47()
 1740/719    0.004    0.000    0.005    0.000 sre_parse.py:173(getwidth)
      344    0.001    0.000    0.004    0.000 :361(_get_cached)
    59677    0.004    0.000    0.004    0.000 {built-in method builtins.id}
      356    0.001    0.000    0.004    0.000 :147(__enter__)
        1    0.000    0.000    0.004    0.004 configparser.py:139()
      505    0.000    0.000    0.004    0.000 :94(_path_isfile)
      552    0.001    0.000    0.004    0.000 :85(_path_is_mode_type)
        1    0.000    0.000    0.004    0.004 traceback.py:1()
      190    0.001    0.000    0.004    0.000 manager.py:55(ensure_type)
        1    0.000    0.000    0.004    0.004 context.py:5()
        1    0.000    0.000    0.004    0.004 binding.py:149(init_static_locks)
        3    0.000    0.000    0.004    0.001 binding.py:135(_ensure_ffi_initialized)
        8    0.000    0.000    0.004    0.001 loader.py:508(_load_module_source)
    58376    0.004    0.000    0.004    0.000 copy.py:190(_deepcopy_atomic)
      416    0.002    0.000    0.004    0.000 inspect.py:2840(_hash_basis)
        1    0.000    0.000    0.004    0.004 __init__.py:97()
        8    0.000    0.000    0.004    0.000 loader.py:279(_load_config_defs)
        1    0.000    0.000    0.004    0.004 util.py:300(find_library)
        1    0.000    0.000    0.004    0.004 linecache.py:6()
        1    0.000    0.000    0.004    0.004 util.py:248(_findSoname_ldconfig)
       49    0.004    0.000    0.004    0.000 {built-in method posix.read}
    18635    0.004    0.000    0.004    0.000 {built-in method _stat.S_ISREG}
    18590    0.004    0.000    0.004    0.000 manager.py:168(groups)
        2    0.000    0.000    0.004    0.002 ec.py:5()
    18589    0.004    0.000    0.004    0.000 host.py:102(get_name)
      108    0.004    0.000    0.004    0.000 inventory.py:310()
      216    0.000    0.000    0.004    0.000 group.py:159(get_descendants)
      308    0.003    0.000    0.003    0.000 :430(_validate_bytecode_header)
     4598    0.003    0.000    0.003    0.000 nodes.py:27(__init__)
        1    0.001    0.001    0.003    0.003 binding.py:94(build_conditional_library)
      308    0.003    0.000    0.003    0.000 {method 'read' of '_io.FileIO' objects}
        1    0.000    0.000    0.003    0.003 tokenize.py:21()
      882    0.002    0.000    0.003    0.000 inspect.py:2452(__init__)
        1    0.000    0.000    0.003    0.003 tests.py:10()
      109    0.000    0.000    0.003    0.000 data.py:256(add_child)
      109    0.003    0.000    0.003    0.000 data.py:279()
        2    0.000    0.000    0.003    0.002 dsa.py:5()
       68    0.000    0.000    0.003    0.000 six.py:882(wrapper)
      109    0.001    0.000    0.003    0.000 group.py:171(add_child_group)
      463    0.002    0.000    0.003    0.000 :157(_get_module_lock)
      408    0.001    0.000    0.003    0.000 enum.py:797(__or__)
       26    0.000    0.000    0.003    0.000 singleton.py:21(__call__)
        2    0.000    0.000    0.003    0.001 padding.py:5()
        2    0.000    0.000    0.003    0.001 rsa.py:5()
        1    0.000    0.000    0.003    0.003 constructor.py:157(SafeConstructor)
        1    0.000    0.000    0.003    0.003 configparser.py:559(RawConfigParser)
        1    0.000    0.000    0.003    0.003 manager.py:309(_parse_config_file)
        1    0.000    0.000    0.003    0.003 decimal.py:2()
      190    0.000    0.000    0.003    0.000 manager.py:189(get_ini_config_value)
        1    0.000    0.000    0.003    0.003 inventory.py:57(init_parser)
        2    0.000    0.000    0.003    0.001 dh.py:5()
      732    0.001    0.000    0.003    0.000 :1080(_path_importer_cache)
        1    0.000    0.000    0.003    0.003 sys_info.py:5()
      416    0.002    0.000    0.003    0.000 inspect.py:2732(__init__)
     4598    0.002    0.000    0.003    0.000 constructor.py:103(construct_scalar)
1834/1818    0.001    0.000    0.003    0.000 {built-in method builtins.next}
      336    0.001    0.000    0.003    0.000 :1228(_get_spec)
        1    0.000    0.000    0.002    0.002 decoder.py:2()
        1    0.000    0.000    0.002    0.002 connection.py:10()
        1    0.000    0.000    0.002    0.002 process.py:10()
        1    0.000    0.000    0.002    0.002 random.py:38()
        1    0.000    0.000    0.002    0.002 configparser.py:720(read_string)
        1    0.000    0.000    0.002    0.002 configparser.py:705(read_file)
        1    0.002    0.002    0.002    0.002 configparser.py:991(_read)
        1    0.000    0.000    0.002    0.002 x509.py:5()
        1    0.000    0.000    0.002    0.002 _base.py:4()
        1    0.000    0.000    0.002    0.002 charset.py:6()
     1271    0.001    0.000    0.002    0.000 _weakrefset.py:58(__iter__)
        1    0.000    0.000    0.002    0.002 coroutines.py:1()
       45    0.000    0.000    0.002    0.000 :1281(_fill_cache)
        1    0.000    0.000    0.002    0.002 parse.py:28()
      308    0.000    0.000    0.002    0.000 :840(path_stats)
        2    0.002    0.001    0.002    0.001 {built-in method _posixsubprocess.fork_exec}
        1    0.000    0.000    0.002    0.002 pickle.py:24()
      642    0.002    0.000    0.002    0.000 constructor.py:166(flatten_mapping)
      341    0.001    0.000    0.002    0.000 :318(__exit__)
        1    0.000    0.000    0.002    0.002 dis.py:1()
      190    0.001    0.000    0.002    0.000 configparser.py:765(get)
      337    0.001    0.000    0.002    0.000 enum.py:70(__setitem__)
        1    0.000    0.000    0.002    0.002 context.py:32(_init_global_context)
        1    0.000    0.000    0.002    0.002 context_objects.py:80(from_options)
        2    0.000    0.000    0.002    0.001 context_objects.py:74(__init__)
        1    0.000    0.000    0.002    0.002 display.py:18()
       50    0.000    0.000    0.002    0.000 path.py:36(unfrackpath)
    20/19    0.000    0.000    0.002    0.000 context_objects.py:20(_make_immutable)
        1    0.000    0.000    0.002    0.002 modes.py:5()
        2    0.000    0.000    0.002    0.001 __init__.py:108(import_module)
        2    0.000    0.000    0.002    0.001 :982(_gcd_import)
     3125    0.001    0.000    0.002    0.000 sre_parse.py:248(match)
       13    0.001    0.000    0.002    0.000 enum.py:625()
        1    0.000    0.000    0.002    0.002 name.py:5()
       47    0.000    0.000    0.002    0.000 manager.py:164(resolve_path)
        1    0.000    0.000    0.002    0.002 signal.py:1()
      344    0.001    0.000    0.002    0.000 :524(spec_from_file_location)
        1    0.000    0.000    0.002    0.002 ipaddress.py:9()
        2    0.000    0.000    0.002    0.001 hmac.py:5()
       50    0.002    0.000    0.002    0.000 {built-in method posix.listdir}
        1    0.000    0.000    0.002    0.002 __init__.py:307(Cacheable)
        1    0.000    0.000    0.002    0.002 __init__.py:282(__init__)
        1    0.000    0.000    0.002    0.002 string.py:15()
        1    0.000    0.000    0.002    0.002 __init__.py:9(swig_import_helper)
       21    0.000    0.000    0.002    0.000 manager.py:134()
       67    0.000    0.000    0.002    0.000 nodes.py:64(__new__)
        1    0.000    0.000    0.002    0.002 collections.py:4()
        1    0.000    0.000    0.002    0.002 ssh.py:5()
        1    0.000    0.000    0.002    0.002 interfaces.py:5()
        1    0.000    0.000    0.002    0.002 splitter.py:19()
        6    0.000    0.000    0.002    0.000 display.py:153(display)
        1    0.000    0.000    0.002    0.002 version.py:27()
        1    0.000    0.000    0.002    0.002 string.py:65(__init__)
        1    0.000    0.000    0.002    0.002 dumper.py:2()
        1    0.000    0.000    0.002    0.002 compiler.py:10()
      356    0.000    0.000    0.002    0.000 :151(__exit__)
       45    0.001    0.000    0.002    0.000 :1067(_path_hooks)
        1    0.000    0.000    0.002    0.002 scanner.py:2()
      416    0.001    0.000    0.002    0.000 inspect.py:485(unwrap)
        1    0.000    0.000    0.002    0.002 __init__.py:271(init_parser)
        1    0.000    0.000    0.002    0.002 pbkdf2.py:5()
       21    0.000    0.000    0.002    0.000 argparse.py:1307(add_argument)
        1    0.000    0.000    0.002    0.002 option_helpers.py:190(create_base_parser)
        1    0.000    0.000    0.002    0.002 option_helpers.py:4()
      137    0.000    0.000    0.001    0.000 sre_parse.py:828(fix_flags)
        1    0.000    0.000    0.001    0.001 collection_loader.py:4()
      139    0.000    0.000    0.001    0.000 sre_parse.py:96(closegroup)
        1    0.000    0.000    0.001    0.001 selectors.py:5()
        1    0.000    0.000    0.001    0.001 __init__.py:163(parse)
        1    0.000    0.000    0.001    0.001 __init__.py:438(__init__)
     3545    0.001    0.000    0.001    0.000 :222(_verbose_message)
      463    0.001    0.000    0.001    0.000 :103(release)
      463    0.001    0.000    0.001    0.000 :78(acquire)
    40/35    0.000    0.000    0.001    0.000 constructor.py:44(construct_yaml_map)
      592    0.001    0.000    0.001    0.000 _weakrefset.py:36(__init__)
        2    0.000    0.000    0.001    0.001 __init__.py:10()
        1    0.000    0.000    0.001    0.001 parser.py:899(parse)
      180    0.000    0.000    0.001    0.000 manager.py:390(_loop_entries)
        1    0.000    0.000    0.001    0.001 argparse.py:1604(__init__)
     2371    0.001    0.000    0.001    0.000 sre_parse.py:171(append)
      508    0.001    0.000    0.001    0.000 {built-in method builtins.any}
        1    0.000    0.000    0.001    0.001 util.py:10()
       20    0.000    0.000    0.001    0.000 constructor.py:51(construct_mapping)
        2    0.000    0.000    0.001    0.001 x25519.py:5()
        1    0.000    0.000    0.001    0.001 parser.py:851(subparse)
       30    0.001    0.000    0.001    0.000 enum.py:160()
        1    0.000    0.000    0.001    0.001 _distro.py:33()
      3/1    0.000    0.000    0.001    0.001 parser.py:586(parse_tuple)
     6173    0.001    0.000    0.001    0.000 resolver.py:114(ascend_resolver)
      6/1    0.000    0.000    0.001    0.001 parser.py:426(parse_expression)
      6/1    0.000    0.000    0.001    0.001 parser.py:435(parse_condexpr)
      6/1    0.000    0.000    0.001    0.001 parser.py:448(parse_or)
      6/1    0.000    0.000    0.001    0.001 parser.py:457(parse_and)
      6/1    0.000    0.000    0.001    0.001 parser.py:466(parse_not)
        2    0.000    0.000    0.001    0.001 x448.py:5()
      6/1    0.000    0.000    0.001    0.001 parser.py:472(parse_compare)
      6/1    0.000    0.000    0.001    0.001 parser.py:494(parse_math1)
      7/1    0.000    0.000    0.001    0.001 parser.py:505(parse_concat)
      7/1    0.000    0.000    0.001    0.001 parser.py:515(parse_math2)
      7/1    0.000    0.000    0.001    0.001 parser.py:526(parse_pow)
      7/1    0.000    0.000    0.001    0.001 parser.py:536(parse_unary)
      7/1    0.000    0.000    0.001    0.001 parser.py:552(parse_primary)
        1    0.000    0.000    0.001    0.001 version.py:93(StrictVersion)
        1    0.000    0.000    0.001    0.001 datetime.py:5()
        1    0.000    0.000    0.001    0.001 six.py:21()
     6173    0.001    0.000    0.001    0.000 resolver.py:91(descend_resolver)
      624    0.001    0.000    0.001    0.000 :63(_path_split)
        1    0.000    0.000    0.001    0.001 runtime.py:10()
       45    0.000    0.000    0.001    0.000 :1322(path_hook_for_FileFinder)
     1876    0.001    0.000    0.001    0.000 sre_compile.py:102(fixup)
      219    0.001    0.000    0.001    0.000 data.py:158(add_group)
     1566    0.001    0.000    0.001    0.000 sre_parse.py:285(tell)
     2907    0.001    0.000    0.001    0.000 {method 'rpartition' of 'str' objects}
        1    0.000    0.000    0.001    0.001 oid.py:5()
      440    0.001    0.000    0.001    0.000 sre_compile.py:388(_simple)
      191    0.000    0.000    0.001    0.000 manager.py:173(get_config_type)
        1    0.000    0.000    0.001    0.001 _parseaddr.py:7()
     1171    0.001    0.000    0.001    0.000 sre_parse.py:111(__init__)
     1575    0.001    0.000    0.001    0.000 nodes.py:36(__init__)
      354    0.001    0.000    0.001    0.000 :176(cb)
      422    0.001    0.000    0.001    0.000 enum.py:353(__setattr__)
        1    0.001    0.001    0.001    0.001 manager.py:428()
        1    0.000    0.000    0.001    0.001 cyaml.py:2()
      124    0.000    0.000    0.001    0.000 abc.py:135()
      344    0.001    0.000    0.001    0.000 sre_parse.py:342(_escape)
        1    0.000    0.000    0.001    0.001 futures.py:1()
        1    0.000    0.000    0.001    0.001 collection_loader.py:334(AnsibleCollectionRef)
        2    0.000    0.000    0.001    0.000 plugin_docs.py:4()
        1    0.000    0.000    0.001    0.001 constant_time.py:5()
        1    0.000    0.000    0.001    0.001 process.py:4()
        1    0.000    0.000    0.001    0.001 hashlib.py:54()
       11    0.000    0.000    0.001    0.000 enum.py:760(_missing_)
      259    0.000    0.000    0.001    0.000 os.py:664(__getitem__)
        2    0.000    0.000    0.001    0.000 ed448.py:5()
        1    0.000    0.000    0.001    0.001 __init__.py:45(CLI)
        1    0.000    0.000    0.001    0.001 __init__.py:26()
       80    0.000    0.000    0.001    0.000 functools.py:44(update_wrapper)
       11    0.000    0.000    0.001    0.000 enum.py:767(_create_pseudo_member_)
      118    0.000    0.000    0.001    0.000 re.py:169(match)
     2738    0.001    0.000    0.001    0.000 {built-in method builtins.min}
       58    0.000    0.000    0.001    0.000 sre_compile.py:376(_mk_bitmap)
       16    0.000    0.000    0.001    0.000 lexer.py:349(__next__)
     1697    0.001    0.000    0.001    0.000 sre_parse.py:159(__len__)
        1    0.000    0.000    0.001    0.001 argparse.py:1733(parse_args)
        1    0.000    0.000    0.001    0.001 argparse.py:1740(parse_known_args)
      354    0.001    0.000    0.001    0.000 :58(__init__)
      107    0.000    0.000    0.001    0.000 :194(_lock_unlock_module)
        2    0.000    0.000    0.001    0.000 ed25519.py:5()
        1    0.000    0.000    0.001    0.001 bccache.py:16()
       16    0.000    0.000    0.001    0.000 lexer.py:558(wrap)
        1    0.000    0.000    0.001    0.001 environment.py:536(_generate)
        1    0.000    0.000    0.001    0.001 compiler.py:74(generate)
     1298    0.001    0.000    0.001    0.000 inspect.py:2841()
     1108    0.001    0.000    0.001    0.000 :847(__exit__)
       11    0.000    0.000    0.001    0.000 enum.py:839(_decompose)
     1298    0.001    0.000    0.001    0.000 inspect.py:2781()
      616    0.000    0.000    0.001    0.000 :52(_r_long)
     41/1    0.000    0.000    0.001    0.001 visitor.py:34(visit)
        1    0.000    0.000    0.001    0.001 compiler.py:695(visit_Template)
      184    0.000    0.000    0.001    0.000 py3compat.py:47(__getitem__)
        1    0.000    0.000    0.001    0.001 algorithms.py:5()
        1    0.000    0.000    0.001    0.001 argparse.py:1775(_parse_known_args)
       56    0.000    0.000    0.001    0.000 coroutines.py:194(coroutine)
        1    0.000    0.000    0.001    0.001 re.py:179(search)
      145    0.000    0.000    0.001    0.000 constructor.py:89(construct_yaml_str)
        1    0.000    0.000    0.001    0.001 ciphers.py:5()
        4    0.000    0.000    0.001    0.000 loader.py:205(_all_directories)
      344    0.001    0.000    0.001    0.000 {method 'format' of 'str' objects}
      364    0.000    0.000    0.001    0.000 _weakrefset.py:26(__exit__)
     1108    0.000    0.000    0.001    0.000 :843(__enter__)
        3    0.000    0.000    0.001    0.000 {built-in method builtins.compile}
        1    0.000    0.000    0.001    0.001 argparse.py:62()
      112    0.001    0.000    0.001    0.000 _oid.py:11(__init__)
        1    0.000    0.000    0.001    0.001 __init__.py:24()
    20/12    0.000    0.000    0.001    0.000 os.py:277(walk)
        1    0.000    0.000    0.001    0.001 encoder.py:2()
        1    0.000    0.000    0.001    0.001 exceptions.py:5()
        1    0.001    0.001    0.001    0.001 extensions.py:1371(CRLReason)
      416    0.001    0.000    0.001    0.000 inspect.py:2844()
        1    0.000    0.000    0.001    0.001 ipaddress.py:2233(_IPv6Constants)
        1    0.000    0.000    0.001    0.001 loader.py:816(_configure_collection_loader)
        1    0.000    0.000    0.001    0.001 context_objects.py:5()
        1    0.000    0.000    0.001    0.001 lzma.py:9()
       61    0.000    0.000    0.001    0.000 posixpath.py:232(expanduser)
       28    0.000    0.000    0.001    0.000 ipaddress.py:2143(__init__)
       46    0.000    0.000    0.001    0.000 loader.py:446()
        1    0.000    0.000    0.001    0.001 collection_loader.py:34(__init__)
       16    0.000    0.000    0.001    0.000 argparse.py:2352(_get_formatter)
      681    0.000    0.000    0.001    0.000 {built-in method from_bytes}
        1    0.000    0.000    0.001    0.001 hmac.py:4()
       58    0.001    0.000    0.001    0.000 sre_compile.py:378()
      549    0.000    0.000    0.001    0.000 _weakrefset.py:81(add)
      242    0.000    0.000    0.001    0.000 enum.py:20(_is_descriptor)
        1    0.000    0.000    0.001    0.001 numbers.py:6()
        1    0.000    0.000    0.001    0.001 asyncsupport.py:169(patch_all)
        1    0.000    0.000    0.001    0.001 certificate_transparency.py:5()
        1    0.000    0.000    0.001    0.001 errors.py:5()
      944    0.000    0.000    0.001    0.000 inspect.py:159(isfunction)
        1    0.000    0.000    0.001    0.001 ipaddress.py:1542(_IPv4Constants)
      316    0.001    0.000    0.001    0.000 :393(_check_name_wrapper)
       16    0.000    0.000    0.001    0.000 argparse.py:157(__init__)
       45    0.000    0.000    0.001    0.000 :1196(__init__)
        1    0.000    0.000    0.001    0.001 file.py:4()
      416    0.000    0.000    0.001    0.000 inspect.py:505(_is_wrapper)
      497    0.000    0.000    0.001    0.000 :416(parent)
       19    0.000    0.000    0.001    0.000 ipaddress.py:1460(__init__)
      112    0.001    0.000    0.001    0.000 ajson.py:30(object_hook)
        1    0.000    0.000    0.001    0.001 subprocess.py:42()
        1    0.000    0.000    0.001    0.001 getpass.py:11()
      176    0.000    0.000    0.001    0.000 __init__.py:877(__getitem__)
        4    0.000    0.000    0.001    0.000 manager.py:416(get_config_value)
      441    0.000    0.000    0.001    0.000 inspect.py:2561(__eq__)
       30    0.000    0.000    0.001    0.000 enum.py:114(__prepare__)
        1    0.000    0.000    0.001    0.001 opcode.py:5()
     2072    0.000    0.000    0.000    0.000 {built-in method _sre.getlower}
      190    0.000    0.000    0.000    0.000 configparser.py:1131(_unify_values)
     1440    0.000    0.000    0.000    0.000 {method 'find' of 'bytearray' objects}
        1    0.000    0.000    0.000    0.000 asyncsupport.py:163(patch_filters)
       10    0.000    0.000    0.000    0.000 enum.py:857()
        6    0.000    0.000    0.000    0.000 argparse.py:1226(__init__)
        1    0.000    0.000    0.000    0.000 __init__.py:71(search_function)
        1    0.000    0.000    0.000    0.000 base64.py:3()
      308    0.000    0.000    0.000    0.000 {built-in method _imp._fix_co_filename}
      176    0.000    0.000    0.000    0.000 configparser.py:245(__init__)
      137    0.000    0.000    0.000    0.000 sre_parse.py:223(__init__)
     1925    0.000    0.000    0.000    0.000 {built-in method _imp.acquire_lock}
        5    0.000    0.000    0.000    0.000 display.py:244(verbose)
      361    0.000    0.000    0.000    0.000 :369(__init__)
      139    0.000    0.000    0.000    0.000 sre_parse.py:84(opengroup)
        5    0.000    0.000    0.000    0.000 {built-in method builtins.dir}
        1    0.000    0.000    0.000    0.000 cmac.py:5()
        1    0.000    0.000    0.000    0.000 {built-in method _openssl.SSL_library_init}
       38    0.000    0.000    0.000    0.000 constructor.py:114(construct_yaml_seq)
        1    0.000    0.000    0.000    0.000 pickle.py:181()
        1    0.000    0.000    0.000    0.000 version.py:267(LooseVersion)
        4    0.000    0.000    0.000    0.000 argparse.py:1843(consume_optional)
       30    0.000    0.000    0.000    0.000 enum.py:464(_find_new_)
        3    0.000    0.000    0.000    0.000 gettext.py:611(gettext)
      109    0.000    0.000    0.000    0.000 group.py:62(__init__)
        3    0.000    0.000    0.000    0.000 gettext.py:572(dgettext)
      308    0.000    0.000    0.000    0.000 :35(_new_module)
     1925    0.000    0.000    0.000    0.000 {built-in method _imp.release_lock}
      259    0.000    0.000    0.000    0.000 os.py:742(encode)
        1    0.000    0.000    0.000    0.000 toml.py:4()
      347    0.000    0.000    0.000    0.000 :780(find_spec)
        2    0.000    0.000    0.000    0.000 __init__.py:18()
        3    0.000    0.000    0.000    0.000 gettext.py:506(translation)
        6    0.000    0.000    0.000    0.000 {method 'write' of '_io.TextIOWrapper' objects}
        1    0.000    0.000    0.000    0.000 loader.py:43(add_all_plugin_dirs)
        1    0.000    0.000    0.000    0.000 calendar.py:6()
        1    0.000    0.000    0.000    0.000 locks.py:1()
        2    0.000    0.000    0.000    0.000 fnmatch.py:38(_compile_pattern)
        3    0.000    0.000    0.000    0.000 gettext.py:466(find)
      364    0.000    0.000    0.000    0.000 _weakrefset.py:20(__enter__)
        1    0.000    0.000    0.000    0.000 streams.py:1()
       47    0.000    0.000    0.000    0.000 :99(_path_isdir)
       73    0.000    0.000    0.000    0.000 enum.py:419(_get_mixins_)
        1    0.000    0.000    0.000    0.000 parameters.py:5()
     1364    0.000    0.000    0.000    0.000 :321()
        1    0.000    0.000    0.000    0.000 encoders.py:5()
        1    0.000    0.000    0.000    0.000 ini.py:3()
     2731    0.000    0.000    0.000    0.000 {built-in method builtins.ord}
        1    0.000    0.000    0.000    0.000 ajson.py:5()
        1    0.000    0.000    0.000    0.000 glob.py:1()
        1    0.000    0.000    0.000    0.000 scrypt.py:5()
      341    0.000    0.000    0.000    0.000 :311(__enter__)
      351    0.000    0.000    0.000    0.000 :707(find_spec)
        1    0.000    0.000    0.000    0.000 bisect.py:1()
        2    0.000    0.000    0.000    0.000 __init__.py:327(__init__)
        1    0.000    0.000    0.000    0.000 shlex.py:1()
        1    0.000    0.000    0.000    0.000 collection_loader.py:531(get_collection_name_from_path)
        1    0.000    0.000    0.000    0.000 pkcs7.py:5()
        1    0.000    0.000    0.000    0.000 tokens.py:2()
      555    0.000    0.000    0.000    0.000 sre_parse.py:81(groups)
        3    0.000    0.000    0.000    0.000 {method 'read' of '_io.BufferedReader' objects}
        1    0.000    0.000    0.000    0.000 ocsp.py:104(_OCSPResponse)
      105    0.000    0.000    0.000    0.000 sre_compile.py:441(_get_charset_prefix)
      137    0.000    0.000    0.000    0.000 {built-in method _sre.compile}
  166/122    0.000    0.000    0.000    0.000 sre_compile.py:414(_get_literal_prefix)
        7    0.000    0.000    0.000    0.000 argparse.py:2050(_match_argument)
      119    0.000    0.000    0.000    0.000 utils.py:39(read_only_property)
        1    0.000    0.000    0.000    0.000 representer.py:2()
      338    0.000    0.000    0.000    0.000 enum.py:28(_is_dunder)
      201    0.000    0.000    0.000    0.000 _oid.py:67(__hash__)
        1    0.000    0.000    0.000    0.000 option_helpers.py:274(add_inventory_options)
        1    0.000    0.000    0.000    0.000 manager.py:331(_match_list)
       56    0.000    0.000    0.000    0.000 types.py:211(coroutine)
       19    0.000    0.000    0.000    0.000 ocsp.py:32(_requires_successful_response)
      718    0.000    0.000    0.000    0.000 {built-in method _thread.allocate_lock}
      353    0.000    0.000    0.000    0.000 socket.py:86()
       56    0.000    0.000    0.000    0.000 :861(_find_spec_legacy)
        1    0.000    0.000    0.000    0.000 exceptions.py:10()
     1764    0.000    0.000    0.000    0.000 inspect.py:2514(kind)
      354    0.000    0.000    0.000    0.000 socket.py:91()
        1    0.000    0.000    0.000    0.000 events.py:4()
        1    0.000    0.000    0.000    0.000 tempfile.py:347(mkdtemp)
        1    0.000    0.000    0.000    0.000 events.py:1()
      113    0.000    0.000    0.000    0.000 nodes.py:164(iter_child_nodes)
       20    0.000    0.000    0.000    0.000 ipaddress.py:1099(_ip_int_from_string)
      927    0.000    0.000    0.000    0.000 {built-in method _thread.get_ident}
       12    0.000    0.000    0.000    0.000 display.py:398(_output_encoding)
       22    0.000    0.000    0.000    0.000 lexer.py:599(tokeniter)
      109    0.000    0.000    0.000    0.000 group.py:32(to_safe_group_name)
        1    0.000    0.000    0.000    0.000 subprocess.py:1()
        1    0.000    0.000    0.000    0.000 utils.py:53(register_decorator)
        1    0.000    0.000    0.000    0.000 argparse.py:1920(consume_positionals)
        1    0.000    0.000    0.000    0.000 tasks.py:1()
        1    0.000    0.000    0.000    0.000 collection_loader.py:539()
      351    0.000    0.000    0.000    0.000 socket.py:76()
      131    0.000    0.000    0.000    0.000 __init__.py:422()
        1    0.000    0.000    0.000    0.000 manager.py:79(__init__)
        1    0.000    0.000    0.000    0.000 {built-in method _openssl.SSL_load_error_strings}
      832    0.000    0.000    0.000    0.000 {method 'values' of 'mappingproxy' objects}
      341    0.000    0.000    0.000    0.000 :307(__init__)
      356    0.000    0.000    0.000    0.000 :143(__init__)
       51    0.000    0.000    0.000    0.000 loader.py:442()
      352    0.000    0.000    0.000    0.000 socket.py:81()
        1    0.000    0.000    0.000    0.000 argparse.py:2071(_match_arguments_partial)
       12    0.000    0.000    0.000    0.000 locale.py:631(getpreferredencoding)
       32    0.000    0.000    0.000    0.000 sre_parse.py:266(getuntil)
        1    0.000    0.000    0.000    0.000 configparser.py:521(LegacyInterpolation)
        1    0.000    0.000    0.000    0.000 ast.py:26(

@ansibot ansibot added affects_2.9 This issue/PR affects Ansible v2.9 bug This issue/PR relates to a bug. inventory Inventory category needs_triage Needs a first human triage before being processed. labels Jan 4, 2023
@nitzmahone nitzmahone removed the needs_triage Needs a first human triage before being processed. label Jan 5, 2023
@nitzmahone
Copy link
Member

2.9 is no longer supported- can you reproduce this on a currently supported version of Ansible?

@mdidomenico4
Copy link
Author

using pip3.6 and ansible-core-2.11

$ ANSIBLE_SKIP_CONFLICT_CHECK=1 pip3.6 install --prefix $HOME/ansitest ansible==4.10.0
Collecting ansible==4.10.0
  Using cached https://files.pythonhosted.org/packages/fd/f8/071905c6a67592d0852a9f340f6ab9226861eeeb97fdf4068642b22edcf3/ansible-4.10.0.tar.gz
Collecting ansible-core~=2.11.7 (from ansible==4.10.0)
  Downloading https://files.pythonhosted.org/packages/98/ea/2935bf0864196cd2c9d14548e399a110f48b3540664ddc462b39ff0b822d/ansible-core-2.11.12.tar.gz (7.1MB)
    100% |████████████████████████████████| 7.1MB 129kB/s
Requirement already satisfied: jinja2 in /usr/lib/python3.6/site-packages (from ansible-core~=2.11.7->ansible==4.10.0)
Requirement already satisfied: PyYAML in /usr/lib64/python3.6/site-packages (from ansible-core~=2.11.7->ansible==4.10.0)
Requirement already satisfied: cryptography in /usr/lib64/python3.6/site-packages (from ansible-core~=2.11.7->ansible==4.10.0)
Collecting packaging (from ansible-core~=2.11.7->ansible==4.10.0)
  Downloading https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl (40kB)
    100% |████████████████████████████████| 40kB 7.4MB/s
Collecting resolvelib<0.6.0,>=0.5.3 (from ansible-core~=2.11.7->ansible==4.10.0)
  Downloading https://files.pythonhosted.org/packages/eb/11/bda2b7dee2c84d1f1923ae273023bb94d3e5ab3d1a46b4bd8cf5eb81a241/resolvelib-0.5.4-py2.py3-none-any.whl
Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib64/python3.6/site-packages (from jinja2->ansible-core~=2.11.7->ansible==4.10.0)
Requirement already satisfied: six>=1.4.1 in /usr/local/lib/python3.6/site-packages (from cryptography->ansible-core~=2.11.7->ansible==4.10.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/lib64/python3.6/site-packages (from cryptography->ansible-core~=2.11.7->ansible==4.10.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3.6/site-packages (from packaging->ansible-core~=2.11.7->ansible==4.10.0)
Requirement already satisfied: pycparser in /usr/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography->ansible-core~=2.11.7->ansible==4.10.0)
Installing collected packages: packaging, resolvelib, ansible-core, ansible
  Running setup.py install for ansible-core ... done
  Running setup.py install for ansible ... done
Successfully installed ansible-4.10.0 ansible-core-2.11.12 packaging-21.3 resolvelib-0.5.4

$ export LD_LIBRARY_PATH=$HOME/ansitest/lib
$ export PYTHONPATH=$HOME/ansitest/lib/python3.6/site-packages


[user@ansible1 inventory-testing]$ ~/ansitest/bin/ansible --version
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Oct 20 2022, 09:31:56) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)].
 This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ansible [core 2.11.12]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/u/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /u/user/ansitest/lib/python3.6/site-packages/ansible
  ansible collection location = /u/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /u/user/ansitest/bin/ansible
  python version = 3.6.8 (default, Oct 20 2022, 09:31:56) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)]
  jinja version = 2.10.1
  libyaml = True

[user@ansible1 inventory-testing]$ time ~/ansitest/bin/ansible-inventory -i ./inv.py --list >/dev/null
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the
controller starting with Ansible 2.12. Current version: 3.6.8 (default, Oct 20
2022, 09:31:56) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)]. This feature will be
removed from ansible-core in version 2.12. Deprecation warnings can be disabled
 by setting deprecation_warnings=False in ansible.cfg.

real	1m47.501s
user	1m41.303s
sys	0m5.709s

@mdidomenico4
Copy link
Author

mdidomenico4 commented Jan 5, 2023 •
edited

using pip3.9 and ansible-core-2.14

[user@ansible1 ~]$ pip3.9 install --prefix ansi39 ansible==7.1.0
Collecting ansible==7.1.0
  Downloading ansible-7.1.0-py3-none-any.whl (42.4 MB)
     |████████████████████████████████| 42.4 MB 10.2 MB/s
Collecting ansible-core~=2.14.1
  Downloading ansible_core-2.14.1-py3-none-any.whl (2.2 MB)
     |████████████████████████████████| 2.2 MB 76.7 MB/s
Collecting cryptography
  Downloading cryptography-39.0.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB)
     |████████████████████████████████| 4.2 MB 73.9 MB/s
Collecting packaging
  Downloading packaging-22.0-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 5.3 MB/s
Collecting PyYAML>=5.1
  Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB)
     |████████████████████████████████| 661 kB 77.7 MB/s
Collecting jinja2>=3.0.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 80.2 MB/s
Collecting resolvelib<0.9.0,>=0.5.3
  Downloading resolvelib-0.8.1-py2.py3-none-any.whl (16 kB)
Collecting cffi>=1.12
  Downloading cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
     |████████████████████████████████| 441 kB 84.0 MB/s
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     |████████████████████████████████| 118 kB 103.5 MB/s
Installing collected packages: pycparser, cffi, cryptography, packaging, PyYAML, MarkupSafe, jinja2, resolvelib, ansible-core, ansible
  WARNING: The scripts ansible, ansible-config, ansible-connection, ansible-console, ansible-doc, ansible-galaxy, ansible-inventory, ansible-playbook, ansible-pull and ansible-vault are installed in 'ansi39/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script ansible-community is installed in 'ansi39/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed MarkupSafe-2.1.1 PyYAML-6.0 ansible-7.1.0 ansible-core-2.14.1 cffi-1.15.1 cryptography-39.0.0 jinja2-3.1.2 packaging-22.0 pycparser-2.21 resolvelib-0.8.1
[user@ansible1 ~]$ pip3.9 install --prefix ansi39 markupsafe
Collecting markupsafe
  Using cached MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Installing collected packages: markupsafe
Successfully installed markupsafe-2.1.1

$ export LD_LIBRARY_PATH=$HOME/ansi39/lib:$HOME/ansi39/lib64
$ export PYTHONPATH=$HOME/ansi39/lib/python3.9/site-packages:$HOME/ansi39/lib64/python3.9/site-packages

[user@ansible1 inventory-testing]$ time ~/ansi39/bin/ansible --version
ansible [core 2.14.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/u/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /u/user/ansi39/lib/python3.9/site-packages/ansible
  ansible collection location = /u/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /u/user/ansi39/bin/ansible
  python version = 3.9.13 (main, Nov  9 2022, 13:16:24) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] (/usr/bin/python3.9)
  jinja version = 3.1.2
  libyaml = True

[user@ansible1 inventory-testing]$ time ~/ansi39/bin/ansible-inventory -i ./inv.py --list >/dev/null

real	0m33.662s
user	0m28.221s
sys	0m5.261s

@nitzmahone
Copy link
Member

nitzmahone commented Jan 5, 2023 •
edited

Cool, thanks- I've put together a standalone inventory script that roughly matches your description, and definitely see what appears to be some unintended exponential behavior in ansible-inventory that isn't occurring when using the same inventory at runtime in Ansible (eg, on devel, using the inventory in ansible takes ~6s, passing to ansible-inventory takes ~65s). Stay tuned!

@nitzmahone nitzmahone added affects_2.14 affects_2.15 verified This issue has been verified/reproduced by maintainer and removed affects_2.9 This issue/PR affects Ansible v2.9 labels Jan 5, 2023
@bcoca
Copy link
Member

bcoca commented Jan 6, 2023

probably related on how vars_plugins are called and groups processed, these were made so --export would return closer to 'original inventory definition' and not the 'inventory as ansible sees it'.

@mdidomenico4
Copy link
Author

Cool, thanks- I've put together a standalone inventory script that roughly matches your description, and definitely see what appears to be some unintended exponential behavior in ansible-inventory that isn't occurring when using the same inventory at runtime in Ansible (eg, on devel, using the inventory in ansible takes ~6s, passing to ansible-inventory takes ~65s). Stay tuned!

that's an interesting fact. i'm still new to ansible and i hadn't even testing running a playbook with this massive inventory. because ansible-inventory was taking minutes to process the inventory (and since i was following the docs) i assumed it wasn't a feasible option to run it with ansible -i. i'm a little surprised ansible-inventory and ansible are using different code paths, seems like a duplication of effort

i haven't been able to find yet if there's an ansible command that matches 'ansible-inventory -i ./inv.py --list' that also shows the inventory in the same json formatted way. i tried a test like this which is close and seems to pan out with what you see.

[inventory-testing]$ time ansible -i ./inv.py -c local --list-hosts all > /dev/null
real 0m2.412s
user 0m2.182s
sys 0m0.218s

@nitzmahone
Copy link
Member

They're not so much using different code paths as interacting with the same code in wildly different ways. ansible-inventory was added (relatively) recently to expose "partially-chewed" inventory data to AWX. So where core calls a bunch of that stuff lazily as it hits a host for the first time (ie "never" if your playbook doesn't hit that host), ansible-inventory might cause it to get called millions of times in a row since it's doing it for every host/group in the inventory serially.

Turns out, there's a lot of sneaky and unnecessary disk I/O that's kinda buried in the noise during a typical core run (and thus flies below our usual "hot code paths" radar when we're looking at perf stuff) that is making up the vast majority of time spent by ansible-inventory on large inventories. It's not really an exponential algorithm problem, but the disk I/O in question is so much slower than the CPU-bound stuff that the larger the inventory is, the more outsized effect it has on the wall-clock time. There's also plenty of stuff in the actual var manipulation code that could be further sped up and/or simplified (@mkrizek has been working on some of that), but at least for large inventories, the disk I/O was the real killer.

Once I had a repro of the behavior you reported, I just started doing some iterative cleanup of hot code paths on it under ansible-inventory and was able to get about an order of magnitude performance increase on an inventory with 100 groups and 50000 hosts (from ~50 seconds to 5.5s on 2.15 devel + my changes). My changes aren't "production ready" yet, but they're simple enough that we can get them cleaned up and released with 2.15, and possibly backport fixes for a few of the worst offenders to 2.14. This also lines up well with some work the AWX folks are doing where they were like "why is ansible-inventory so much slower than core on XYZ?", so thanks for the bug report! You should see some nice boosts start showing up soon on this.

@mdidomenico4
Copy link
Author

thanks, thats great news. i also recently learned that back in april redhat swapped the ansible rpm/repos around and i should have used ansible-core instead of ansible during my dnf install. now i'm at ansible 2.13 on the hosts now. the performance is better, down to 14 seconds or so. hopefully i'll see the performance patches sooner rather then later trough redhat.

@nitzmahone nitzmahone linked a pull request Jan 6, 2023 that will close this issue
Draft
@ansibot ansibot added the has_pr This issue has an associated PR. label Jan 6, 2023
@emmanuel-benoit
Copy link

emmanuel-benoit commented Mar 6, 2023 •
edited

A posteriori edit I misread the initial posts and my own problem is most likely unrelated. Sorry.

I make rather heavy use of dynamic inventories and one major cause for slow inventory build for me seems to be method __getitem__ from class JinjaPluginIntercept in template/__init__.py.

The main reason for that is that, every time the method is called (94648 times in my current example), it calls the plugin loader's get method.

Simply changing the :

            # if a plugin was found/loaded
            if plugin:
                # set in filter cache and avoid expensive plugin load
                self._delegatee[key] = plugin.j2_function
                self._loaded_builtins.add(key)

part of the code to

            # if a plugin was found/loaded
            if plugin:
                # set in filter cache and avoid expensive plugin load
                self._delegatee[key] = plugin.j2_function
            self._loaded_builtins.add(key)

solves it for me (it simply prevents the same lookup from being executed time and time again). The self._loaded_builtins set is not used anywhere else, the actual checks later on in the method are based on self._delegatee.

⚠️ Huge disclaimer though, I haven't tried running an actual playbook with that change in place yet, I've only been concerned with ansible-inventory.

time output without the change:

22.38user 3.80system 0:27.20elapsed 96%CPU (0avgtext+0avgdata 71188maxresident)k
0inputs+0outputs (0major+20729minor)pagefaults 0swaps

time output with the change:

3.73user 0.10system 0:04.67elapsed 81%CPU (0avgtext+0avgdata 71016maxresident)k
0inputs+96outputs (0major+20706minor)pagefaults 0swaps

@nitzmahone
Copy link
Member

If everything were in collections, it'd be (something close to) that easy, but unfortunately to preserve some of the pre-existing non-collection Jinja plugin load semantics, we have to go back to the plugin loader to be safe... That said, there are plenty of cases where we can short-circuit to what's already loaded and/or safely cache the Jinja environment across Templar calls/instances. There's work actively occurring to figure out which ones we can do in a backportable fashion.

@rptaylor rptaylor mentioned this issue Jan 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects_2.14 affects_2.15 affects_2.16 bug This issue/PR relates to a bug. has_pr This issue has an associated PR. inventory Inventory category verified This issue has been verified/reproduced by maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prototype vars and inventory dump performance tweaks nitzmahone/ansible
6 participants
@bcoca @ansibot @nitzmahone @jborean93 @mdidomenico4 @emmanuel-benoit