copy: Reduce calls to stat() by 50% in chown_recursive

[moreati]

SUMMARY

This reduces the number of times lstat() or chown() are called by the copy module. Reductions come from

• combining loops for owner and group, thus not looping twice
• not looping over the list of directories returned by os.walk(), each directory is already handled when os.walk() descends into it

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

copy

ADDITIONAL INFORMATION

Possible further reductions could be achieved by

• exiting early from the check_mode loop
• module_utils: Consolidate set_owner_if_different & set_group_if_different #75742.

I propose to do these seperately, if there is interest.

[moreati]

Some profiling of an extreme case, calls to lstat are cut by 50%

Setup

Create a deep directory tree of approx 18 000 nodes

# rm -rf /tmp/{src,dst} && mkdir -p /tmp/src/{a..z}/{a..z}/{a..z}
# ls -l /tmp/src | head -n2
drwxr-xr-x 28 root root 4096 Sep 22 21:24 a
drwxr-xr-x 28 root root 4096 Sep 22 21:24 b
# find /tmp/src/ | wc -l
18279

Without

# rm -rf /tmp/{src,dst} && mkdir -p /tmp/src/{a..z}/{a..z}/{a..z}
# python3 -m cProfile -s calls -m ansible.modules.copy<<<'{"ANSIBLE_MODULE_ARGS": {"src":"/tmp/src", "dest":"/tmp/dst", "owner":"alex", "group":"alex", "remote_src":true}}' | egrep '(os|posix)\.(chown|lchown|lstat|stat)'
   219396    0.517    0.000    0.517    0.000 {built-in method posix.lstat}
    92156    0.189    0.000    0.189    0.000 {built-in method posix.stat}
    36560    0.143    0.000    0.143    0.000 {built-in method posix.lchown}

With

# rm -rf /tmp/{src,dst} && mkdir -p /tmp/src/{a..z}/{a..z}/{a..z}
# python3 -m cProfile -s calls -m ansible.modules.copy<<<'{"ANSIBLE_MODULE_ARGS": {"src":"/tmp/src", "dest":"/tmp/dst", "owner":"alex", "group":"alex", "remote_src":true}}' | egrep '(os|posix)\.(chown|lchown|lstat|stat)'
   109806    0.261    0.000    0.261    0.000 {built-in method posix.lstat}
    92158    0.191    0.000    0.191    0.000 {built-in method posix.stat}
    36560    0.137    0.000    0.137    0.000 {built-in method posix.lchown}

without.cprof.gz
with.cprof.gz

[moreati]

Rebased on devel, and included the headline number in the summary

[ansibot]

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/modules/copy.py:370:21: disallowed-name: Disallowed name "_"
lib/ansible/modules/copy.py:383:17: disallowed-name: Disallowed name "_"

click here for bot help

[webknjaz]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[webknjaz]

@moreati would you mind rebasing?

[moreati]

@moreati would you mind rebasing?

on it

[moreati]

Notes to self:

• os.walk() is still the lowest common denominator as of upcoming Ansible 11 (ansible-core 2.18, Python 3.11-3.13 on controller, 3.8-3.13 on targets) . pathlib.Path.walk() (using scandir(3)) was introduced in Python 3.12

[bcoca]

check the action plugin, it has it's own 'recursive' code and might need mirroring of these changes

[moreati]

ok, I'll try to take a look tomorrow.

[moreati]

check the action plugin, it has it's own 'recursive' code and might need mirroring of these changes

I don't think ansible.plugins.action.copy._walk_dirs() requires any changes. It's possible there are speedup opportunities, but they would require more in-depth investigation due to extra symlink handling that function does. That could be tackled independantly of this PR. There's no other directory walking code in that Python module that I noticed.

The other major directory walking code I'm aware of are the ansible.module_utils.basic.set_*_if_different() functions. PR #75742 combines the owner and group variants so that chown_recursive() can be sped up even more. If you prefer, then I could combine all the speed-ups into a single PR.

[bcoca]

no, separate is fine, just wanted to make sure they kept as in sync as possible ... hate the duplicity

[s-hertel]

FYI, I have an alternative PR that simplifies this bit of code further #84313.

[mkrizek]

Closing in favor of (already merged) #84313. Thank you for your contribution!