Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new parameter for 'git submodule update' that fixes a performance issue. #65634

Open
wants to merge 1 commit into
base: devel
Choose a base branch
from

Conversation

elmeriniemela
Copy link

@elmeriniemela elmeriniemela commented Dec 8, 2019

SUMMARY

Currently, if you use ansible with git submodules every time you update the supermodule you must use recursive=True or ansible will not run 'git submodule update'. Problem with recursive=True is that it fetches every submodule even if there are changes in only one of them. This causes a huge performance issue with repositories that have a lot of submodules.

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

git

ADDITIONAL INFORMATION

Since our product consists of thousands of small modules, handle our customer and testing deployments with git submodules. A typical customer has from 5-100 modules. When we're testing customer environments we may have to do the submodule pull every 20 minutes.

Example situation: A git supermodule having 78 submodules and only one new commit having only one of the submodules changed.

  • Currently with the parameter recursive=True running the git task takes 228 seconds.
  • After this change with recursive=False and update_submodules=True it takes 18 seconds.

That's a 92% decrease in time spent in that git task and the result is exactly the same.

@ansibot ansibot added affects_2.10 core_review feature module needs_triage new_contributor performance source_control support:core labels Dec 8, 2019
@ansibot
Copy link
Contributor

@ansibot ansibot commented Dec 8, 2019

The test ansible-test sanity --test validate-modules [explain] failed with 1 error:

lib/ansible/modules/source_control/git.py:0:0: option-incorrect-version-added: version_added for new option (update_submodules) should be '2.10'. Currently StrictVersion ('2.9')

click here for bot help

@ansibot ansibot added needs_revision and removed core_review labels Dec 8, 2019
@elmeriniemela elmeriniemela force-pushed the git-submodule-performance-fix branch 2 times, most recently from c4b0ec7 to 19aed86 Compare Dec 8, 2019
@ansibot ansibot added core_review and removed needs_revision labels Dec 8, 2019
@ansibot ansibot added the stale_ci label Dec 16, 2019
Copy link
Contributor

@samdoran samdoran left a comment

Please create a changelog fragment and integration tests. See this fragment as an example.

lib/ansible/modules/source_control/git.py Show resolved Hide resolved
@samdoran samdoran removed the needs_triage label Dec 19, 2019
@ansibot ansibot added needs_revision and removed core_review labels Dec 19, 2019
@elmeriniemela elmeriniemela force-pushed the git-submodule-performance-fix branch from 19aed86 to a0aa14e Compare Jan 12, 2020
@ansibot ansibot added support:community and removed stale_ci labels Jan 12, 2020
@elmeriniemela
Copy link
Author

@elmeriniemela elmeriniemela commented Jan 12, 2020

Please create a changelog fragment and integration tests. See this fragment as an example.

Added a changelog fragment. This is already in production for us with daily usage and it's working well, so I won't spend time learning how you're testing framework works. This bug fix will come with take it or leave it terms.

@ansibot ansibot added stale_ci stale_review labels Jan 20, 2020
@elmeriniemela elmeriniemela requested a review from samdoran Jan 26, 2020
@elmeriniemela elmeriniemela force-pushed the git-submodule-performance-fix branch from a0aa14e to ca4e112 Compare Feb 13, 2020
@ansibot ansibot removed stale_ci stale_review labels Feb 13, 2020
@ansibot ansibot added stale_ci stale_review labels Feb 21, 2020
@ansibot ansibot added core_review and removed needs_revision stale_review labels Mar 8, 2020
@ansibot ansibot added needs_revision stale_review and removed core_review labels Mar 24, 2020
@ansibot ansibot added needs_rebase and removed stale_review labels May 16, 2020
@ansibot
Copy link
Contributor

@ansibot ansibot commented May 28, 2020

@elmeriniemela this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

@ansibot ansibot added merge_commit and removed stale_ci labels May 28, 2020
@elmeriniemela elmeriniemela force-pushed the git-submodule-performance-fix branch from 79870f8 to ca4e112 Compare May 28, 2020
@ansibot ansibot removed the merge_commit label May 28, 2020
@ansibot ansibot added the stale_ci label Jun 5, 2020
@ansibot ansibot added pre_azp and removed stale_ci labels Dec 7, 2020
@ansibot ansibot removed the support:community label Mar 4, 2021
@bcoca
Copy link
Member

@bcoca bcoca commented May 4, 2022

waiting_on_contributor

@ansibot ansibot added the waiting_on_contributor label May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects_2.10 feature module needs_rebase needs_revision new_contributor performance pre_azp source_control support:core waiting_on_contributor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants