feat(reposerver): Skip calling git fetch if commit to checkout exists locally

[shady-canva]

Problem

Our monorepo is frequently updated with hundreds of application changes in a very short period of time (within 15 to 30 minutes). This leads to a large amount of calls to checkoutRevision in the reposerver, where git fetch is called each time. Since checkoutRevision is wrapped in a lock, it becomes a bottleneck that causes the application controller's work-queue depth to blow up:

Solution

When we have a lot of commits landing around the same time, chances are a previous git fetch would have already pulled in the commit SHA we want to checkout. Therefore, this PR introduces a check to see if the revision being checked out already exists locally. If so, git fetch is skipped and we move on directly to git checkout.

Outcome

Decreased git fetch calls

We've had this change deployed in production for a month now, and so far we've seen an 80% decrease in git fetch calls. Over a two week period, we ended up skipping 134,000 calls we would have otherwise made:

Decreased work-queue depth

Our application-controller has been a lot healthier since we made this change. During most peak periods, we only see a work-queue depth in the single digits now. Previously we were dealing with a depth exceeding 400.

Decreased sync times

Our application sync durations are now steady and predictable, usually in the range of 20 to 30 seconds, no matter the load. Prior to the change, we were dealing with sync durations in the 5 to 10 minute range.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[codecov]

Codecov Report

Attention: Patch coverage is 66.66667% with 9 lines in your changes missing coverage. Please review.

Project coverage is 47.58%. Comparing base (9f1e2e8) to head (6ec03b8).
Report is 521 commits behind head on master.

Files with missing lines	Patch %	Lines
util/git/mocks/Client.go	50.00%	2 Missing and 3 partials ⚠️
reposerver/repository/repository.go	75.00%	1 Missing and 1 partial ⚠️
util/git/client.go	77.77%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18657      +/-   ##
==========================================
+ Coverage   47.56%   47.58%   +0.01%     
==========================================
  Files         332      332              
  Lines       46746    46771      +25     
==========================================
+ Hits        22236    22256      +20     
  Misses      21665    21665              
- Partials     2845     2850       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ishitasequeira]

Thanks @shady-canva. The change LGTM 🚀
@crenshaw-dev do you have any other concerns?

Old

[crenshaw-dev]

lgtm, let's see how it behaves!

[christianh814]

Awesome, thanks for this!

[jenting]

Awesome 👏

Will this backport to the previous versions?

[blakeromano]

This is much appreciated! 🙌

[bouchrihamomo]

thanks for this 🙌