Skip to content

Warm up git commit history in parallel#6130

Merged
yufeih merged 2 commits into
dotnet:v3from
yufeih:commit-history-warmup
Jun 29, 2020
Merged

Warm up git commit history in parallel#6130
yufeih merged 2 commits into
dotnet:v3from
yufeih:commit-history-warmup

Conversation

@yufeih
Copy link
Copy Markdown
Contributor

@yufeih yufeih commented Jun 23, 2020

AB#240717

LoadCommitIds method is currently a blocking method. When it executes, the CPU waits for its completion without doing other works. This is a waste of CPU cycles. This PR parallelizes the execution of LoadCommitIds with other pre processors, to better utilize CPU time.

Reduces azure-docs-pr build time to < 50s.

Before:

image

After:

image

Microsoft Reviewers: Open in CodeFlow

@yufeih yufeih added the v3 label Jun 23, 2020
@yufeih yufeih force-pushed the commit-history-warmup branch from 9a11d5e to 10fd8f0 Compare June 23, 2020 12:50
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 23, 2020

Codecov Report

Merging #6130 into v3 will increase coverage by 0.23%.
The diff coverage is 89.79%.

Impacted file tree graph

@@            Coverage Diff             @@
##               v3    #6130      +/-   ##
==========================================
+ Coverage   91.97%   92.21%   +0.23%     
==========================================
  Files         285      285              
  Lines       11779    11800      +21     
==========================================
+ Hits        10834    10881      +47     
+ Misses        945      919      -26     
Impacted Files Coverage Δ
src/docfx/lib/git/FileCommitProvider.cs 92.57% <88.63%> (+0.90%) ⬆️
src/docfx/build/context/Context.cs 100.00% <100.00%> (ø)
src/docfx/lib/git/RepositoryProvider.cs 97.05% <100.00%> (+0.18%) ⬆️
src/docfx/lib/git/GitCommitCache.cs 62.50% <0.00%> (-31.25%) ⬇️
src/docfx/lib/process/ProcessUtility.cs 84.00% <0.00%> (-10.00%) ⬇️
src/docfx/lib/schema/JsonSchemaValidator.cs 99.65% <0.00%> (+0.34%) ⬆️
src/docfx/Errors.cs 88.77% <0.00%> (+1.02%) ⬆️
src/docfx/config/Config.cs 82.43% <0.00%> (+1.35%) ⬆️
src/docfx/lib/moniker/MonikerList.cs 90.32% <0.00%> (+1.61%) ⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5593cb0...f71b686. Read the comment docs.

@ghost
Copy link
Copy Markdown

ghost commented Jun 23, 2020

azure-docs-pr(00:00:47.5302374, 0 more diff)
diff --git "a/D:/docfx.TestData\\regression-test.azure-docs-pr\\output/.links.json" "b/D:/docfx.TestData\\regression-test.azure-docs-pr\\.temp/.links.json"
index c965613..a01050b 100644
--- "a/D:/docfx.TestData\\regression-test.azure-docs-pr\\output/.links.json"
+++ "b/D:/docfx.TestData\\regression-test.azure-docs-pr\\.temp/.links.json"
@@ -1563397,7 +1563397,7 @@
       "target_url": "/azure/remote-rendering/resources/troubleshoot"
     },
     {
-      "source_git_url": "https://github.com/MicrosoftDocs/azure-docs-pr/blob/master/articles/remote-rendering/samples/toc.yml",
+      "source_git_url": "https://github.com/MicrosoftDocs/azure-docs-pr/blob/master/articles/remote-rendering/resources/tools/toc.yml",
       "source_line": 4,
       "source_url": "/azure/remote-rendering/toc.json",
       "target_url": "/azure/remote-rendering/samples/azure-remote-rendering-asset-tool"

Comment thread src/docfx/build/context/Context.cs Outdated
@yufeih yufeih force-pushed the commit-history-warmup branch from 9c3edab to e8b1054 Compare June 29, 2020 04:16
@ghost
Copy link
Copy Markdown

ghost commented Jun 29, 2020

roslyn-api-docs(00:00:51.1819060, 7182361 more diff)
diff --git "a/D:/docfx.TestData\\regression-test.roslyn-api-docs\\output/dotnet/.dependencymap.json" "b/D:/docfx.TestData\\regression-test.roslyn-api-docs\\.temp/dotnet/.dependencymap.json"
index 30c4595..b4716ec 100644
--- "a/D:/docfx.TestData\\regression-test.roslyn-api-docs\\output/dotnet/.dependencymap.json"
+++ "b/D:/docfx.TestData\\regression-test.roslyn-api-docs\\.temp/dotnet/.dependencymap.json"
@@ -4959,20 +4959,12 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Formatting/BinaryOperatorSpacingOptions.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Formatting/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/ns-Microsoft.CodeAnalysis.CSharp.Formatting.xml",
         "type": "uid"
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Formatting/CSharpFormattingOptions.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Formatting/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Formatting/BinaryOperatorSpacingOptions.xml",
         "type": "uid"
@@ -4991,10 +4983,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Formatting/LabelPositionOptions.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Formatting/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/ns-Microsoft.CodeAnalysis.CSharp.Formatting.xml",
         "type": "uid"
@@ -5071,10 +5059,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AccessorListSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AccessorDeclarationSyntax.xml",
         "type": "uid"
@@ -5125,10 +5109,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AliasQualifiedNameSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ExpressionSyntax.xml",
         "type": "uid"
@@ -5327,10 +5307,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AnonymousObjectCreationExpressionSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AnonymousObjectMemberDeclaratorSyntax.xml",
         "type": "uid"
@@ -5385,10 +5361,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AnonymousObjectMemberDeclaratorSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ExpressionSyntax.xml",
         "type": "uid"
@@ -5435,10 +5407,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArgumentListSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArgumentSyntax.xml",
         "type": "uid"
@@ -5493,10 +5461,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArgumentSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ExpressionSyntax.xml",
         "type": "uid"
@@ -5547,10 +5511,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArrayCreationExpressionSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArrayRankSpecifierSyntax.xml",
         "type": "uid"
@@ -5609,10 +5569,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArrayRankSpecifierSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ExpressionSyntax.xml",
         "type": "uid"
@@ -5663,10 +5619,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArrayTypeSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArrayRankSpecifierSyntax.xml",
         "type": "uid"
@@ -5721,10 +5673,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ArrowExpressionClauseSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ExpressionSyntax.xml",
         "type": "uid"
@@ -5771,10 +5719,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AssignmentExpressionSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ExpressionSyntax.xml",
         "type": "uid"
@@ -5821,10 +5765,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AttributeArgumentListSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AttributeArgumentSyntax.xml",
         "type": "uid"
@@ -5875,10 +5815,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AttributeArgumentSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/ExpressionSyntax.xml",
         "type": "uid"
@@ -5929,10 +5865,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AttributeListSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AttributeSyntax.xml",
         "type": "uid"
@@ -5987,10 +5919,6 @@
       }
     ],
     "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AttributeSyntax.xml": [
-      {
-        "source": "api/_splitted/Microsoft.CodeAnalysis.CSharp.Syntax/TOC.yml",
-        "type": "metadata"
-      },
       {
         "source": "xml/Microsoft.CodeAnalysis.CSharp.Syntax/AttributeArgumentListSyntax.xml",

@yufeih yufeih merged commit 19ab0dc into dotnet:v3 Jun 29, 2020
@yufeih yufeih deleted the commit-history-warmup branch June 29, 2020 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants