Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1165139	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120379/merge to s	Log	dotnet/runtime#120379
1164469	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51117/merge to s	Log	dotnet/sdk#51117
1162942	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51087/merge to s	Log	dotnet/sdk#51087
1161053	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63851/merge to s	Log	dotnet/aspnetcore#63851
1161054	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63851/merge to s	Log	dotnet/aspnetcore#63851
1159179	dotnet/installer	Checkout dotnet/installer@refs/pull/20623/merge to s	Log	dotnet/installer#20623
2802312	dotnet-installer	Clone dotnet/installer	Log	#53816
2802291	dotnet-sdk	Checkout dotnet-sdk@refs/pull/53814/merge to s	Log	#53814
1158406	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120111/merge to s	Log	dotnet/runtime#120111
1158388	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120110/merge to s	Log	dotnet/runtime#120110
1158375	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120107/merge to s	Log	dotnet/runtime#120107
2801336	dotnet-installer	Clone dotnet/installer	Log	#53739
1157737	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50998/merge to s	Log	dotnet/sdk#50998
1157693	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118941/merge to s	Log	dotnet/runtime#118941
1157730	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119374/merge to s	Log	dotnet/runtime#119374
1157716	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116109/merge to s	Log	dotnet/runtime#116109
1157711	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120088/merge to s	Log	dotnet/runtime#120088
1157662	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120086/merge to s	Log	dotnet/runtime#120086
1157675	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120087/merge to s	Log	dotnet/runtime#120087
1157663	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120086/merge to s	Log	dotnet/runtime#120086
1157586	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120082/merge to s	Log	dotnet/runtime#120082
1157576	dotnet/installer	Checkout dotnet/installer@refs/pull/20621/merge to s	Log	dotnet/installer#20621
1157564	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117168/merge to s	Log	dotnet/runtime#117168
1157562	dotnet/runtime	Checkout dotnet/runtime@refs/pull/117168/merge to s	Log	dotnet/runtime#117168
1157561	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1157560	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50901/merge to s	Log	dotnet/sdk#50901
1157555	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119998/merge to s	Log	dotnet/runtime#119998
1157534	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1157551	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120077/merge to s	Log	dotnet/runtime#120077
1157536	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1080/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1080
1157514	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119360/merge to s	Log	dotnet/runtime#119360
1157488	dotnet/runtime	Checkout dotnet/runtime@release/9.0-staging to s	Log
1157520	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1157207	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63800/merge to s	Log	dotnet/aspnetcore#63800
1157209	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63800/merge to s	Log	dotnet/aspnetcore#63800
1157192	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50984/merge to s	Log	dotnet/sdk#50984
1157105	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119805/merge to s	Log	dotnet/runtime#119805
1157116	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119959/merge to s	Log	dotnet/runtime#119959
1156268	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16160/merge to s	Log	#16160
1156267	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16159/merge to s	Log	#16159
1155373	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63748/merge to s	Log	dotnet/aspnetcore#63748
1155374	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63748/merge to s	Log	dotnet/aspnetcore#63748
1154887	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119951/merge to s	Log	dotnet/runtime#119951
1153634	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119197/merge to s	Log	dotnet/runtime#119197
1153591	dotnet/aspnetcore	Checkout dotnet/aspnetcore@release/10.0 to s	Log
1152799	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63641/merge to s	Log	dotnet/aspnetcore#63641
1152798	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63641/merge to s	Log	dotnet/aspnetcore#63641
1152800	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63641/merge to s	Log	dotnet/aspnetcore#63641
2797089	dotnet-sdk	Checkout dotnet-sdk@refs/pull/53502/merge to s	Log	#53502
1152396	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119867/merge to s	Log	dotnet/runtime#119867
1151959	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1077/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1077
1151377	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1076/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1076
1151312	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1075/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1075
2795917	dotnet-sdk	Checkout dotnet-sdk@refs/pull/53437/merge to s	Log	#53437
1149632	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119754/merge to s	Log	dotnet/runtime#119754
1149363	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50861/merge to s	Log	dotnet/sdk#50861
1149364	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50858/merge to s	Log	dotnet/sdk#50858
1147252	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1074/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1074
1146001	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50758/merge to s	Log	dotnet/sdk#50758
1146280	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63504/merge to s	Log	dotnet/aspnetcore#63504
1146281	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63504/merge to s	Log	dotnet/aspnetcore#63504
1145394	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118541/merge to s	Log	dotnet/runtime#118541
1145384	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118536/merge to s	Log	dotnet/runtime#118536
1145390	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118539/merge to s	Log	dotnet/runtime#118539
1145388	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118538/merge to s	Log	dotnet/runtime#118538
1144707	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119338/merge to s	Log	dotnet/runtime#119338
1144585	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63624/merge to s	Log	dotnet/aspnetcore#63624
1144586	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63624/merge to s	Log	dotnet/aspnetcore#63624
1144148	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50729/merge to s	Log	dotnet/sdk#50729
1143322	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63601/merge to s	Log	dotnet/aspnetcore#63601
1143290	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63600/merge to s	Log	dotnet/aspnetcore#63600
1142662	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63592/merge to s	Log	dotnet/aspnetcore#63592
1142478	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/63589/merge to s	Log	dotnet/aspnetcore#63589

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	5	73

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?