Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checkout failure: "Git fetch failed with exit code 128" #9009

Open
2 tasks
carlossanlop opened this issue Apr 8, 2022 · 14 comments
Open
2 tasks

Checkout failure: "Git fetch failed with exit code 128" #9009

carlossanlop opened this issue Apr 8, 2022 · 14 comments
Labels
Known Build Error Ops - Facilely Operations issues that are easily accomplished or attained Ops - P2 Operations task, priority 2

Comments

@carlossanlop
Copy link
Member

carlossanlop commented Apr 8, 2022

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

  • Open the failing build above and investigate
  • Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

  1. Add the label "Known Build Error"
  2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation
{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build Definition Step Name Console log Pull Request
728522 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/572/merge to s Log maestro-auth-test/helix-queue-insights-test#572
728001 dotnet/performance Checkout dotnet/performance@refs/pull/4295/merge to s Log dotnet/performance#4295
727857 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/571/merge to s Log maestro-auth-test/helix-queue-insights-test#571
726722 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/568/merge to s Log maestro-auth-test/helix-queue-insights-test#568
726721 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/567/merge to s Log maestro-auth-test/helix-queue-insights-test#567
726578 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/566/merge to s Log maestro-auth-test/helix-queue-insights-test#566
724419 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41855
2484471 dotnet-arcade Checkout dotnet-arcade@main to s Log
723795 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/565/merge to s Log maestro-auth-test/helix-queue-insights-test#565
723590 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/564/merge to s Log maestro-auth-test/helix-queue-insights-test#564
722639 dotnet/sdk Checkout dotnet/sdk@refs/pull/41824/merge to s Log dotnet/sdk#41824
722324 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/563/merge to s Log maestro-auth-test/helix-queue-insights-test#563
721955 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/562/merge to s Log maestro-auth-test/helix-queue-insights-test#562
721609 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/560/merge to s Log maestro-auth-test/helix-queue-insights-test#560
720983 dotnet/runtime Checkout dotnet/runtime@refs/pull/104025/merge to s Log dotnet/runtime#104025
720452 dotnet/runtime Checkout dotnet/runtime@refs/pull/103948/merge to s Log dotnet/runtime#103948
720104 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/557/merge to s Log maestro-auth-test/helix-queue-insights-test#557
719860 dotnet/sdk Checkout dotnet/sdk@refs/pull/41789/merge to s Log dotnet/sdk#41789
719670 dotnet/sdk Checkout dotnet/sdk@refs/pull/41742/merge to s Log dotnet/sdk#41742
719861 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41789
719359 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/556/merge to s Log maestro-auth-test/helix-queue-insights-test#556
719278 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/555/merge to s Log maestro-auth-test/helix-queue-insights-test#555
719224 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/554/merge to s Log maestro-auth-test/helix-queue-insights-test#554
719126 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/552/merge to s Log maestro-auth-test/helix-queue-insights-test#552
719058 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/551/merge to s Log maestro-auth-test/helix-queue-insights-test#551
718993 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/550/merge to s Log maestro-auth-test/helix-queue-insights-test#550
718842 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/548/merge to s Log maestro-auth-test/helix-queue-insights-test#548
718638 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/546/merge to s Log maestro-auth-test/helix-queue-insights-test#546
718322 dotnet/runtime Checkout dotnet/runtime@refs/pull/103820/merge to s Log dotnet/runtime#103820
716424 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/545/merge to s Log maestro-auth-test/helix-queue-insights-test#545
716090 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/542/merge to s Log maestro-auth-test/helix-queue-insights-test#542
715172 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/541/merge to s Log maestro-auth-test/helix-queue-insights-test#541
715019 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56352/merge to s Log dotnet/aspnetcore#56352
714365 dotnet/sdk Checkout dotnet/sdk@refs/pull/41616/merge to s Log dotnet/sdk#41616
714439 dotnet/runtime Checkout dotnet/runtime@refs/pull/103772/merge to s Log dotnet/runtime#103772
713323 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/538/merge to s Log maestro-auth-test/helix-queue-insights-test#538
713258 dotnet/roslyn-sdk Checkout dotnet/roslyn-sdk@refs/pull/1164/merge to s Log dotnet/roslyn-sdk#1164
713223 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/536/merge to s Log maestro-auth-test/helix-queue-insights-test#536
712701 dotnet/arcade Checkout dotnet/arcade@refs/pull/14863/merge to s Log #14863
712672 dotnet/runtime Checkout dotnet/runtime@refs/pull/103695/merge to s Log dotnet/runtime#103695
712322 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/535/merge to s Log maestro-auth-test/helix-queue-insights-test#535
712006 dotnet/runtime Checkout dotnet/runtime@refs/pull/103275/merge to s Log dotnet/runtime#103275
712044 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/534/merge to s Log maestro-auth-test/helix-queue-insights-test#534
711771 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56297/merge to s Log dotnet/aspnetcore#56297
711773 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56297/merge to s Log dotnet/aspnetcore#56297
2476565 dotnet-runtime Checkout dotnet-runtime@main to s Log
711282 dotnet/installer Checkout dotnet/installer@refs/pull/19915/merge to s Log dotnet/installer#19915
2476349 dotnet-runtime Checkout dotnet-runtime@main to s Log
2474420 dotnet-sdk Checkout dotnet-sdk@refs/pull/40485/merge to s Log #40485
2474448 dotnet-sdk Checkout dotnet-sdk@refs/pull/40469/merge to s Log #40469
709654 dotnet/runtime Checkout dotnet/runtime@refs/pull/103546/merge to s Log dotnet/runtime#103546
2474895 dotnet-roslyn Checkout dotnet-roslyn@main to s Log
2474382 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/40480/merge to s Log #40480
2473607 dotnet-winforms Checkout dotnet-winforms@refs/pull/40441/merge to s Log #40441
707793 dotnet/runtime Checkout dotnet/runtime@refs/pull/101796/merge to s Log dotnet/runtime#101796
2473876 dotnet-efcore Checkout dotnet-efcore@main to s Log
2473748 dotnet-sdk Checkout dotnet-sdk@main to s Log
2473686 dotnet-installer Checkout dotnet-installer@refs/pull/40443/merge to s Log #40443
2473416 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/40410/merge to s Log #40410
706315 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41568
706318 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41568
706317 dotnet/sdk Checkout dotnet/sdk@refs/pull/41568/merge to s Log dotnet/sdk#41568
706143 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/533/merge to s Log maestro-auth-test/helix-queue-insights-test#533
2472531 dotnet-performance Checkout dotnet-performance@main to s Log
705964 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/532/merge to s Log maestro-auth-test/helix-queue-insights-test#532
705914 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/531/merge to s Log maestro-auth-test/helix-queue-insights-test#531
705724 dotnet/runtime Checkout dotnet/runtime@main to s
705073 dotnet/runtime Checkout dotnet/runtime@refs/pull/103337/merge to s Log dotnet/runtime#103337
704949 dotnet/runtime Checkout dotnet/runtime@refs/pull/103322/merge to s Log dotnet/runtime#103322
2471471 dotnet-roslyn Checkout dotnet-roslyn@release/dev17.8 to s Log
704107 dotnet/runtime Checkout dotnet/runtime@refs/pull/103299/merge to s Log dotnet/runtime#103299
703815 dotnet/runtime Checkout dotnet/runtime@refs/pull/103283/merge to s Log dotnet/runtime#103283
703246 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41500
703248 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41500
703317 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/526/merge to s Log maestro-auth-test/helix-queue-insights-test#526
703247 dotnet/sdk Checkout dotnet/sdk@refs/pull/41500/merge to s Log dotnet/sdk#41500
702839 dotnet/runtime Checkout dotnet/runtime@refs/pull/102506/merge to s Log dotnet/runtime#102506
702660 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41492
702663 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41492
702661 dotnet/sdk Checkout dotnet/sdk@refs/pull/41492/merge to s Log dotnet/sdk#41492
699983 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/523/merge to s Log maestro-auth-test/helix-queue-insights-test#523
699881 dotnet/runtime Checkout dotnet/runtime@refs/pull/103139/merge to s Log dotnet/runtime#103139
699838 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/522/merge to s Log maestro-auth-test/helix-queue-insights-test#522
699802 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/521/merge to s Log maestro-auth-test/helix-queue-insights-test#521
699680 dotnet/runtime Checkout dotnet/runtime@refs/pull/103133/merge to s Log dotnet/runtime#103133
699022 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/520/merge to s Log maestro-auth-test/helix-queue-insights-test#520
698985 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/519/merge to s Log maestro-auth-test/helix-queue-insights-test#519
698806 dotnet/sdk Checkout dotnet/sdk@refs/pull/41421/merge to s Log dotnet/sdk#41421
698805 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41421
698804 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41421
698807 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41421
698510 dotnet/performance Checkout dotnet/performance@refs/pull/4252/merge to s Log dotnet/performance#4252
697672 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56078/merge to s Log dotnet/aspnetcore#56078
697675 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56078/merge to s Log dotnet/aspnetcore#56078
696681 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/516/merge to s Log maestro-auth-test/helix-queue-insights-test#516
696619 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/515/merge to s Log maestro-auth-test/helix-queue-insights-test#515
696599 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/514/merge to s Log maestro-auth-test/helix-queue-insights-test#514
696193 dotnet/runtime Checkout dotnet/runtime@refs/pull/103005/merge to s Log dotnet/runtime#103005
696234 dotnet/arcade-services Checkout dotnet/arcade-services@refs/pull/3582/merge to s Log dotnet/arcade-services#3582

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
3 15 99

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

@ChadNedzlek
Copy link
Member

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

@ulisesh
Copy link
Contributor

ulisesh commented Apr 13, 2022

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

@markwilkie
Copy link
Member

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

@ulisesh
Copy link
Contributor

ulisesh commented Apr 13, 2022

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

@ilyas1974 ilyas1974 self-assigned this Apr 19, 2022
@ilyas1974 ilyas1974 removed their assignment Jun 22, 2022
@ilyas1974 ilyas1974 removed Ops - First Responder Detected By - Customer Issue was reported by a customer labels Jun 22, 2022
@danmoseley danmoseley changed the title Checkout dotnet/runtime step failure: "Git fetch failed with exit code 128" Checkout failure: "Git fetch failed with exit code 128" Oct 7, 2022
@ViktorHofer
Copy link
Member

ViktorHofer commented Jan 11, 2023

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?

@premun
Copy link
Member

premun commented Jan 11, 2023

GitHub reported degraded git operations yesterday: https://www.githubstatus.com/incidents/00km0kwpz23b
We saw problems cloning all over. Maybe it's related to that?

@dougbu
Copy link
Member

dougbu commented Jan 8, 2024

This issue may have started in early 2022 but seems to remain w/ 29 matching failures in the last 24 hours. https://www.githubstatus.com/ and various Azure / Azure DevOps status pages aren't showing any current or recent degradations. Unclear what's going on this time beyond what appears to be DNS flakiness. I looked at the last 7 reports; all but one showed fatal: unable to access 'https://github.com/dotnet/runtime/': Could not resolve host: github.com and a few also showed fatal: unable to access 'https://github.com/dotnet/runtime/': Failed to connect to github.com port 443 after 75012 ms: Couldn't connect to server.

Without more information, I think the best we could do is auto retry when the problem occurs. @AlitzelMendez and @carlossanlop what are your thoughts on including "BuildRetry": true in the JSON content in this description❔

@carlossanlop
Copy link
Member Author

It's fine, we can try that, as long as we monitor it closely, because if we keep seeing the failure even with a retry, I think we should just go back to "BuildRetry": false, to avoid wasting resources.

@dougbu
Copy link
Member

dougbu commented Jan 8, 2024

you mean "BuildRetry": true was tried in the past and didn't help much @carlossanlop❔ if the extra time it takes PR creators to click the Rerun failed builds actually helps avoid issues in the retry, I'd lean toward leaving things as-is.

@carlossanlop
Copy link
Member Author

I do not know if it was tried in the past.

@AlitzelMendez
Copy link
Member

sounds like a good plan @dougbu, let's try "BuildRetry": true, I will put together a query to monitor if it is helpful for this issue

@dougbu
Copy link
Member

dougbu commented Jan 8, 2024

updated the description to try retries

@AlitzelMendez
Copy link
Member

query:

let buildIds =
KnownIssues
| where ingestion_time() > datetime(2024-01-08 23:10:44.8683418)
| where IssueId  == 9009
| distinct BuildId;
TimelineBuilds
| where BuildId  in (buildIds)
| summarize arg_max(FinishTime, *) by BuildId
| summarize count() by Result, bin(FinishTime, 1d)
| order by FinishTime asc 
| render columnchart with (xcolumn=FinishTime)

I will check this on Monday to see how it is going, posting here the query so I don't loose it :)

@AlitzelMendez
Copy link
Member

the success rate (having a green build) after the retry was between 30% to 50% per day, so this doesn't solve the issue, it just helps to mitigate it a little bit.

@AlitzelMendez AlitzelMendez removed their assignment Jan 16, 2024
@ilyas1974 ilyas1974 added the Ops - Facilely Operations issues that are easily accomplished or attained label Feb 21, 2024
@garath garath added the Ops - P2 Operations task, priority 2 label Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Known Build Error Ops - Facilely Operations issues that are easily accomplished or attained Ops - P2 Operations task, priority 2
Projects
None yet
Development

No branches or pull requests

10 participants