Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checkout failure: "Git fetch failed with exit code 128" #9009

Open
2 tasks
carlossanlop opened this issue Apr 8, 2022 · 14 comments
Open
2 tasks

Checkout failure: "Git fetch failed with exit code 128" #9009

carlossanlop opened this issue Apr 8, 2022 · 14 comments
Labels
Known Build Error Ops - Facilely Operations issues that are easily accomplished or attained Ops - P2 Operations task, priority 2

Comments

@carlossanlop
Copy link
Member

carlossanlop commented Apr 8, 2022

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

  • Open the failing build above and investigate
  • Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

  1. Add the label "Known Build Error"
  2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation
{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build Definition Step Name Console log Pull Request
748869 dotnet/runtime Checkout dotnet/runtime@refs/pull/105182/merge to s Log dotnet/runtime#105182
2498629 dotnet-performance Checkout dotnet-performance@main to s Log
748328 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/606/merge to s Log maestro-auth-test/helix-queue-insights-test#606
747858 dotnet/runtime Checkout dotnet/runtime@refs/pull/105149/merge to s Log dotnet/runtime#105149
747582 dotnet/runtime Checkout dotnet/runtime@refs/pull/105140/merge to s Log dotnet/runtime#105140
747249 dotnet/runtime Checkout dotnet/runtime@refs/pull/105128/merge to s Log dotnet/runtime#105128
2497789 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/41196/merge to s Log #41196
2497788 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/41195/merge to s Log #41195
745860 dotnet/runtime Checkout dotnet/runtime@refs/pull/104503/merge to s dotnet/runtime#104503
745739 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/604/merge to s Log maestro-auth-test/helix-queue-insights-test#604
745626 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1364/merge to s Log maestro-auth-test/build-result-analysis-test#1364
745627 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1364/merge to s Log maestro-auth-test/build-result-analysis-test#1364
745550 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/600/merge to s Log maestro-auth-test/helix-queue-insights-test#600
745538 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/599/merge to s Log maestro-auth-test/helix-queue-insights-test#599
745531 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/598/merge to s Log maestro-auth-test/helix-queue-insights-test#598
745482 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1354/merge to s Log maestro-auth-test/build-result-analysis-test#1354
745481 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1354/merge to s Log maestro-auth-test/build-result-analysis-test#1354
745444 maestro-auth-test/build-result-analysis-test Checkout maestro-auth-test/build-result-analysis-test@refs/pull/1352/merge to s Log maestro-auth-test/build-result-analysis-test#1352
745089 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/597/merge to s Log maestro-auth-test/helix-queue-insights-test#597
744786 dotnet/runtime Checkout dotnet/runtime@refs/pull/104964/merge to s Log dotnet/runtime#104964
744131 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/595/merge to s Log maestro-auth-test/helix-queue-insights-test#595
744100 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/593/merge to s Log maestro-auth-test/helix-queue-insights-test#593
744049 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/592/merge to s Log maestro-auth-test/helix-queue-insights-test#592
2496390 dotnet-aspnetcore Checkout dotnet-aspnetcore@refs/pull/41107/merge to s Log #41107
743818 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/591/merge to s Log maestro-auth-test/helix-queue-insights-test#591
742643 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/590/merge to s Log maestro-auth-test/helix-queue-insights-test#590
742097 dotnet/runtime Checkout dotnet/runtime@refs/pull/103597/merge to s Log dotnet/runtime#103597
742043 dotnet/runtime Checkout dotnet/runtime@refs/pull/104395/merge to s Log dotnet/runtime#104395
741788 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/587/merge to s Log maestro-auth-test/helix-queue-insights-test#587
741311 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56798/merge to s Log dotnet/aspnetcore#56798
741313 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56798/merge to s Log dotnet/aspnetcore#56798
740863 dotnet/sdk Checkout dotnet/sdk@refs/pull/42148/merge to s Log dotnet/sdk#42148
740808 dotnet/runtime Checkout dotnet/runtime@refs/pull/104437/merge to s Log dotnet/runtime#104437
739787 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/585/merge to s Log maestro-auth-test/helix-queue-insights-test#585
739113 dotnet/arcade Checkout dotnet/arcade@refs/pull/14933/merge to s Log #14933
2493509 dotnet-runtime Checkout dotnet-runtime@release/6.0-staging to s Log
738721 dotnet/sdk Checkout dotnet/sdk@refs/pull/42114/merge to s Log dotnet/sdk#42114
738447 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/583/merge to s Log maestro-auth-test/helix-queue-insights-test#583
738408 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/582/merge to s Log maestro-auth-test/helix-queue-insights-test#582
738324 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/581/merge to s Log maestro-auth-test/helix-queue-insights-test#581
738237 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/580/merge to s Log maestro-auth-test/helix-queue-insights-test#580
737319 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56743/merge to s Log dotnet/aspnetcore#56743
737317 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56743/merge to s Log dotnet/aspnetcore#56743
737318 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56743/merge to s Log dotnet/aspnetcore#56743
737222 dotnet/runtime Checkout dotnet/runtime@refs/pull/103985/merge to s Log dotnet/runtime#103985
737123 dotnet/sdk Checkout dotnet/sdk@refs/pull/42095/merge to s Log dotnet/sdk#42095
735259 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/576/merge to s Log maestro-auth-test/helix-queue-insights-test#576
734843 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#42047
734491 dotnet/runtime Checkout dotnet/runtime@refs/pull/104597/merge to s Log dotnet/runtime#104597
734337 dotnet/runtime Checkout dotnet/runtime@refs/pull/104226/merge to s Log dotnet/runtime#104226
733879 dotnet/runtime Checkout dotnet/runtime@refs/pull/104581/merge to s Log dotnet/runtime#104581
733855 dotnet/runtime Checkout dotnet/runtime@refs/pull/101631/merge to s Log dotnet/runtime#101631
733653 dotnet/performance Checkout dotnet/performance@refs/pull/4300/merge to s Log dotnet/performance#4300
2489943 dotnet-performance Checkout dotnet-performance@main to s Log
731531 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/574/merge to s Log maestro-auth-test/helix-queue-insights-test#574
731378 dotnet/runtime Checkout dotnet/runtime@refs/pull/104471/merge to s Log dotnet/runtime#104471
731004 dotnet/runtime Checkout dotnet/runtime@refs/pull/104452/merge to s Log dotnet/runtime#104452
730522 dotnet/sdk Checkout dotnet/sdk@refs/pull/41979/merge to s Log dotnet/sdk#41979
730521 dotnet/sdk Checkout dotnet/sdk@refs/pull/41977/merge to s Log dotnet/sdk#41977
730458 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41966
730460 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41966
728860 dotnet/installer Checkout dotnet/installer@refs/pull/19924/merge to s Log dotnet/installer#19924
728522 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/572/merge to s Log maestro-auth-test/helix-queue-insights-test#572
728001 dotnet/performance Checkout dotnet/performance@refs/pull/4295/merge to s Log dotnet/performance#4295
727857 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/571/merge to s Log maestro-auth-test/helix-queue-insights-test#571
726722 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/568/merge to s Log maestro-auth-test/helix-queue-insights-test#568
726721 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/567/merge to s Log maestro-auth-test/helix-queue-insights-test#567
726578 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/566/merge to s Log maestro-auth-test/helix-queue-insights-test#566
724419 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41855
2484471 dotnet-arcade Checkout dotnet-arcade@main to s Log
723795 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/565/merge to s Log maestro-auth-test/helix-queue-insights-test#565
723590 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/564/merge to s Log maestro-auth-test/helix-queue-insights-test#564
722639 dotnet/sdk Checkout dotnet/sdk@refs/pull/41824/merge to s Log dotnet/sdk#41824
722324 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/563/merge to s Log maestro-auth-test/helix-queue-insights-test#563
721955 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/562/merge to s Log maestro-auth-test/helix-queue-insights-test#562
721609 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/560/merge to s Log maestro-auth-test/helix-queue-insights-test#560
720983 dotnet/runtime Checkout dotnet/runtime@refs/pull/104025/merge to s Log dotnet/runtime#104025
720452 dotnet/runtime Checkout dotnet/runtime@refs/pull/103948/merge to s Log dotnet/runtime#103948
720104 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/557/merge to s Log maestro-auth-test/helix-queue-insights-test#557
719860 dotnet/sdk Checkout dotnet/sdk@refs/pull/41789/merge to s Log dotnet/sdk#41789
719670 dotnet/sdk Checkout dotnet/sdk@refs/pull/41742/merge to s Log dotnet/sdk#41742
719861 dotnet/sdk Clone dotnet/sdk Log dotnet/sdk#41789
719359 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/556/merge to s Log maestro-auth-test/helix-queue-insights-test#556
719278 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/555/merge to s Log maestro-auth-test/helix-queue-insights-test#555
719224 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/554/merge to s Log maestro-auth-test/helix-queue-insights-test#554
719126 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/552/merge to s Log maestro-auth-test/helix-queue-insights-test#552
719058 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/551/merge to s Log maestro-auth-test/helix-queue-insights-test#551
718993 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/550/merge to s Log maestro-auth-test/helix-queue-insights-test#550
718842 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/548/merge to s Log maestro-auth-test/helix-queue-insights-test#548
718638 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/546/merge to s Log maestro-auth-test/helix-queue-insights-test#546
718322 dotnet/runtime Checkout dotnet/runtime@refs/pull/103820/merge to s Log dotnet/runtime#103820
716424 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/545/merge to s Log maestro-auth-test/helix-queue-insights-test#545
716090 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/542/merge to s Log maestro-auth-test/helix-queue-insights-test#542
715172 maestro-auth-test/helix-queue-insights-test Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/541/merge to s Log maestro-auth-test/helix-queue-insights-test#541
715019 dotnet/aspnetcore Checkout dotnet/aspnetcore@refs/pull/56352/merge to s Log dotnet/aspnetcore#56352

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
2 33 95

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

@ChadNedzlek
Copy link
Member

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

@ulisesh
Copy link
Contributor

ulisesh commented Apr 13, 2022

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

@markwilkie
Copy link
Member

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

@ulisesh
Copy link
Contributor

ulisesh commented Apr 13, 2022

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

@ilyas1974 ilyas1974 self-assigned this Apr 19, 2022
@ilyas1974 ilyas1974 removed their assignment Jun 22, 2022
@ilyas1974 ilyas1974 removed Ops - First Responder Detected By - Customer Issue was reported by a customer labels Jun 22, 2022
@danmoseley danmoseley changed the title Checkout dotnet/runtime step failure: "Git fetch failed with exit code 128" Checkout failure: "Git fetch failed with exit code 128" Oct 7, 2022
@ViktorHofer
Copy link
Member

ViktorHofer commented Jan 11, 2023

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?

@premun
Copy link
Member

premun commented Jan 11, 2023

GitHub reported degraded git operations yesterday: https://www.githubstatus.com/incidents/00km0kwpz23b
We saw problems cloning all over. Maybe it's related to that?

@dougbu
Copy link
Member

dougbu commented Jan 8, 2024

This issue may have started in early 2022 but seems to remain w/ 29 matching failures in the last 24 hours. https://www.githubstatus.com/ and various Azure / Azure DevOps status pages aren't showing any current or recent degradations. Unclear what's going on this time beyond what appears to be DNS flakiness. I looked at the last 7 reports; all but one showed fatal: unable to access 'https://github.com/dotnet/runtime/': Could not resolve host: github.com and a few also showed fatal: unable to access 'https://github.com/dotnet/runtime/': Failed to connect to github.com port 443 after 75012 ms: Couldn't connect to server.

Without more information, I think the best we could do is auto retry when the problem occurs. @AlitzelMendez and @carlossanlop what are your thoughts on including "BuildRetry": true in the JSON content in this description❔

@carlossanlop
Copy link
Member Author

It's fine, we can try that, as long as we monitor it closely, because if we keep seeing the failure even with a retry, I think we should just go back to "BuildRetry": false, to avoid wasting resources.

@dougbu
Copy link
Member

dougbu commented Jan 8, 2024

you mean "BuildRetry": true was tried in the past and didn't help much @carlossanlop❔ if the extra time it takes PR creators to click the Rerun failed builds actually helps avoid issues in the retry, I'd lean toward leaving things as-is.

@carlossanlop
Copy link
Member Author

I do not know if it was tried in the past.

@AlitzelMendez
Copy link
Member

sounds like a good plan @dougbu, let's try "BuildRetry": true, I will put together a query to monitor if it is helpful for this issue

@dougbu
Copy link
Member

dougbu commented Jan 8, 2024

updated the description to try retries

@AlitzelMendez
Copy link
Member

query:

let buildIds =
KnownIssues
| where ingestion_time() > datetime(2024-01-08 23:10:44.8683418)
| where IssueId  == 9009
| distinct BuildId;
TimelineBuilds
| where BuildId  in (buildIds)
| summarize arg_max(FinishTime, *) by BuildId
| summarize count() by Result, bin(FinishTime, 1d)
| order by FinishTime asc 
| render columnchart with (xcolumn=FinishTime)

I will check this on Monday to see how it is going, posting here the query so I don't loose it :)

@AlitzelMendez
Copy link
Member

the success rate (having a green build) after the retry was between 30% to 50% per day, so this doesn't solve the issue, it just helps to mitigate it a little bit.

@AlitzelMendez AlitzelMendez removed their assignment Jan 16, 2024
@ilyas1974 ilyas1974 added the Ops - Facilely Operations issues that are easily accomplished or attained label Feb 21, 2024
@garath garath added the Ops - P2 Operations task, priority 2 label Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Known Build Error Ops - Facilely Operations issues that are easily accomplished or attained Ops - P2 Operations task, priority 2
Projects
None yet
Development

No branches or pull requests

10 participants