install-native-dependencies.sh failing in CI

[jkotas]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=95035

Build leg reported

Build / CoreCLR Product Build OSX x64 checked / Install native dependencies

Pull Request

dotnet/runtime#78922

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Error: Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core failed!",
   "BuildRetry": true
}

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Report

Build	Definition	Step Name	Console log	Pull Request
96256	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78354
96323	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78852
96320	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78852
96235	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78557
95054	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78924
95045	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78919
95063	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78852
95050	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78923
94991	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78918
95056	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78924
95035	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78922
95027	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78920
94987	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78557
94986	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78557
94940	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#77256
94939	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#77256
93713	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78849
93715	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78849
94910	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78828
94900	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78817
94682	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78888
94683	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78888

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	22	22

Report

Build	Definition	Step Name	Console log	Pull Request
96256	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78354
96323	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78852
96320	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78852
96235	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78557
95054	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78924
95045	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78919
95063	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78852
95050	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78923
94991	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78918
95056	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78924
95035	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78922
95027	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78920
94987	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78557
94986	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78557
94940	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#77256
94939	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#77256
93713	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78849
93715	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78849
94910	dotnet/runtime	Install native dependencies	Log	dotnet/runtime#78828
94900	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78817
94682	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78888
94683	dotnet/runtime	Install Build Dependencies	Log	dotnet/runtime#78888

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	22

[ulisesh]

@jkotas the error doesn't seem to be something that the engineering team owns. The issue should be opened in the runtime repo. Please let us know if you need help moving the issue

[jkotas]

I believe that the engineering team owns the installation of the native dependencies like python on the build machines. You can see number of issues about that in the arcade repo, e.g. #2673 .

The specific script where this is failing is dotnet/runtime specific because we never achieved a grand unified way to install the native dependencies. There is not much that the engineers working on dotnet/runtime can do about installation of the native dependencies on the build machines. The solution for it has to be driven from the engineering team - for both secure supply chain aspects and reliability aspects.

[ulisesh]

Fair points

I see that your build failed in a Hosted OSX agent (which we don't own) and it passed on retry. I'll keep the issue open for today but I don't see much we can investigate/do here. Does it sound fair?

[ulisesh]

I see some hits today, given that retrying the build helped yesterday. I'll turn on build retry for this known issue and keep it open for tracking

[markwilkie]

Adding @jonfortescue and @Chrisboh who are likely interested in this from the native dependency / secure supply chain perspective as a good case in point where runtime has unique challenges.

[jonfortescue]

Ah, yeah, macOS was out of scope for my effort because we don't produce our own macOS build images. I don't have strong opinions one way or the other with this -- it's a tricky situation.

[ulisesh]

@jkotas given that the script is failing to talk to a service we don't control, is it possible to add a retry to runtime's scripts?

[jkotas]

The script is talking to github.com. Notice that the first error is fatal: Could not read from remote repository.. This error is produced by git command.

What is our usual retry policy for git commands? I have looked around in dotnet/runtime and I have not found any retry loops around git commands.

[jonfortescue]

@jkotas there's not a specific example to my knowledge, but generally we just do an exponential retry loop.

[ilyas1974]

As we have not seen additional instances of this for the last week, I am closing this issue. Feel free to re-open it should you experience this again.