Intermittent failure to restore IBC optimization data

[runfoapp]

Runfo Tracking Issue: Intermittent failure to restore IBC optimization data

Definition	Build	Kind	Job Name
runtime	1097184	Rolling	Build Linux arm Release AllSubsets_Mono
runtime	1096348	PR 51200	Installer Build and Test coreclr Linux_musl_arm64 Debug
runtime	1095579	PR 51488	Libraries Build Linux_musl arm64 Release
runtime	1094759	PR 51439	Libraries Build Linux_musl arm Release
runtime	1094469	PR 51440	Mono Product Build Linux arm64 release

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
1	5	5

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Runfo Tracking Issue: Intermittent failure to restore IBC optimization data

Definition	Build	Kind	Job Name
runtime	1097184	Rolling	Build Linux arm Release AllSubsets_Mono
runtime	1096348	PR 51200	Installer Build and Test coreclr Linux_musl_arm64 Debug
runtime	1095579	PR 51488	Libraries Build Linux_musl arm64 Release
runtime	1094759	PR 51439	Libraries Build Linux_musl arm Release
runtime	1094469	PR 51440	Mono Product Build Linux arm64 release

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
1	5	5

Author:	runfoapp[bot]
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[jeffschwMSFT]

@agocke do we still have IBC in our pipeline or is the name of the stage just not been updated?

[hoyosjs]

@jeffschwMSFT IBC comes from arcade support, it's looking for these IBC data packages:

These are available at https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=dotnet6-transport&package=optimization.linux-x64.IBC.CoreCLR&protocolType=NuGet&version=99.99.99-master-20200806.6&view=overview. Not sure why they are not restored

[jeffschwMSFT]

Thanks! The root of my question is that since we are moving from the ibc format to mibc, I was curious if we were just reusing the name or if we still were flowing ibc data.

[hoyosjs]

These are old packages (newest is February), so there's no flow. As for if it's used, I am not sure if that part is still hooked.

[agocke]

@davidwrighton would know for sure, but I see the same packages coming out of dotnet-optimization with what looks like fresh mibc files in them.

[hoyosjs]

The mIBC packages are slightly different: optimization.windows_nt-x86.mibc.runtime. After talking to @ViktorHofer, this is a previously-seen NuGet issue where packages are not getting downloaded, but there's no error getting reported.

[davidwrighton]

Flow is almost there, but as @hoyosjs says this is a nuget problem. If someone knows how to reduce the likelihood of this fault. I'd be happy to help, but without that, I don't think I can do anything here.

[ViktorHofer]

Copying this from another issue but it might help here to share some context. In the last years I worked on reducing NuGet invocations where possible down to three:

1. The msbuild sdk resolver (when an msbuild sdk is downloaded from nuget which happens before restore during evaluation)
2. Arcade's pre restore when using the build.cmd/build.sh scripts.
3. The repo's overall up-front restore (Build.proj or in other repos through a solution file)

Any extra invocation like

runtime/src/mono/mono.proj

Lines 611 to 612 in 432a459

	<MSBuild Projects="$(RepoRoot)eng\empty.csproj"
	Targets="Restore"/>

is adding risk as every additional restore invocation increases the chance for a misbehavior either on the client or on the server's side. Also such extra invocations reduce perf as additional NuGet invocations are actually quite slow.

CoreClr has some of such extra invocations in their build runtime scripts when shelving out to dotnet.exe to restore:

1. runtime/src/coreclr/build-runtime.cmd

   Lines 327 to 328 in 432a459

   	powershell -NoProfile -ExecutionPolicy ByPass -NoLogo -File "%__RepoRootDir%\eng\common\msbuild.ps1" /clp:nosummary %__ArcadeScriptArgs%^
   	"%__RepoRootDir%\eng\empty.csproj" /t:GenerateRuntimeVersionFile /restore^

2. runtime/src/coreclr/build-runtime.cmd

   Lines 349 to 350 in 432a459

   	powershell -NoProfile -ExecutionPolicy ByPass -NoLogo -File "%__RepoRootDir%\eng\common\msbuild.ps1" /clp:nosummary %__ArcadeScriptArgs%^
   	"%OptDataProjectFilePath%" /t:Restore^

Removing these invocation will make the native build faster and will improve the overall reliability.

[hoyosjs]

Closing as there's no hits; #54921 tracks the centralized restore.