Workflow fails to download dependencies from NuGet.org

[AaronVanGeffen]

Description
For the OpenLoco project, we gratefully make use of GitHub Actions for our CI. To this end, we set up a workflow for our Windows CI:
https://github.com/OpenLoco/OpenLoco/blob/master/.github/workflows/ci.yml#L18-L59

The Visual Studio project this workflow builds has been set up to depend on a package from NuGet.org:
https://github.com/OpenLoco/OpenLoco/blob/master/src/OpenLoco/openloco.vcxproj#L352-L354
https://www.nuget.org/packages/openloco.dependencies

This workflow worked fine until two days ago. However, we have noticed the package no longer gets retrieved properly, causing most (but not all) runs to fail.

Area for Triage:
C/C++

Question, Bug, or Feature?:
Bug

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

Image version
20210321.1 (broken)
20210316.1 (fine)

Expected behavior
Successful run on image version 20210316.1, 3 days ago
https://github.com/OpenLoco/OpenLoco/runs/2181845951

Restore:
  Restoring packages for D:\a\OpenLoco\OpenLoco\src\OpenLoco\openloco.vcxproj...
    GET https://api.nuget.org/v3-flatcontainer/openloco.dependencies/index.json
    OK https://api.nuget.org/v3-flatcontainer/openloco.dependencies/index.json 173ms
    GET https://api.nuget.org/v3-flatcontainer/openloco.dependencies/1.3.0/openloco.dependencies.1.3.0.nupkg
    OK https://api.nuget.org/v3-flatcontainer/openloco.dependencies/1.3.0/openloco.dependencies.1.3.0.nupkg 105ms
  Installed OpenLoco.Dependencies 1.3.0 from https://api.nuget.org/v3/index.json with content hash 2kTdy9fJCCYiCDi5QPxbJQ8st/7AfT+gt+rjAz++yONLyxOIBQbdk2g5Oqv5VSU/UAPZ8BrNSanNdxcYDszkFA==.
  Committing restore...
  Generating MSBuild file D:\a\OpenLoco\OpenLoco\src\OpenLoco\obj\openloco.vcxproj.nuget.g.props.
  Generating MSBuild file D:\a\OpenLoco\OpenLoco\src\OpenLoco\obj\openloco.vcxproj.nuget.g.targets.
  Writing assets file to disk. Path: D:\a\OpenLoco\OpenLoco\src\OpenLoco\obj\project.assets.json
  Restored D:\a\OpenLoco\OpenLoco\src\OpenLoco\openloco.vcxproj (in 7.18 sec).

Actual behavior
Failed run on 20210321.1 today:
https://github.com/OpenLoco/OpenLoco/runs/2204773635

Restore:
  Restoring packages for D:\a\OpenLoco\OpenLoco\src\OpenLoco\openloco.vcxproj...
D:\a\OpenLoco\OpenLoco\src\OpenLoco\openloco.vcxproj : error NU1101: Unable to find package OpenLoco.Dependencies. No packages exist with this id in source(s): Microsoft Visual Studio Offline Packages [D:\a\OpenLoco\OpenLoco\openloco.sln]
  Committing restore...
  Generating MSBuild file D:\a\OpenLoco\OpenLoco\src\OpenLoco\obj\openloco.vcxproj.nuget.g.props.
  Generating MSBuild file D:\a\OpenLoco\OpenLoco\src\OpenLoco\obj\openloco.vcxproj.nuget.g.targets.
  Writing assets file to disk. Path: D:\a\OpenLoco\OpenLoco\src\OpenLoco\obj\project.assets.json
  Failed to restore D:\a\OpenLoco\OpenLoco\src\OpenLoco\openloco.vcxproj (in 942 ms).

Repro steps
Please look at https://github.com/OpenLoco/OpenLoco/actions

[maxim-lobanov]

Hello @AaronVanGeffen , do you have nuget.config file in your repo?
I think it is known issue when you don't have config file.
Could you please try to add simple nuget.config like described in the message: actions/setup-dotnet#155 (comment)

[AaronVanGeffen]

Hello @maxim-lobanov, thank you for the suggestion. Indeed, there was no nuget.config file in our repository. Previously this worked fine, so I was not aware we would need any. However, adding the config file as you suggested appears to have helped: OpenLoco/OpenLoco#855

[maxim-lobanov]

Cool, glad to hear that it helped.

Looks like sometimes dotnet doesn't resolve packages from remote when doesn't find packages in local cache.
I think It is not the new issue since based on actions/setup-dotnet#155 (comment), it fails pretty randomly without nuget.config file.

The root cause is still unclear but using nuget.config is recommendation from NuGet team to deal with this issue: NuGet/Home#10586 (comment)

[AaronVanGeffen]

Thank you for the explanation. Just to confirm, CI has been working reliably tonight with the nuget.config file in.

[miketimofeev]

@AaronVanGeffen thanks for the confirmation!
I'm going to close the issue but feel free to contact us if you have any concerns.

[rokups]

I bumped into same issue. While workaround does work i noticed that PATH printed by setup-dotnet action differs on failing and succeeding builds. Failing builds have C:\hostedtoolcache\windows\Java_Adopt_jdk\8.0.282+8\x64\bin in the PATH and succeeding builds have C:\Program Files\Java\jdk8u282-b08\bin. I suspect not all build workers use same image even if all of them use windows-latest.

[miketimofeev]

@rokups it takes 3-4 days to propagate the new image (with Java in the hostedtoolcache directory) to all the environments. We're going to finish the deployment on Monday

[fabriciomurta]

Hello, I believe this issue should be better investigated and (hopefully) solved without the need of wrapping our own nuget.config file. The NuGet config should be the same across different instances of Windows runners and I would also expect them to match Mac and Linux (which does not suffer from this problem) runners;

It sounds reasonable to have nuget.org source enabled by default if any packages are not found in local cache (which makes sense for the runtime environments not to keep downloading packages form NuGet.org every run).

Note: I have two different runs in the same 20210330.2 environment version, one failing and another succeeding. It really looks like runners are not being properly cleaned up after a previous run.

[miketimofeev]

@fabriciomurta no, it's not possible — every run is performed on a clean agent

[asklar]

@miketimofeev @maxim-lobanov Can you please reactivate this issue and help us get to a resolution? The core NuGet issue NuGet/Home#10586 was closed without a resolution other than "clean the nuget cache" or "have a nuget.config" which don't seem productive, since restoring projects used to work fine before the last AzDO pipelines/GHA updates.
You're saying it's not possible that this is due to non-clean state on the agent (which I'd agree with), but nuget folks are implying the contrary.

Both this and the NuGet issue are closed and we need someone to step up in either of the teams and provide a solution. We have multiple repos - both internal and external - that got broken since the latest update.

[miketimofeev]

@asklar reopened. Could you please share some repo where the issue persists? We've tried to reproduce last time without any luck

[rokups]

You may use https://github.com/rokups/rbfx if it is not too fat. Delete nuget.config from root folder to make it fail.

[asklar]

@miketimofeev thanks. We're hitting this in https://github.com/microsoft/react-native-windows/ among others

[miketimofeev]

Hello, everyone!
The new image has been deployed. Please check if you have any issues now.
We're going to close this issue after a few confirmations that everything works fine now.

[miketimofeev]

We've found the root cause of the empty config file and it's related to the chocolatey chocolatey/choco#2233

[anpaz]

@miketimefeev, what image version was this fixed on? I just tried on '20210411.1' and I still see the problem.

[asklar]

@miketimofeev thanks a lot for drilling down and finding the issue, much appreciated! This would also explain why some of our customers who use our dependencies script (which uses choco) are hitting this too!

[xt0rted]

We're going to close this issue after a few confirmations that everything works fine now.

I just updated my test PR that was doing Remove-Item $env:APPDATA\NuGet\NuGet.Config and I'm no longer getting the NuGet restore error so it looks to be working now.

My setup is .net sdk 5.0.202, no nuget.config, and setting a GPR feed in the actions/setup-dotnet step. Prior to this update only packages in the GPR feed would restore but now the ones from nuget.org are as well.

[ltrzesniewski]

I removed a build step which executed the following on 2 of my repos:

dotnet nuget locals all --clear
dotnet nuget add source https://api.nuget.org/v3/index.json --name nuget.org

And the builds run fine now! Thanks!

Additional details:

• One of the builds uses dotnet restore, the other one uses msbuild /t:Restore
• I don't have nuget.config files in the repos

[robgilliam]

@miketimofeev when/where will this be fixed?

This started to be a problem for us on 29th April (4am, UK time) and is still failing today on the 20210425.1 release of windows-latest for any of our Cake-based builds, which are trying (and failing) to restore Cake from NuGet.org:

due to the $env:AppData\NuGet\NuGet.Config having no sources specified:

(Note: this content is slightly different to the one shown in chocolatey/nuget-chocolatey#23)

This is affecting all our Cake-based actions, only one of which is explicitly using Chocolatey (all are using actions/checkout@v2 and actions/setup-dotnet@v1, if that's significant).

Adding our internal company NuGet server as a source using a run: step that executes before the attempt to download Cake "fixes" (i.e. works around) the problem as it adds the default nuget.org source to the configuration as well as the company server.

[robgilliam]

I've dug into this a bit more, and in (at least one of) the actions that don't use Chocolatey, the bad NuGet.Config file is created by the following lines in the .yaml file for the action:

  run: |
    Import-Module PowerShellGet
    $env:PSModulePath = "C:\Modules\azurerm_6.13.1;C:\ProgramFiles\WindowsPowerShell\Modules;C:\windows\system32\WindowsPowerShell\v1.0\Modules"
    Install-Module AzureAD -Force
    # Invoke PowerShell script which runs Cake build here.

The last one (Install-Module) is the actual culprit: remove that and the source-less NuGet.Config file isn't created, and the Cake package restore works fine.

[miketimofeev]

@robgilliam chocolatey already fixed the issue on their side, but we have to wait for a new release. Speaking about Powershell module installation — it's a new case we've never seen before. Does any module installation cause the issue or it's AzureAD specific?

[robgilliam]

@miketimofeev I don't know - all of my "not using Chocolatey" actions that are affected are just using PowerShellGet to install the AzureAD module.

Having identified the extent of the impact on our projects and the easiest work-around (add a nuget sources list step after setup-dotnet runs and before anything else), and spent nearly three days (and one very late night) fixing it, I can't spend more time investigating further. Sorry.

[miketimofeev]

@robburger thanks! I can confirm — the issue is reproducible with any PowerShell module. We will engage the Dotnet team.

[miketimofeev]

We've added NuGet.config with nuget.org source in the user's home directory so that neither choco nor PowerShell won't create it using their embedded nuget.
I'm going to close the issue. If anyone sees any other cases please contact us.
Thank you!

[fsarter]

I removed a build step which executed the following on 2 of my repos:

dotnet nuget locals all --clear
dotnet nuget add source https://api.nuget.org/v3/index.json --name nuget.org

And the builds run fine now! Thanks!

Additional details:

• One of the builds uses dotnet restore, the other one uses msbuild /t:Restore
• I don't have nuget.config files in the repos

You saved my life !!! I enounter the same issue after installing Visual Studio 2022 for the first time. I think i should get some details about nuget. Thanks a lot.