Nuget: No packages exist with this id in source(s): Microsoft Visual Studio Offline Packages

[wahyuen]

Describe the bug
We suspect something has changed in the recent Windows image that is causing our builds to fail. We attempt to run a dotnet build command without running an explicit restore and builds currently fail. Builds work fine if we add dotnet restore or if we run it on an ubuntu image. This seems to have only started to occur recently in the last week.

Area for Triage:
Nuget
.NET Core

Question, Bug, or Feature?:
Bug

Virtual environments affected

• macOS 10.15
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Ubuntu 20.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

Expected behavior
Build to pass and Nuget packages to resolve

Actual behavior
dotnet build fails with the following Nuget errors across various packages

Error NU1101: Unable to find package Moq. No packages exist with this id in source(s): Microsoft Visual Studio Offline Packages

[MaksimZhukov]

Hello @wahyuen , thank you for the report!
Unfortunately, we could not reproduce this issue from our side. Could you please provide public repository with minimal repro steps affected by this issue? It would help us with our investigation.

Also you can invoke "dotnet build" with verbosity flag to simplify debugging: "dotnet build -v detailed"

[wahyuen]

Hi @MaksimZhukov

We've managed to narrow it down to a Console application inside our solution. We have been issuing builds against the whole Solution for quite some time, after digging into this issue, it appears the NuGet packages only fail to resolve for projects built AFTER the console application is built. After explicitly only building the other projects (mixture of WebAPI and FunctionApps) and excluding that particular Console application, things seem to work fine with the implicit resolve.

We aren't really in a position to make our repo public at this stage, and given we have a workaround I might close this issue. We do feel that its related to the latest set of image changes given that it does work locally and on all Ubuntu variants. We even attempted to execute builds that previously passed to rule out code changes and those now fail on the latest Windows image.

Hopefully the above comments help in some way in case anyone else runs into the same issue!

[kerryjiang]

I have the same problem and just started happening from yesterday.

https://github.com/kerryjiang/SuperSocket/runs/1580782645?check_suite_focus=true

[maxim-lobanov]

@kerryjiang , image was not changed in your repo. Both first failed build and last green build have the same image version 20201210.0. Usually, it means that the VM is identical between builds.

[GewoonJaap]

I updated a few of my nuGet packages, build worked, rerun the build, it failed. Hmm. Somesort of caching issue?

[GewoonJaap]

Found the fix, add this before the build (Ofcourse change the bumbo-web stuff to your own projectfolder/name):
- name: Clean
run: dotnet clean ./Bumbo-Web/Bumbo-Web.sln --configuration Release && dotnet nuget locals all --clear

[Tyrrrz]

Had the same problem, added a clean step as suggested by @GewoonJaap and it helped.

[BigGazMetro]

Had the same issue, switch to ubuntu-latest fixed it for us.
Our solution is multi project, first project the restore works fine, subsequent ones failed with this error, also only started in last few days.

[ramveera]

I too had the same issue, with multiple projects in the same solution.
Have applied the mitigation suggested and it helped. Thank you so much @GewoonJaap

[eduherminio]

This has been a real pain is Azure DevOps for months, it's scary/painful/sad to see the same behavior in GItHub actions virtual environments now.

[mmahditalachi]

take a look at this: it works for me .

[jzabroski]

I think the root cause of this issue is some kind of service level agreement that is not bubbling up to the user. I suspect this only happens when you do a lot of builds in a certain time window, and that is why it mysteriously cannot find this packages, because you did too many requests to nuget.org, so it's falling back to this.

As for why adding nuget.org directly as a package source works, my guess is the rate metering can be bypassed if you do a random script, since they're not monitoring the equivalent of your netstat -ab for all active connections from the ci/cd process.

Anyone else think I am right?

[ramveera]

Seems to be correct but we have a workaround for the same now worries! @jzabroski

[jzabroski]

@snveera Uh, I think you missed the point that both @eduherminio and I cannot stand the fact this is not designed correctly and produces cryptic error messages. It's also not clear why the workaround works. That should bother you.

[ramveera]

Probably nugut source setting back to use offline references instead of nuget packages

[feliwir]

This bug happens to us aswell all the sudden....

[jzabroski]

I think we should open a new issue since this one is closed. Unfortunately that means everyone here will need to click subscribe to notifications.

I guess my concern is we just don't know why this happens.

[mikernet]

Just ran into this. I don't think this issue should be closed, there's an underlying issue that needs to be resolved - petition to reopen the issue until it is fixed.

[vsafonkin]

Hey there, I've opened the issue for this bug in dotnet/core repository:
dotnet/core#5881

[Turnerj]

Now that NuGet/Home#10586 is closed, saying that we "are currently relying on machine ambient state and that would lead to an unpredictable behavior", it seems that this issue is relevant again and probably should be reopened. Something is strange with the environment that is affecting NuGet.

Given the unpredictability I've seen in my own repos, it seems like a race condition of some sort on Windows between our code restoring and something else clearing the NuGet package sources.

[wokket]

Adding a "Me Too" to this, no changes to our builds in AZDO for months and out of the blue we're getting "No packages exist with this id in source(s): Microsoft Visual Studio Offline Packages" from a dotnet tool update that works fine on local PC's.

[Turnerj]

About a month ago I created a new repo designed specifically to debug this issue and set it to trigger the GitHub action every 30 minutes. Each run does 14 builds (7 Windows, 7 Ubuntu) with different combinations of .NET SDKs being installed. It builds a test project that has no code (though may change that in the future) but a single dependency to Newtonsoft.Json.

So far I haven't been able to recreate it at all in this special repo while still encountering it on and off in other repos of mine. Part of me suspects there might be something related to the repo itself (maybe its age of the repo, maybe which datacenter the repo data is stored) which is determining some additional state for the virtual environment. Might need to move the code into a branch on an existing repo of mine to see if that changes anything...

[jjuncker]

Just saw this issue again today. We use a script step in our Azure pipeline to build our project, and it seems that the latest update to dotnet 5, 5.0.202 (https://devblogs.microsoft.com/dotnet/net-april-2021-updates/), broke it, so that nuget is not used when restoring. The symptoms we see are (notice the Feeds used list):

DO NOT WORK

	NuGet Config files used:
             C:\Users\VssAdministrator\AppData\Roaming\NuGet\NuGet.Config
             C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.FallbackLocation.config
             C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config
             C:\Program Files (x86)\NuGet\Config\Xamarin.Offline.config
         
        Feeds used:
             C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

WORKS

	NuGet Config files used:
             C:\Users\VssAdministrator\AppData\Roaming\NuGet\NuGet.Config
             C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.FallbackLocation.config
             C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config
             C:\Program Files (x86)\NuGet\Config\Xamarin.Offline.config
         
        Feeds used:
             https://api.nuget.org/v3/index.json
             C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

We fixed our pipeline by using a DotNetCoreCLI@2 task instead. I guess an alternative fix would be to revert to a previous version of dotnet or wait for an update.

[miketimofeev]

@jjuncker we've reopened the related issue #3038 and will continue the investigation, thanks for your report!