NuGetSdkResolver doesn't respect pivots that come through msbuild properties (example --config-file, RestoreSources)

[zarenner]

The NuGet-based SDK resolver doesn't appear to respect --configfile, and instead just uses the nuget.config next to the project.

e.g. if I have a project-level nuget.config with a single source 'sourcenexttoproject', and a separate --configfile nuget.config with a single source 'differentsource':

c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest>dotnet restore --configfile ..\blah\nuget.config
Restoring packages for C:\Users\zarenner\AppData\Local\Temp\c8ce94a0-3218-4650-ab34-1f8f61b09f3b...
c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj : warning : Unable to load the service index for source https://sourcenexttoproject/v3/index.json.
c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj : error : Unable to find package Microsoft.Build.CentralPackageVersions. No packages exist with this id in source(s): sourcenexttoproject
  Restoring packages for c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj...
C:\Program Files\dotnet\sdk\2.2.104\NuGet.targets(114,5): error : Unable to load the service index for source https://differentsource/v3/index.json. [c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj]
C:\Program Files\dotnet\sdk\2.2.104\NuGet.targets(114,5): error :   No such host is known [c:\repos\CodeSharing.Sandbox\Samples\sdkresolvertest\sdkresolvertest.csproj]

The SDK is trying to restore from sourcenexttoproject, instead of from differentsource as I would expect it to. Note the sources are both invalid themselves, so the fact that there are "unable to load" errors on each is expected.

This bug completely breaks the DotNetCoreCli build task in Azure Pipelines when used against authed sources, as the task writes a temporary nuget.config with credentials and passes it via --configfile.

@jeffkl as FYI since I had discussed with him using his project SDKs for our own codebase, and this is blocking our adoption.

[nkolev92]

This is becoming kind of challenge, because restore itself has various configuration pivots that likely don't work with the SDK resolver.

--config-file is one. I imagine RestoreSources & similar msbuild settings for NuGet don't work.

I'd imagine the whole list from https://github.com/NuGet/Home/wiki/%5BSpec%5D-NuGet-settings-in-MSBuild is an issue.

And that'll continue to be a moving target in the future.

//cc @rrelyea

[jeffkl]

Anything settings coming from MSBuild will not work with the SDK resolver because the project has not been evaluated yet. So no properties or items are loaded to even pass along. The feeds must be defined in the root nuget.config and auth tokens will have to be in that file that can be found by the SDK resolver or the credential provider needs to get them.

We could possibly plumb the custom path to the nuget.config to the SDK resolver but that wouldn't be available until 16.1 at the earliest. I would like to find a workaround that unblocks people immediately.

Doesn't the newer credential provider use environment variables to pass along auth tokens? Editing the nuget.config in a temp directory and then passing the path to dotnet.exe restore or nuget.exe seems less stable than setting env vars that the credential provider can pick up.

[nkolev92]

A similar issue was raised in #8183

[markwilkie]

What the current state of this issue? We (.NET Core) are running into this as well and are curious what the current thinking is.

cc/ @jcagme

[nkolev92]

No progress has been made on it.

Note that this scenario adds some inconvenient coupling so if you can at all avoid it, please do so.

[jcagme]

You are saying we should not try restoring SDK packages from a private feed?

[nkolev92]

Don't specify the config file directly & allow the tooling to discover it by convention.

If you put the config file at the repo root and/or near the solution files, ti will be discovered by the tooling automatically.
You can disable other cascading configs interfering with yours by adding clear tags like the ones we use in the NuGet Client repo.
https://github.com/NuGet/NuGet.Client/blob/dev/NuGet.Config

[jcagme]

Well, at least for our case explained here we don't pass in the location of nuget.config and use the one in the root. Since we are trying to restore packages from a private AzDO feed, probably cred provider is doing that under the covers?

[nkolev92]

The cred providers were enabled for the sdk resolver so that should be working the same as regular restore.

I'm not sure why it's having trouble detecting the plugin however.
It should be under ~/.nuget/plugins

[zarenner]

I should note that the Azure Pipelines DotNetCoreCLI task does not use the credential provider today. It uses a flow where it generates a temporary config, and passes --configfile to dotnet. That means that the DotNetCoreCLI task definitely won't work today with MSBuild SDKs in authenticated feeds.

If I understand correctly, @jcagme might be seeing another issue where nuget doesn't invoke the credential properly for MSBuild SDKs. I think we need to investigate that further - I haven't had a chance to look into that yet.

[jcagme]

The cred providers were enabled for the sdk resolver so that should be working the same as regular restore.

I'm not sure why it's having trouble detecting the plugin however.
It should be under ~/.nuget/plugins

Yes, the installation script put the plugin in ~/.nuget/plugins/netcore

I should note that the Azure Pipelines DotNetCoreCLI task does not use the credential provider today. It uses a flow where it generates a temporary config, and passes --configfile to dotnet. That means that the DotNetCoreCLI task definitely won't work today with MSBuild SDKs in authenticated feeds.

If I understand correctly, @jcagme might be seeing another issue where nuget doesn't invoke the credential properly for MSBuild SDKs. I think we need to investigate that further - I haven't had a chance to look into that yet.

Yeah, for our builds we don't use the task but use dotnet restore directly. This is what I've done:

Windows

1. Run installation script
2. Run build script
   Output: successfully restored packages in private feed

Mac

1. Run installation script
2. Run build script
3. Since things didn't work, installed VS for Mac and signed in using my [at]microsoft account
4. Run build script
   Output:
   /Users/jcagme/code/jcagme/arcade/Directory.Build.props(3,3): warning : Unable to load the service index for source https://xxx.pkgs.visualstudio.com/_packaging/xxx-test/nuget/v3/index.json. /Users/jcagme/code/jcagme/arcade/Directory.Build.props(3,3): error : Unable to find package Microsoft.DotNet.Arcade.Sdk. No packages exist with this id in source(s): dotnet-core, nuget.org /Users/jcagme/code/jcagme/arcade/Directory.Build.props(3,3): warning : The plugin credential provider could not acquire credentials. Authentication may require manual action. Consider re-running the command with --interactive for dotnet, /p:NuGetInteractive="true" for MSBuild or removing the -NonInteractive switch for NuGet

Azure DevOps build agent

1. Run installation script
2. Run build script
3. Run build script
   Output: same as in Mac

[zarenner]

Is it possible your VSS_NUGET_EXTERNAL_FEED_ENDPOINTS envvar isn't getting set or is incorrect?

If it's a domain account perhaps the windows case is working because it's silently / non-interactively authenticating and thus doesn't need VSS_NUGET_EXTERNAL_FEED_ENDPOINTS? Not sure what identities are in play here for you.

[jcagme]

I thought that to be the case but I then included a step to echo the variable value and it is correct. It looks like {"endpointCredentials": [{"endpoint":"https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json", "password":"<BotPAT>"}]}

BotPAT is the PAT of an account I explicitly added as a contributor of the feed.

[zarenner]

Unable to load the service index for source https://xxx.pkgs.visualstudio.com/_packaging/xxx-test/nuget/v3/index.json

{"endpointCredentials": [{"endpoint":"https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json", "password":"<BotPAT>"}]}

pkgs.dev.azure.com/<organization> vs <organization>.pkgs.visualstudio.com mismatch?

[jcagme]

True, although https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json is what I have in nuget.config in Windows (the one which works) and also that's what I get in Azure DevOps artifacts "Connect to feed". Should I attempt https://xxx.pkgs.visualstudio.com/_packaging/xxx-test/nuget/v3/index.json?

[zarenner]

It must match up exactly between nuget.config and endpointCredentials. Putting both forms into endpointCredentials won't hurt.

[jcagme]

0>F:\workspace\_work\1\s\artifacts\toolset\restore.proj : error : Unable to load the service index for source https://pkgs.dev.azure.com/xxx/_packaging/xxx-test/nuget/v3/index.json. 0>F:\workspace\_work\1\s\artifacts\toolset\restore.proj : error : Unable to load the service index for source https://xxx.pkgs.dev.azure.com/_packaging/xxx-test/nuget/v3/index.json.

It fails with both endpoints. Both endpoints live in nuget.config

[nkolev92]

@jeffkl
Do you think the general problem is still something we should try to address?
Specifically

Anything settings coming from MSBuild will not work with the SDK resolver because the project has not been evaluated yet. So no properties or items are loaded to even pass along.

[jeffkl]

The only thing we could plumb through would be global properties since they are loaded before a project is evaluated. You could then set the global property via the command-line or Directory.Build.rsp. We will never be able to use project properties in the SDK resolver since there is no way to "peek" ahead.

The issue for this proposed feature is here: dotnet/msbuild#2095

[ghost]

Note, posted stand alone repro in item #10886

@jeffkl , any news on this?

[jeffkl]

I do not think we'll be adding support for this any time soon. The workaround is to just check in your NuGet.config and use the same sources for all builds. The ability to dynamically use different sources at build time is just not a very common scenario. If you want to dynamically change sources at build time, you'll need to write a script that changes the NuGet.config before restore and build.

[ghost]

I'm aware of the workaround, but it wastes developer time to have a documented feature that doesn't work.

Would highly recommend updating the docs for --config-file to indicate that it doesnt' work for Sdk imports.

[ghost]

Also, consider piping the appropriate config parameters as environment variables. E.g. add NUGETSDKRESOLVER_CONFIG_FILE and update dotnet.exe --config-file to configure it appropriately.

[Pvlerick]

Also, consider piping the appropriate config parameters as environment variables. E.g. add NUGETSDKRESOLVER_CONFIG_FILE and update dotnet.exe --config-file to configure it appropriately.

Having an env variable would indeed be helpful...

[m0sa]

Running into the same issue, the RestoreConfigFile MsBuild property / env variable is not being picked up.

[amaltinsky]

I'm aware of the workaround, but it wastes developer time to have a documented feature that doesn't work.

Would highly recommend updating the docs for --config-file to indicate that it doesnt' work for Sdk imports.

I just wasted several hours tracking down this issue in a CI pipeline. I'd go one step further than the documentation and suggest changing the error message shown by NuGet in this case from "error MSB4236: The SDK .. specified could not be found" to something more explicit about NuGet not being able to restore SDKs when --config-file option is provided.