DotNetCoreCLI restore - No credentials are available in the security package

[kstabins]

Question, Bug, or Feature?
Type: Bug

Enter Task Name: DotNetCoreCLI

Environment

• Server - Azure Pipelines

  • Azure DevOps Server 2019 (17.143.28621.4)

• Agent - Private:

  • Agent.OS: Windows_NT
  • Agent.OSArchitecture: X64
  • Agent.OSVersion: 6.3.9600 (Widows Server 2012 r2)
  • Agent.Version: 2.144.2

Issue Description

donet restore actions fail when using Azure Artifacts.

The generated tempNuGet.config key does not match the xml element in the packageSourceCredentials.

Task logs

error : Unable to load the service index for source
error : No credentials are available in the security package

log.txt

[sachinma]

@alexmullans can you take a look at this

[zarenner]

I believe this is due to a known issue in dotnet 2.1.500 when the key starts with a number. Could you please try using a newer version of .NET Core (via https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/dotnet-core-tool-installer?view=azure-devops) and see if that solves the issue?

We added a workaround to prefix the generated feed names with "feed-", but it looks like that workaround didn't get into 2019 RTW. The suggested fix however is still to avoid .NET Core 2.1.500.

[kstabins]

@zarenner, thanks, but we are using dotnet\sdk\2.2.101

I was able to work around the problem by taking the generated NuGet.config and adjusting the feed name like you suggested. The only problem is now we need to hard code the NuGet.config.

[NeoXtreem]

@zarenner, thanks, but we are using dotnet\sdk\2.2.101

I was able to work around the problem by taking the generated NuGet.config and adjusting the feed name like you suggested. The only problem is now we need to hard code the NuGet.config.

@kstabins I tried the following steps and ran my build multiple times. Occasionally it succeeds but mostly not, therefore I cannot state whether these steps have any impact.

1. Add a preceding step .NET Core SDK Installer and set Version to 2.2.300.
2. Set Clean to true.

I've no idea why it would be temperamental. But this is a big problem for us as it means packages are very difficult to upgrade.

@zarenner Any idea when this will be in a patch release?

[zarenner]

@kstabins I believe 2.1.101 may also be affected. I know that 2.1.100 is affected. @nkolev92 may know.

@NeoXtreem If you mean when the workaround we added made it into Azure DevOps Server (on-premises), I don't know off-hand.
@satbai could you please help me determine the first on-prem patch release that da020cd#diff-2167ac79815cbb9808f0d90234539acd made it into?
Also, please note that this workaround doesn't do anything in cases where you have an existing nuget.config with a package source name that starts with a number. Upgrading your version of .NET Core SDK is the recommended solution.

Links for reference:
NuGet/Home#7516
NuGet/Home#7517
NuGet/Home#7524

[nkolev92]

2.2.100 and 2.1.500 carry the same NuGet bits, so it is affected.

More info in the release notes: https://docs.microsoft.com/en-us/nuget/release-notes/nuget-4.9-rtm#summary-whats-new-in-492

[zarenner]

Thanks, that's a very useful page. According to that though, looks like 2.2.101 (which @kstabins said they're using) has nuget 4.9.2 though, which should have the fix?

[satbai]

@zarenner, I don't think the commit you mentioned made it to Azure DevOps 2019 so it's not available on-prem yet.

[zarenner]

Looks like it did make it into Azure DevOps Server 2019.0.1, though.
not sure about this, still checking.

[NeoXtreem]

Upgrading your version of .NET Core SDK is the recommended solution.

@zarenner Are you suggesting that if I upgrade .NET Core on the CI server to 2.2 (it's currently on 2.1), then this may solve the problem?

[zarenner]

Assuming that the encoding issue is the cause of your issue (there are other reasons why you might be getting "No credentials are available in the security package"), then yes, upgrading either to the latest 2.1 or 2.2 SDK should solve it. The specific SDK versions are problematic, not 2.1 vs 2.2 overall.

Have you confirmed that you have the same issue as @kstabins?
If you're seeing the "No credentials are available in the security package" error, can you check if you have an encoding mismatch between packageSources and packageSourceCredentials in the generated tempNuGet.config?

[NeoXtreem]

@zarenner Yes, I am getting that error and I have the encoding mismatch in the generate tempNuGet.config file (the GUID key is prefixed with "x0030" in packageSourceCredentials).

[zarenner]

Cool, then upgrading your .NET Core SDK should fix it.

[NeoXtreem]

Cool, then upgrading your .NET Core SDK should fix it.

@zarenner Just to note, I did add a .NET Core SDK Installer step in my build pipeline specifying version 2.2.300 which is then used for the subsequent dotnet restore, but it still failed. Is that what you would expect, and that I should still upgrade the SDK on the server?

And what's more interesting is that I tried @kstabin's workaround of including my own NuGet.config as a copy of the temporary one generated, and just changing the key in packageSources and packageSourceCredentials to be the same and beginning with a letter. However, I still get the same error.

And an aside (not sure if relevant), if I try putting my own credentials in packageSourceCredentials instead of the generated VssSessionToken one, when I look at the generated tempNuGet.config, it replaces them with generated VssSessionToken credentials again (although the package source key remains the same). I'm not sure if this is clever or part of the problem.

[kstabins]

@zarenner We updated to .Net Core 2.2.300 and we still receive the same error.

[NeoXtreem]

Cool, then upgrading your .NET Core SDK should fix it.

As @kstabins found also, it doesn't fix it.

As a workaround, I've resorted to using NuGet restore instead with a preceding NuGet Tool Installer step specifying version 5.1.0. Poor show by the Azure Pipelines team on this really.

[zarenner]

Hi @NeoXtreem and @kstabins,

Sorry for the delay. I'm glad you've found a workaround.

I completely agree with you and apologize that we (as Azure Artifacts and the NuGet/dotnet team) have dropped the ball in many ways on making auth a good experience today. The truth is that throughout the history of the NuGet client, there have been a number of auth-related issues that have often required us to implement workarounds in the build tasks. That has sometimes led to further issues, or at least inconsistencies between tasks in how they do auth especially as new versions of nuget/dotnet come out.

In this particular case, I believe what's happening is:

• You hit the encoding issue (.NET SDK 2.1.500 Auth problem with NuGet Source with space in its name (the credentials are not encoded) NuGet/Home#7516, VS/dotnet.exe/nuget.exe/msbuild.exe restore doesn't use credentials when source name contains a whitespace NuGet/Home#7517, Regression in .NET SDK 500: 'dotnet tool install' fails with 401 (Unauthorized) when there is a private feed in NuGet.config NuGet/Home#7524) and worked around it manually. Our task's workaround for this issue was Prefix package sources with feed- when writing packageSourceCredentials in nuget.config #8992 but did not make it into Azure DevOps Server 2019 RTM. Upgrading your .NET Core SDK should have definitely fixed the actual issue in NuGet. However...

• You likely then hit a separate issue which can lead to the "No credentials are available in the security package" error, related to oddities in NuGet around on-premises NTLM/Windows authentication. I think you're hitting NuGet prefers NTLM auth even if there are credentials in NuGet.config - add config option to filter auth types for credentials NuGet/Home#5286, where NuGet incorrectly tries to use NTLM authentication even if there are credentials specified in the (in this case generated) nuget.config. I believe this issue can be worked around by setting the 'ValidAuthenticationTypes' setting in nuget.config (Add config option to filter authn types for credentials NuGet/NuGet.Client#2297) to 'basic'.
  However...

• The dotnet task doesn't automatically do the ValidAuthenticationTypes workaround for you (you'd have to manually do it in your own nuget.config I believe). The dotnet task also still uses the old generated temporary nuget.config mechanism instead of the newer credential provider.

• The credential provider mechanism is unaffected by nuget's odd NTLM behavior. The NuGet build task uses the credential provider, hence why switching to it solved your issue.

Moving forward, we're planning to introduce build tasks that set up the credential provider for you so you can then simply use dotnet or nuget on their own without needing the existing dotnet/NuGet build tasks at all. However, it will be still a while until these tasks make it into Azure DevOps Server.

Adding @infin8x, @elbatk, @zjrunner, @rrelyea, and @nkolev92 to make them aware of these issues and hopefully increase our team's priority on making auth "just work" and adding appropriate documentation. Also adding @jmyersmsft since he has context on NuGet's NTLM issues in case I've misstated something.

[NeoXtreem]

@zarenner, NuGet/Home#5286 was fixed in NuGet 5.0 which is released. Should the dotnet task (or .NET Core SDK?) be upgraded to use this version, to fix the issue properly?

Also, your colleague Krishna Verma from Microsoft Support contacted me (I raised a support request via our MPN Partner support plan), and he suggested this solution which uses a Command Line task to call dotnet.exe directly. As with the poster of that solution, this works, and I guess it's because it also uses the credential provider mechanism, and is unaffected by the NTLM authentication issue. But I think my own workaround is better as it avoids command lines and absolute paths.

Please let me know your thoughts on the above, and whether I should continue using my existing workaround until you introduce the new build task.

[zarenner]

@NeoXtreem my understanding of that 'fix' is that is it's simply the added support for ValidAuthenticationTypes (which still must be specified, so it's not an 'automatic' fix). @nkolev92 in case there are more changes than that. Therefore I'm not sure that upgrading the dotnet pipelines task to use a new SDK (which you can already do with a tool installer task) is worthwhile for this issue.

Directly invoking 'dotnet' via a script/cmdline task will be be our strong recommendation going forward once the auth tasks are ready, so if a solution involving directly invoking 'dotnet' works for you today, I would recommend that. Specifying an absolute path shouldn't be necessary at least when using a Tool Installer task, which should be putting it on the path.
That said, it's not exactly clear to me exactly how just running 'dotnet' is solving the issue for you, unless you've also set up the credential provider to use the build identity somehow. If it was still using NTLM auth, I'd expect the same error.

[jmyersmsft]

The fix for 5286 on its own does indeed only add support for reading ValidAuthenticationTypes from the config.

My guess is that the build agent is running as a Windows domain user which has access to the feed, and when running dotnet.exe directly, there's an environmental difference such that credentials from the config are (for whatever reason) not being picked up at all, so NuGet is actually authenticating as that Windows domain user over NTLM/Kerberos rather than authenticating as the build identity.

[zarenner]

Yeah that seems likely. As such, @NeoXtreem here's some options:

1. If you're ok with using the Windows user identity running the agent (rather than the build identity), use dotnet directly.

2. If you want to use the build identity while still using the NuGet task, then make sure you're on a .NET Core SDK equivalent to NuGet 5.0 or higher (see https://docs.microsoft.com/en-us/nuget/release-notes/nuget-5.0-rtm) and set ValidAuthenticationTypes.

3. Once the DotNetAuthenticate task is available, you could use that to use the build identity while directly using dotnet.

4. It's actually possible to mostly do the equivalent of the upcoming DotNetAuthenticate task today by:
   a. Acquiring https://github.com/Microsoft/artifacts-credprovider yourself (either installing it to the user directory or downloading it during the build)
   b. Setting NUGET_PLUGIN_PATHS to the credential provider dll path (if not installed to the user directory)
   c. Setting the envvar VSS_NUGET_URI_PREFIXES to e.g. https://pkgs.dev.azure.com/{yourorganizationname};https://{yourorganizationname}.pkgs.visualstudio.com/
   d. Setting VSS_NUGET_ACCESSTOKEN to $(System.AccessToken)
   e. If you use service connections, you'd have to instead manually populate those credentials into VSS_NUGET_EXTERNAL_FEED_ENDPOINTS (see artifacts-credprovider docs).

   This is a pain to do manually (which is why we're releasing the authentication tasks), but I wanted to mention it in case the existing solutions don't work for you.

5. If on Windows, instead use the NuGetCommand task to do the restore. You may also need a NuGet Tool Installer task in order to obtain a recent version of nuget.exe (e.g. 5.1). This will (in most cases) use the credential provider, thus authenticating as the build identity.

[NeoXtreem]

... it's not exactly clear to me exactly how just running 'dotnet' is solving the issue for you, unless you've also set up the credential provider to use the build identity somehow. If it was still using NTLM auth, I'd expect the same error.

@zarenner I'm not running 'dotnet', as I mentioned earlier:

I've resorted to using NuGet restore instead with a preceding NuGet Tool Installer step specifying version 5.1.0.

^This seems to be a fifth option to your list. Would you not agree? If so, then I would prefer to continue using this option (as it works) until your option 3 is possible. The other options seem overkill given the current workaround and upcoming solution (also mentioned by @elbatk in #10800).

[zarenner]

Sorry, I interpreted your message as that you had also tried directly invoking dotnet and it worked.

Completely agree. If you only build on Windows, then using the NuGetCommand task is definitely also a valid workaround and probably easiest. If you're building across multiple platforms, the other workarounds may be easier today and may closer match our recommendations in the future. I've updated my previous comment, thanks.

[MarkKharitonov]

We face the same problem. Unbelievable!

[aasim]

@phil-hodgson
Verify this commit made it into 2019.1.0.

[Sodadk]

We have just update our Azure DevOps server yesterday and seem to have the some problem with one of our projects that have been updated from .Net Core 2.2 to .Net Core 3.1. (We had the same issue before upgrading the DevOps server but thought it might solve the problem since this thread is quite old now).

Any news on a fix to this problem?

I've been through the entire thread and tried all solutions mentioned here with no luck.

I have not tried edit the NuGet.Config file with the solution from MarkKharitonovs reply as I cannot find the location of this file in my solution or in my build pipiline.

[TysonMN]

I have not tried edit the NuGet.Config file with the solution from MarkKharitonovs reply as I cannot find the location of this file in my solution or in my build pipeline.

I don't think a NuGet.Config is required. You can read the documentation for it here.

If I remember correctly, Visual Studio looks for any file called NuGet.Config (ignoring case) in the same directory as the solution (sln) file or recursively in any parent directory. I put my NuGet.Config file in the root of the repository and then put the solution file in a directory called src.

Here is an example of one of my NuGet.Config files (after removing the personal details).

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <solution>
    <add key="disableSourceControlIntegration" value="true" />
  </solution>
  <packageRestore>
    <add key="automatic" value="True" />
  </packageRestore>
  <activePackageSource>
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
  <packageSources>
    <add key="NuGet v3" value="https://api.nuget.org/v3/index.json" />
    <add key="PersonalNuGetSource1" value="https://pkgs.dev.azure.com/ProjectName/_packaging/FeedName1/nuget/v3/index.json" />
    <add key="PersonalNuGetSource2" value="https://pkgs.dev.azure.com/ProjectName/_packaging/FeedName2/nuget/v3/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <FeedName1>
      <add key="Username" value="MyUserName" />
      <add key="ClearTextPassword" value="MyPassword" />
    </FeedName1>
    <FeedName2>
      <add key="Username" value="MyUserName" />
      <add key="ClearTextPassword" value="MyPassword" />
    </FeedName2>
  </packageSourceCredentials>
</configuration>

[greektreat]

I am having Similar problems I don't understand what is going on the I am running azure devops 2019 onprem the VSTS Agent is running as [DOMAIN]\svc_azureDevops I have followed these steps and added this user as owner of the Azure Artifacts feed and i still get get the credential issue above.

[satbai]

If you're still experiencing auth issues, please add NuGet Authenticate task in the beginning of the pipeline. You may want to create a separate script that runs the dotnet command that you need if the dotnet task still isn't working after you've added the NuGet Authenticate task.

If you're using the latest version of the on premise Azure DevOps Server, you can use the script described here to add the auth tasks to your on prem instance.

[greektreat]

If you're still experiencing auth issues, please add NuGet Authenticate task in the beginning of the pipeline. You may want to create a separate script that runs the dotnet command that you need if the dotnet task still isn't working after you've added the NuGet Authenticate task.

If you're using the latest version of the on premise Azure DevOps Server, you can use the script described here to add the auth tasks to your on prem instance.

Thanks, I figured out I needed to restart the VSTS Service Agent and the permissions were applied

[greektreat]

Hi, I am stuck with this issue again. This is only happening because I set up a new agent on a new server (WIndows 2019) on my original server which is also hosting Azure devops I do not have this issue.

[greektreat]

A co-worker found a work around. We used --force command argument described here which solved the issue.https://stackoverflow.com/questions/53430896/nuget-restore-fails-on-azure-devops-with-message-unable-to-load-the-service-ind

[vinads]

Indeed had the same error when using dotnet restore task in yaml in Azure DevOps 2019 server . Agent service is running under service account

instead I've replaced the dotnet restore task with a powershell task that uses the following code :

$projs = gci -Path . -Recurse -Filter "*.csproj";

$projs.FullName | %{dotnet restore "$_." --configfile Nuget.config --verbosity Detailed};

Here we are using a Nuget.config to list the sources used (one Azure Artifact feed and one Nuget.org)

hope this workaround helps others

[ChristoWolf]

I am also having the originally described issue. For now, I am using the NuGet restore work-around.
Is there an ETA for the release of DotNetAuthenticate task? I could not find any pointers to it while googling.
Also, just for my understanding: why is it not possible for dotnet restore to implicitely authenticate similarly to NuGet restore?

[satbai]

@ChristoWolf, NuGet Authenticate task can be used with dotnet as well. In your yaml you can do something similar to below.

- task: NuGetAuthenticate@0
- script:
   dotnet restore example.csproj --configfile nuget.config --verbosity Detailed
  displayName: 'Dotnet restore'

[ChristoWolf]

@satbai: Thanks for the hint, I of course also wanted to try that before posting, but I do not have that task available.

[modestotech]

Hi, got redirected here from by Microsoft trough a thread I created in Microsoft Developer Community.
I'm facing similar issues, and after reading the thread the workaround seems to be to use
- task: NuGetAuthenticate@0, but this task is not available for me.
Is there anything more that we can try, I don't want to install more than necessary on the build server.

[detilium]

I'm still having this issue with dotnet 3.1. Am I supposed to make some workaround to get this to work? This has been as issue since May last year.

[benzhi2011]

I am using the Microsoft Visual Studio Team Foundation Server Version 16.131.28507.4. The package (.net core 3.1)was builded and published to TFS feed. I can reference to the package locally without any problem. I got same error when I setup the build for my project, the restore task failed on TFS.
Any workaround?

[tinohager]

Please fix this problem on Azure DevOps Server - it has been open for over a year now. (dotnet 3.1)

[satbai]

Azure DevOps Server 2020 release comes with the NuGet Authenticate task and the RC1 is out.

[ReneSchumacher]

Since I encountered the issue during a customer engagement and waiting for Azure DevOps Server 2020 is no option in our case, here's a workaround that seems to work for all .NET Core versions:

Prerequisites:

• Install NodeJS
• Install Git for Windows or a Git client of your choice for your platform

Build the latest version of the DotNetCoreCLIV2 task:

1. Clone the repo from https://github.com/microsoft/azure-pipelines-tasks. I recommend doing a shallow fetch as you don't need all the history (e.g., git clone --depth 10 https://github.com/microsoft/azure-pipelines-tasks.git)
2. Restore packages by running npm install
3. Build the DotnetCoreCLIV2 task with the following command: node make.js build --task DotNetCoreCLIV2
   Note: I ran into an issue when the process tried to download 7zip. If you encounter the same problem, you can fix it with a small update to node_modules/sync-request/index.js. Change line 75 from

     var res = spawnSync(command, args, {input: req + '\r\n'});

   to

     var res = spawnSync(command, args, {input: req + '\r\n', maxBuffer: 1000 * 1000 * 100});

Publish the task to your Azure DevOps collections:

4. Ensure that you can run PowerShell scripts without restrictions by opening PowerShell and running

   Set-ExecutionPolicy Unrestricted -Scope Process -Force

5. Run the script New-TaskInstallScript.ps1 from the folder docs/pinToTaskVersion in the cloned repo to upload the task into each collection in your on-prem server (run it from the repository root folder):

   ./docs/pinToTaskVersion/New-TaskInstallScript.ps1 -CollectionUrl https://myAzureDevOpsServer/DefaultCollection -taskDir _build/Tasks/DotNetCoreCLIV2

These steps update the dotnet task that comes with the Azure DevOps server installation to the latest version. Any future update overwrites the manually installed task so there's no risk of applying this workaround.

[timmkrause]

@ReneSchumacher Did you receive a response from the product group regarding the overwrite during an update.

[ReneSchumacher]

@timmkrause Sorry for not updating my comment properly. The PG was unsure so I ran my own tests and can confirm that future updates are indeed applied. I'll also update my original comment so all information is in one place.

[timmkrause]

@ReneSchumacher The code on master has evolved and step 3 from your approach is not applicable anymore.

It seems line 75 needs to be changed from

var res = spawnSync(command, args, {input: req + '\r\n'});

to

var res = spawnSync(command, args, {input: req + '\r\n', maxBuffer: 1000 * 1000 * 100});

Works on my machine certified.

Do you think it makes sense to edit your solution/comment?

[NKnusperer]

This issue has been resolved with Azure DevOps Server 2020 and the task now works as expected:

- task: DotNetCoreCLI@2
  displayName: 'dotnet restore'
  inputs:
    command: restore
    projects: Projects.sln
    vstsFeed: 'my-artifacts-feed'

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days