Package Source Mapping: Difficulty using it on Azure DevOps CI because it ends up needing feed- prefixes in the source mapping configuration.

[cheesi]

Maintainer Edit

The Azure DevOps problems occur when the DotNetCoreCLI is used.
The recommendation is to stop using the DotNetCoreCLI task and use NuGetAuthenticate for authentication and then use a Command Line Task to invoke dotnet.

See: #11406 (comment)

NuGet Product(s) Involved

NuGet.exe, dotnet.exe

The Elevator Pitch

We tried to adapt the new package source mapping feature today, and found one case which is missing.
If I understand it correctly, the main reason behind the source mapping is, to accidentally restore internal nuget packages from nuget.org (ofc not the whole reason, but for me at least the biggest part of the cake).

Let's try this test setup from your blog post:

<packageSourceMapping>
  <packageSource key="nuget.org">
    <package pattern="*" />
  </packageSource>
  <packageSource key="contoso.com">
    <package pattern="Contoso.*" />
  </packageSource>
</packageSourceMapping>

This basically specifies, that everything can be resolved via nuget.org, but for Contoso.* contoso.com should be used. Fine. But based from the log, it is still parsed as Package source mapping matches found for package ID 'PACKAGE' are: 'nuget.org,feed-orga-project'.

Based on the documentation, the closest match is used, therefore contoso.com should be used. But what happens, when contoso.com is not available (for whatever reason)? Will it then fall back to nuget.org? Definitely not the thing I would like to configure.

I guess it would be nice if there would be an exclude feature. Because I don't really care, if nuget packages are restored via nuget.org or an internal feed, but I would like to make sure, that internal packages can only be restored by the internal feed.

Or am I missing anything?

Additional Context and Details

No response

[nkolev92]

Thanks for trying out our new feature @cheesi!

Based on the documentation, the closest match is used, therefore contoso.com should be used. But what happens, when contoso.com is not available (for whatever reason)? Will it then fall back to nuget.org? Definitely not the thing I would like to configure.

Feed availability does not affect package source mapping, so whether contoso is available or not, there will be no fallback to NuGet.org.

But based from the log, it is still parsed as Package source mapping matches found for package ID 'PACKAGE' are: 'nuget.org,feed-orga-project'.

This really depends on the source mapping configuration.
Would you be able to share the config (consider Find-And-Replace internal names you don't want to share.

Best match should be the only source used. If you're seeing anything different, it's probably a bug.

I guess it would be nice if there would be an exclude feature. Because I don't really care, if nuget packages are restored via nuget.org or an internal feed, but I would like to make sure, that internal packages can only be restored by the internal feed.

Hopefully my answers above clarify the expectations. Do you still see a need for an exclude option?

[cheesi]

If that's the case, then no, I don't see a reason for an exclude feature.
But then there are definitely some bugs.

This really depends on the source mapping configuration.
Would you be able to share the config (consider Find-And-Replace internal names you don't want to share.

Yeah it's the config from above (just changed to fit our pattern):

  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="orga-project">
      <package pattern="ORGA.*" />
    </packageSource>
  </packageSourceMapping>

I attached the log file, as it quite big: source_mapping.log

I can see two issues from this log file:

1. Line 4418: Package source mapping matches found for package ID 'ORGA.Project.Service.Domain' are: 'nuget.org,feed-ORGA-project'. This is a little bit irritating as this particular thing is not a NuGet package to restore, but instead a project in the solution. (That's why I was a little bit confused regarding which feeds are used, because in the other repo, where we tested it yesterday, the projects mentioned in the log where the actual produced NuGet packages, I just mixed up the repos). So to make it short: I don't think there should be "Package source mapping matches found for package ID" logs for project references, right?
2. Package source mapping worked for our first repository, because we don't consume any internal NuGet packages, as that's the repository where we actually produce them. However, in the second repository, where we already consume those, the restore fails, because package source mapping fails to find a match as seen on line 8125: Package source mapping match not found for package ID 'ORGA.Project.SharedKernel'. The config is mentioned above (with our identifiers of course). I'm 100% sure there is no typo in the key or the package. We already purged our changes two times now, and rewrote them, no success. So something seems to be wrong here.

[cheesi]

We were able to figure it out. After some time, I thought, why does the log always say feed-orga-project?
I then took a blind shot with this config:

  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="orga-project" value="https://pkgs.dev.azure.com/orga-project/_packaging/orga-project/nuget/v3/index.json" />
  </packageSources>

  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="feed-orga-project">
      <package pattern="ORGA.*" />
    </packageSource>
  </packageSourceMapping>

So I had to prefix the packageSource key with a feed-

And now the NuGet restore works:

But that's definitely not how it's documented. And also quite confusing to have different keys.

[glatzert]

We are seeing this too, the workaround from @cheesi helped.
I'd assume it's a bug in the tooling, that creates the temp nuget.config for the build.

I changed my packageSoruceMapping to something like this:

<packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="orga-project" value="https://pkgs.dev.azure.com/orga-project/_packaging/orga-project/nuget/v3/index.json" />
  </packageSources>

  <packageSourceMapping>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
    <packageSource key="orga-project">
      <package pattern="ORGA.*" />
    </packageSource>
    <packageSource key="feed-orga-project">
      <package pattern="ORGA.*" />
    </packageSource>
  </packageSourceMapping>

[GuySerg]

We also were affected by this behavior.

We created the <packageSourceMapping> section in the NuGet.config according to the docs, but also were faced with the message Package source mapping matches found for package ID 'PACKAGE' are: nuget.org,feed-orga-project during dotnet restore on Azure DevOps.

Adding prefix feed- to the packageSource key resolved the issue. Many thanks, @cheesi

Rider 2021.3 EAP 9 works well with the feed- prefix and without it.

[nkolev92]

Thanks for your comments folks.

Checking right now.

I'd be surprised if feed did anything, as we're only doing a string comparison of the keys.

[nkolev92]

Line 4418: Package source mapping matches found for package ID 'ORGA.Project.Service.Domain' are: 'nuget.org,feed-ORGA-project'. This is a little bit irritating as this particular thing is not a NuGet package to restore, but instead a project in the solution. (That's why I was a little bit confused regarding which feeds are used, because in the other repo, where we tested it yesterday, the projects mentioned in the log where the actual produced NuGet packages, I just mixed up the repos). So to make it short: I don't think there should be "Package source mapping matches found for package ID" logs for project references, right?

You are correct, there should be no lines for projects. I have created a separate issue to track this behavior. #11413

But that's definitely not how it's documented. And also quite confusing to have different keys.

That's great analysis @cheesi!

How is your config generated?
as @glatzert mentioned,

I'd assume it's a bug in the tooling, that creates the temp nuget.config for the build.

we need to figure out how the nuget.config was generate and how that feed was added there.

Are you using some azure devops tasks to add sources?

@SergGuy, @glatzert, @cheesi

Apologies for potentially asking the same questions, but if I understand correctly, the only problems you're running into are on the CI?

[nkolev92]

Making progress, seing that there's some magic happening in some azure pipeline tasks:

https://github.com/microsoft/azure-pipelines-tasks/search?q=feed-

If anyone can let me know which tasks they're using that are Nuget related that'd be great!

[nkolev92]

Hey all,

I have made some progress and figured out that Azure DevOps tasks rewrite the configs and add feed- to the nuget.config.

I have filled an issue: microsoft/azure-pipelines-tasks#15542

I'll try to find someone from that end to confirm, but looks like most of the DotnetCoreCLiV2 tasks are affected.

[nkolev92]

I have retitled the issue.

I'd appreciate if any of you could confirm my observations and provide info about the NuGet related tasks you have on your CI.

If it's at all an option, please trying adding the NuGetAuthenticate task before any of the NuGet tasks you may have.

There's a chance that the whole feed- thing doesn't happen when the NuGetAuthenticate task is used, https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/nuget-authenticate?view=azure-devops

[GuySerg]

@nkolev92 I can confirm we are using the DotNetCoreCLI@2 task on Azure DevOps. We are building a solution with only .NET6 projects.

We do not use any NuGet tasks during our CI. There was no need so far. dotnet cli worked just fine.

On my dev machine, I wasn't required to add feed- to the key to get packages from our private feed. All worked as it is in docs. But strangely with the feed- prefix also everything is working.

[glatzert]

I second this. Am 24.11.2021 21:38 schrieb SergGuy ***@***.***>: @nkolev92<https://github.com/nkolev92> I can confirm we are using the ***@***.*** task on Azure DevOps. We are building a solution with only .NET6 projects. We do not use any NuGet tasks during our CI. There was no need so far. dotnet cli worked just fine. On my dev machine, I wasn't required to add feed- to the key to get packages from our private feed. All worked as it is in docs. But strangely with the feed- prefix also everything is working. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#11406 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACNPDCFT32UXAII2K4UPV5DUNVEKRANCNFSM5IR5EMMQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nkolev92]

Thank you for the quick replies folks!

I'll be updating this thread as we make progress with the AzureDevOps tasks, but given that it's the holidays here in the US, it'll likely take a few days.

Regarding feed- working locally, note that it will work if you duplicate the configuration, please refer to the Note here: https://docs.microsoft.com/en-us/nuget/consume-packages/package-source-mapping#setting-default-sources, that talks about packages that are already available on your machine.

[cheesi]

Hey, thanks for creating all the issues and renaming the current one - definitely makes sense.

We are already using NuGetAuthenticate in our CI pipeline:

  - task: NuGetAuthenticate@0
    displayName: 'Authenticate with NuGet'

  - task: UseDotNet@2
    displayName: 'Install .NET 6'
    inputs:
      version: '6.0.x'

  - task: DotNetCoreCLI@2
    displayName: 'NuGet restore'
    inputs:
      command: 'restore'
      feedsToUse: config
      nugetConfigPath: blueprints/service/content/NuGet.config
      projects: 'blueprints/service/**/*.sln'

[nkolev92]

@cheesi
Just to confirm, you're having issues even with NuGetAuthenticate there?

[cheesi]

Yes, I can confirm that.

[magleaso]

Hey, thanks for creating all the issues and renaming the current one - definitely makes sense.

We are already using NuGetAuthenticate in our CI pipeline:

  - task: NuGetAuthenticate@0
    displayName: 'Authenticate with NuGet'

  - task: UseDotNet@2
    displayName: 'Install .NET 6'
    inputs:
      version: '6.0.x'

  - task: DotNetCoreCLI@2
    displayName: 'NuGet restore'
    inputs:
      command: 'restore'
      feedsToUse: config
      nugetConfigPath: blueprints/service/content/NuGet.config
      projects: 'blueprints/service/**/*.sln'

This won't work because the DotNetCoreCLI task handles authentication as well as re-implements dotnet client behavior. When using the _Authenticate tasks, they're intended to handle authentication and to be followed by a simple command line task to invoke the standard CLI client. Using the authenticate task and then the heavy weight DotNetCoreCLI@2 task is redundant.

[cheesi]

@magleaso
I just checked our history, the NuGetAuthenticate was added, as our docker pipeline was set up (where we definitely need it). I guess it was added for "just in case", as I can remember restoring private nuget feeds within the container was a little challenge.

But thanks for the feedback, I did a fast test and the restore indeed works without the NuGetAuthenticate. I will create a task to remove it.

[magleaso]

That's the opposite of what I'd recommend. The Authenticate task is newer and less prone to bugs. I would recommend removing the DotNetCoreCLI task and replacing it with a command line task that invokes dotnet restore

[cheesi]

Okay then what is the purpose of DotNetCoreCLI in general, when it is recommended to use command line tasks instead?
Or is it just for the dotnet restore?

[magleaso]

DotNetCoreCLI is an older task that handles authentication and client behavior. It has been subsumed by the authenticate task because the authenticate task allows you to directly use the dotnet clients that everyone else on the internet is using, so you don't have to deal with situations like the one above where there is a deviation in the behavior of the DotNetCoreCLI task from the standard dotnet client.

[cheesi]

I see.

So switching to a command line task should solve the problem with the package source mapping, as described above?

[magleaso]

Yes

[cheesi]

Thank you very much for the help, we will try this out as soon as possible.

[nkolev92]

I edited the main issue body to include the recommendation.

We're working on updating the docs with this guidance as well.

[nkolev92]

Just an update here.

Thanks to your help we have confirmed the problem is related to the azure devops tasks, as noted in microsoft/azure-pipelines-tasks#15542.

We're going to close the issue on NuGet side, as there is no action from this part of the tooling.
If you're hitting this issue, please refer to the recommendation #11406 (comment) to use NuGetAuthenticate and a commandline task to call restore and any other operations in a similar way that you would on your own machine.

In addition to that, we're going to add a docs clarification about the issues you may run into if you use these tasks.

Issue: https://github.com/NuGet/docs.microsoft.com-nuget/issues/2610 and a PR in progress: NuGet/docs.microsoft.com-nuget#2611.

[david-driscoll]

@nkolev92 FYI I just ran into this today (2 and a half years later) and the azure devops bit isn't fixed yet, just FYI. :)

[cliedeman]

Still an issue now.

Specific to azure devops.

I am using NuGetAuthenticate which is the recomended approach for pipelines according to the docs.

dotnet restore cant download the packages but the restore pipeline task works but runs into the above issue

[nkolev92]

@cliedeman Please don't use the restore task.

Refer to microsoft/azure-pipelines-tasks#15542 (comment).
Please don't use the DotnetCoreCLI tasks.

Copying the suggestion here for others:

Replace

- task: DotNetCoreCLI@2
  displayName: dotnet restore
  inputs:
    command: 'restore'
    restoreArguments: '--nologo'
    feedsToUse: 'config'
    nugetConfigPath: 'Nuget.config'
    verbosityRestore: 'Normal'

with

- task: NuGetAuthenticate@1
  displayName: 'Authenticate with NuGet'

- task: PowerShell@2
  displayName: 'dotnet restore'
  inputs:
    targetType: 'inline'
    script: 'dotnet restore --nologo'

[cliedeman]

@nkolev92

I reverted to using the dotnet cli and its working now  🥴.
Not sure exactly what changed but glad either way