Package Namespaces Proposal

[JonDouglas]

Introduction of a feature concept called Package Namespaces which allows a developer to include or exclude package IDs by specifying package namespaces on their package source(s).

🎉📦Please provide your comments, concerns, and feedback within this PR. 📦🎉

Rendered

[Tyrrrz]

How will it work for packages that are not part of a "namespace"? I see only Microsoft, System, and maybe a few big packages falling under that list. Would the end user have to manually include each of those packages as a namespace?

More concretely, it would be nice to see the linked proposal include scenarios with non-Microsoft packages. If I understand correctly, this feature will have impact on the broad community of package maintainers, not just Microsoft (assuming the end user has this feature enabled).

[KalleOlaviNiemitalo]

Or rather, one package source is allowed per namespace.

[JonDouglas]

That's a great way to summarize it.

[KalleOlaviNiemitalo]

I wanted to point out the error in "one namespace is allowed per package source", rather than summarize.

[JonDouglas]

Thanks! We'll fix it.

[zdavidsen]

This is still very opaque to me. Does strict mode enforce that each <packageSource> can only contain a single <namespace>? Does it mean that I can only include <namespace id="Nuget.*" /> inside a single <packageSource>? Or is it saying that namespaced packages cannot alias to multiple sources, e.g. the following would cause errors for Nuget.Internal.Package because it matches namespaces in multiple different sources?

<packageSource name="nuget.org">
    <namespace id="Nuget.*" />
</packageSource>
<packageSource name="InternalFeed">
    <namespace id="Nuget.Internal.*" />
</packageSource>

[JonDouglas]

Let me update this shortly to help clarify. It means that only one unique package ID can be defined per package source.

In your conflicting scenario, there is both Nuget.* & Nuget.Internal.*. However since you have Nuget.Internal.*, that takes precedence for the specific Internal.* scope & would therefore be resolved on the internal feed rather than the globbed nuget.org source. Everything outside the Internal.* scope would be resolved on nuget.org.

[mmitche]

I posted some comments above https://github.com/NuGet/Home/pull/10660/files#r597213005, but I don't think this will be easy to turn on if it's one package ID per package source. Basically any kind of organization that does pre-release builds on one set of feeds and releases on another will always have id overlap. The same packages will appear on both feeds and it's valid to use them on both feeds. This gets especially important in any kind of scenario where pre-release packages are different than released ones. For instance, the .NET 6 packages built day to day are unsigned, but those we release are signed, pushed to nuget.org, then deleted from our feeds so that nuget.org is the source of truth. This gets even more likely if you had more than one internal feed with the same packages. Perhaps legacy packages live on one, while new packages are pushed to another, etc. etc.

Could that allow for dependency confusion? Possibly, or at least there could be a restore that pulls from location X one time and location Y another. But when creating the namespaces config, the user is making a trust decision about namespaces on specific feeds. To me, strict should be a way of saying that a trust decision has been made about all referenced package IDs on all input sources.

If single namespace->single feed is still an important pivot, could we add an additional mode? Example

• fullySpecified - Any package id must be in one or more matching namespace declarations
• singleSource - No package id may match more than one feed (based on precedence rules)

[JonDouglas]

@Tyrrrz Right now this feature is largely opt-in. Meaning that if you don't do anything today, you have the same experience.

For the non-Microsoft or namespace scenario, you would use the package ID syntax in full to provide the package into your allowed namespaces. This is a bit tedious as mentioned in the proposal, but tooling would be provided to automatically add the namespace when installing through various methods (visual studio, CLI, etc) if you've already opted-in to the feature.

[mnmr]

Can I ask why package signing isn't an integral part of a proposal such as this?

It just seems that "trusting packages from vendor x" would be easier to configure (trust this signing key), while at the same time being configurable independently of the package source. Signature verification would also work better for any author with multiple packages that aren't sharing a namespace prefix (e.g. Marc Gravell's protobuf-net and fast-member).

[JonDouglas]

@mnmr There's some thoughts here that may share my perspective: https://haacked.com/archive/2019/04/03/nuget-package-signing/ & https://haacked.com/archive/2019/05/10/friend-signing-packages/

There already exists many features for this finer level of control & even a recent proposal to bring these to dotnet CLI

So while you can still go trust an author's certificate, you can also scope your namespaces if those packages aren't signed/repo signed only, but you still trust them. This may suffer from the same problem where you have to add every single person you trust to your trusted-signers list, but rather you have convenience at the namespace level with globs where you can define a trust relationship. At least that's my thinking. @nkolev92 may have more thoughts here too.

[nkolev92]

Yes, I agree with @JonDouglas.

trusted-signers is already a feature that's available that you can use to achieve similar results as namespaces, but in order to rely on signatures you would need every package signed.
We did consider the merits of it when designing namespaces.

trusted-signers doesn't have the privacy element that namespaces does. Your in house package ids don't have to looked up on feeds you don't own.
Package existence is looked up with a package id and version, so NuGet would need to download the package from potentially multiple sources to determine which one matches the restrictions. This will affect performance negatively compared to namespaces which should help performance in most scenarios.

[mnmr]

@JonDouglas Thanks for the links - it's hard to disagree with any of what Phil writes.

@nkolev92 From my understanding NuGet already signs all packages. Instead of using a single key for this, could it not use a key per account holder? This would remove the obstacle of getting package authors to do the signing.

As for privacy and performance, those are probably true concerns, but my (completely uninformed) gut feel tells me that any performance issues can be resolved (and if not, at least be reduced to only affect initial restores).

[zkat]

I'm not a fan of "strict" here. It doesn't really tell me much about what that means, and it's already a bit of a confusing feature to think about.

[nkolev92]

Do you have a recommendation here?

signatureValidationMode has require and allow which I don't love either.
strict matches a feature that the compiler has.

[zkat]

Maybe rename the option away from signatureValidationMode (which doesn't tell us much), to something more descriptive like multipleNameSpaceSources, and then it can just be true and false?

[mmitche]

I also think it's a bit confusing, and perhaps not all that useful as it's spec'd out here? The issue is that I think it would be hard to turn on in any case where multiple feeds are present. Consider .NET. We build plenty of pre-release packages internally, and also put them on nuget.org. Thus I would want to put Microsoft.* namespaces on both my nuget.org and dotnet6 feed, and that would be completely valid. It sounds like strict mode would disable that ability?

Perhaps instead, the strict mode would specify that all package id's must be accounted for? E.g. If I have 3 feeds:

• nuget.org
• dotnet-eng
• dotnet6

Let's say I said that NewtonSoft.* comes from nuget.org and Microsoft.* comes from dotnet-eng or dotnet6. If I then try to reference Xunit.Whatever, the restore would fail in strict mode because that package ID/namespace was not assigned to any feed.

[nkolev92]

Should strict mode imply both, so either a 1-1 relationship between package id and source, or a 1-n explicit relationship between package id & source.

I understand the special scenario for the .NET repos, and that's certainly a big consideration, but we also expect the more common scenario to be 1 private feed & multiple additional feeds.

[mmitche]

Should strict mode imply both, so either a 1-1 relationship between package id and source, or a 1-n explicit relationship between package id & source.

How would that be implemented? I'm not sure I follow.

I understand the special scenario for the .NET repos, and that's certainly a big consideration, but we also expect the more common scenario to be 1 private feed & multiple additional feeds.

I don't think .NET's scenario is all that special, but can we get some sense of that from the community? I think you're likely to see ID overlap in any case where pre-release packages are placed on a feed and verified, prior to publishing to nuget.org.

[nkolev92]

How would that be implemented? I'm not sure I follow.

Maybe some examples clarify it:

The packages are:

Microsoft.* and NuGet.*.

• Case 1*

Source 1: Microsoft.*
Source 2: Microsoft., NuGet.

This can be a valid strict mode scenario.

• Case 2:*

Source 1: Microsoft.*
Source 2: *

This a valid strict mode scenario.

Case 3:
Source 1: NuGet.*
Source 2: *
Source 3: *

This is invalid because you are not explicitly opting into using both sources for the specific packages.

[zkat]

Yes.

If you're using this feature, I believe the general intention is to limit something to a single prefix. As such, I think the "default by secure" approach is a better choice.

[ghost]

Seconded. At a minimum, nuget should reject restoring any package that's present on multiple feeds unless whitelisted, to cap the current security defect.

[JVimes]

nuget should reject restoring any package that's present on multiple feeds unless whitelisted

I love this. Very easy threat fix. I assume whitelisting feeds (like private beta vs prod) not just packages.

[tebeco]

or nuget should implement the equivalent of @scope/package like npm
and assign a feed to an @scope
default @scope could be associated to the user or org pushing the package for backward compat and enrich existing package so there's no breaking change

that would address what's mentionned above and at the same time facilitate multi feed

on top of that nuget v7 could make the use of scope mandatory

which would also fix the "name reservation" with something that looks way more clear especially even doing mono repo of multi library

it would also make easier the validation / installation of certitifacte for on premise nuget proxy feed or storage
check how .npmrc allow to whitelist cert thumbprint per source for example
=> in repo certificate white list (easy for docker container build or computer where you can't install certificate)

[nkolev92]

@nkolev92 From my understanding NuGet already signs all packages. Instead of using a single key for this, could it not use a key per account holder? This would remove the obstacle of getting package authors to do the signing.

nuget.org has a solution for the scenario you are articulating actually. Solved in a very similar way as you are proposing in fact! Refer to the blog post here: https://devblogs.microsoft.com/nuget/lock-down-your-dependencies-using-configurable-trust-policies/.
The variant you are looking for is trusted owners, https://docs.microsoft.com/en-us/nuget/reference/cli-reference/cli-ref-trusted-signers#options-for-add-based-on-a-service-index.

You can trust specific nuget.org users, so while the key is the same, there is some metadata on which the client can already pivot.

The challenges that I am referring to are more relevant to dedicated/private feeds which might not have all packages signed.

[KalleOlaviNiemitalo]

I understand from #9611 that package restore in Visual Studio uses a bundled version of NuGet, and upgrading that will require a new version of Visual Studio as well. Is the package namespaces feature planned for Visual Studio 2019 (Dev16), only for later versions, or also for earlier versions that are still supported?

On a build server, if the installed versions of Visual Studio Build Tools do not support this feature, but NuGet.exe and .NET SDK do, then how should I script the package restore to make sure that package namespaces are respected even in .NET Framework projects that use PackageReference?

[kartheekp-ms]

I have following questions to start with.

• What happens if <packageNamespaces> is defined on a source that is disabled using <disabledpackagesources>?
• Does <packageNamespaces> element supports </clear> tag like <packageSources>?
• Let's say, If I have a NuGet.Config defined with <packageNamespaces> in the solution-level directory and project-level directory. Will <packageNamespaces> settings be combined from all configuration files into a single collection like <packageSources>?
  https://docs.microsoft.com/en-us/nuget/consume-packages/configuring-nuget-behavior#how-settings-are-applied

[nkolev92]

What happens if is defined on a source that is disabled using ?

It's ignored. Similar to how PackageSourceCredentials are handled.

Does element supports tag like ?

Afaik, all NuGet.config sections support clear. This will be no different.

Let's say, If I have a NuGet.Config defined with in the solution-level directory and project-level directory. Will settings be combined from all configuration files into a single collection like ?

Namespaces will be merged, similarly to sources and others.
In the technical explanation scenario 3 covers the not recommended experience of having multiple configs to control namespaces.
When applying configurations, there are scopes (depending on how you run the command and the tooling you are using). That's not changing today.

[mmitche]

There's a misconception about open source dependencies being more secure which is known as the Linus' Law, which allowing the world an opportunity at your software supply chain is posing a greater risk as more new packages are created everyday

Nit: wording

[tebeco]

As a general comment, as discussed/asked on Twitter it would be very nice to have it available too as "MsBuild First" if possible
This feature see to move around the "feed" notion as we can see another ongoing proposal for feed here rconard/central-nuget-management-planning#1
I this that's a brilliant idea as it let us use

• Env Variable
• Condition
• Logic
  ...

The current "dsl" proposal to add to the nuget.config would probably be very similar in MsBuild tarm, but it would integrate with the all ecosystem.

[ghost]

Can I ask why package signing isn't an integral part of a proposal such as this?

Seconded. I only just learned about package signing, but it seems as though repository level signing would already solve the use case here for most legitimate uses. Naming collisions are pretty rare (though not unheard of; I know at least two unrelated packages that go by the name "NuProj"), and my team's scenarios for using a package namespace feature would be entirely to lock down which author's we'll accept packages from: precisely the use case of package signing.

That said, our immediate problem is that our internal repositories don't all support repository signing. This feature would greatly help us secure our package supply chain until our internal repositories catch up and implement repository signing, or internal package authors catch up and sign their packages.

[nkolev92]

@tebeco

The current "dsl" proposal to add to the nuget.config would probably be very similar in MsBuild tarm, but it would integrate with the all ecosystem.

We thought that the MSBuild driven approach deserves consideration on its own, separate from namespaces.
We did consider how the syntax would look in MSBuild driven source declaration.

As with anything MSBuild, configuration is project specific but so many other components are repo wide, so we'd have to advise caution when using it :)

[tebeco]

As with anything MSBuild, configuration is project specific but so many other components are repo wide, so we'd have to advise caution when using it :)

I forgot that "small" detail.
Do you know if it's a no go or if it's still possible to achieve ? Kinda like the "Singletion for MSBuild" ?

[nkolev92]

I think it requires more care when working with it.
We'd certainly provide guidance if it's added.

[ghost]

Instead of scoping by package name, could you add an option to scope by package authors? I'd much prefer to whitelist the authors I'd expect from each feed:

    <packagesource key="nuget.org">
        <!-- Whitelist the authors we actually use. We already install some of their packages, so we trust them not to be malicious. -->
        <author id="Microsoft" />
        <author id="Newtonsoft" />
        <author id="dotnetframework" />
    </packageSource>
    <packagesource key="contoso">
        <!-- Contoso is an internal nuget feed -- we trust everything from there implicitly. -->
        <author id="*" />
    </packageSource>

[ghost]

I'm not familiar enough with the v3 http protocol, but does the package feed expose the registered author of the package? If so, we'd much prefer the option to whitelist by author instead of by namespace.

Rationale: With the PackageReference mechanism and .g.props auto-importing certain named files in the package, package authors are granted permission to run arbitrary code during the user's build. Given that, a given author could easily impersonate other packages by rewriting those packages on disk.

When whitelisting a namespace, I'm effectively granting permission to all authors that have the ability to publish into that namespace to impersonate any package. So why not cut out the intermediary and list authors specifically, if feasible?

Secondly, users are likely to use wildcards without first checking that it properly is scoped to an author. Do I know for certain that nuget.org will only let NewtonSoft publish to the NewtonSoft package namespace? I know many users that would jump to the assumption that nuget.org would implement that restriction. Whitelisting authors would remove that temptation -- e.g. whitelisting NewtonSoft would only allow NewtonSoft authored packages onto the user's machine; even if a malicious entity registered a package with a typosquat - e.g.
"NewtonSoft.Jsno", the user's nuget config with an author whitelist would reject the package as that malicious entity is unlikely to be listed on the user's author whitelist.

[JonDouglas]

There's an interesting relationship here:

ID prefixes have a 1 to many relationship with owners.

An allow list of owners/maintainers is interesting. You may establish your trust relationship purely on an owner/maintainer's name at the risk of knowing that if that entity were compromised, your supply chain would be at a larger risk in terms of packages rather than a handful of namespaces. This is somewhat related to the trusted-signers command to add an author's signature of a package to be trusted, but without the certificate fingerprint.

If people are interested in being able to trust owners/maintainers in addition to the namespaces they possess, please feel free to upvote & drop some comments below as well. We'll look into it! It may be out of scope initially, but shouldn't stop the conversation.

Reference:
https://docs.microsoft.com/en-us/nuget/nuget-org/id-prefix-reservation#id-prefix-reservation-on-nugetorg

[JonDouglas]

@aaronla-ms The trust by author is interesting. It gets even more interesting with the idea that NuGet supports both owners and authors. I assume in the case of <author> in your example above, you are referring to the current publisher/maintainer of the package and not necessarily who authored the original package.

The trusted-signers feature currently allows you to trust author signatures in this fashion. There could be merit in specifying the publisher name that you trust in addition to the namespace they may have reserved. We'll discuss this more!

https://docs.microsoft.com/en-us/nuget/reference/cli-reference/cli-ref-trusted-signers#options-for-add-based-on-a-package

[ghost]

@JonDouglas ah, you're spot on, owner or publisher is better. I wouldn't want someone to implicitly trust some library just because Contoso once submitted a pull request -- only if they're a controlling party for that nuget. And if the owner/publisher changed, I'd certainly want to stop implicitly trusting it.

Let me know how the discussion goes!

[JonDouglas]

@nkolev92 @kartheekp-ms @erdembayar Can we get an approval on this PR to merge now that we have requirements for MVP & VS integration?

Thanks!

[vweevers]

I find the structure confusing. Specifically that namespace elements are nested in a packagesource element. This suggests that it defines which namespaces are available in a particular source. Rather than, which source must be used for a particular namespace.

Take the following example:

Click to expand

<packageNamespaces>
    <packagesource key="contoso">
        <namespace id="Contoso.*" />
    </packageSource>
</packageNamespaces>

Could it be inverted instead? E.g.:

<packageNamespaces>
    <namespace id="Contoso.*">
        <packagesource key="contoso" />
    </namespace>
</packageNamespaces>

This feels more natural to me. It reads as "any package ID matching Contoso.* must be downloaded from (or uploaded to) the contoso source".

[vweevers]

In addition, I would prefer a simple "strict by default" behavior, without any modes that make it more difficult for third-party tooling to copy or predict nuget's behavior (not to mention for users to understand it). In a nutshell: use the order as defined by the user in <packageNamespaces> and require fallbacks to be explicitly defined. First match wins and any source that's not present within <packageNamespaces> will simply not be used. For example:

Click to expand

<packageNamespaces>
    <namespace id="Contoso.*">
        <packagesource key="contoso" />
    </namespace>
    <namespace id="*">
        <packagesource key="nuget" />
        <packagesource key="dotnet-public" />
    </namespace>
</packageNamespaces>

[KalleOlaviNiemitalo]

@vweevers, how would you define the order if a <packageNamespaces> element exists in more than one applicable NuGet configuration file? (Docs: Common NuGet configurations)

[vweevers]

@KalleOlaviNiemitalo I see. If <packageNamespaces> elements will be merged - #10660 (comment) - then we can't rely on order. So alternatively, longest prefix wins and only allow three namespace patterns: exact package ID, <prefix>.* (same as ID Prefix Reservation) or a *.

In either case (packagesource > namespace or namespace > packagesource) I'm not sure how "merging" works though. Is it a deep merge?

[JonDouglas]

Hi all,

We're still listening to all feedback for this proposal, but we have begun development & will be sharing iterations with the world shortly. Thus I am moving forward to merge this PR as approved 7 days ago & adjusted to the feedback we've heard thus far.

The syntax is one we went to ensure we would not override older client versions of nuget.config and wanted to find a mixture that was human-readable & understandable at a glance when one would open up the file. If there is a compelling argument to be made to change the syntax as it's presented in the spec, please feel free to make a case here in the next few days before we merge.

The primary reason the syntax is the way it is today mostly relates around previous NuGet feature designs, vertical real estate, readability of configuration files, ease of merging, and not wanting to have users duplicate elements when they may only have a handful of them (i.e. package sources).

We also toyed around with the idea of having users define the sources on the namespace element itself which would be similar syntax as to another proposal found here: https://github.com/NuGet/Home/blob/source-pinning/proposed/2021/SourcePinning.md which didn't fully fit the criteria of the problem we're trying to solve although it was well received.

Please continue to share thoughts around the syntax as it helps us build the best feature possible!

[nkolev92]

@vweevers

As @KalleOlaviNiemitalo pointed out, we chose to go the per source direction because it meshes better with the config precedence concept.

Note that the configs are how the data is represented, we can have CLI commands and UI that may be able to display the information in different ways.

[tebeco]

I would very much love to have all that in a Directory.Feed.props

getting rid of nuget.config would help having

• automation
• reusable msbuild
• mono repo handling with import
• log / trace for diagnostics in binlog
• ...

we already witness the removal from package.config to MsBuild / csproj.
I'm really hoping to see the same to kill nuget.config and .sln file

[ghost]

@tebeco I like the thought, putting it all in one place. However, the limitation with Directory.Feed.props is that is doesn't allow for securing:

• packages restored by invoking nuget.exe
• SDK packages
• dotnet tools

None of the above can parse msbuild props files in a reasonable fashion. I support the notion, (filed an issue, now closed, to that effect), but the feedback from the msbuild team is that it's just not practical at this time, would have to be pursued by nuget, not msbuild.

[tebeco]

oO no that's exactly why it should exists

the fact that dotnet tool absolutely needs the nuget.config and don't even load Directory.Build.props create a mess in your repo the moment you start having both already today
I see that as an opportunity to start standardizing the tool

same goes for nuget.exe, you get a valid point
as a part of the code is share between dotnet / dotnet nuget and nuget it means the fix should be in that common place

I see these point as an excellent reason to make these changes ... fix all at once

[tebeco]

I personally think that "they don't do it why would we" is what slow down evolution of echo system.

We see the same in human society ...
"but they did before so i can also do "

taking the lead on that would drive other forward in a non-breaking way and open possibility of re-use rather than locking down everyone to a an old behavior.

for example ... IIRC what i've read, a part of the msbuild team is whiling to replace sln file with msbuild proj ... but VsTeam is not ready but of the work to do... so nothing move.

I truly hope that the change from package.config to PackageReference should be seen as an example to "move forward"