NuGet pack ignores nuspec versions [unwanted version normalization?]

[timotei]

Hello,

It seems latest NuGet pack ignores the version set in the nuspec (or via the -Version override flag) and the dependencies' versions, by normalising them, even if nobody asked for it :(

It seems this was "fixed" for this #2039, but to me this seems a super counter-intuitive change, that breaks (at least for us) the daily workflows.

For the meantime, we had to do a workaround to "fix" the versions in the generated nupkg. I'm hoping that this can be fixed either by removing this weird logic, or allowing the nuget pack command to generate nuget package which contains the same versions as in the nuspec.

I'm using NuGet Version: 3.4.4.1321.

With this nuspec:

<?xml version="1.0" encoding="UTF-8"?><package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd">
  <metadata>
    <authors>author</authors>
    <version>1.9.0.0-SNAPSHOT</version>
    <id>mylib</id>
    <dependencies>
      <dependency id="dependency" version="3.16.0.0"/>
    </dependencies>
    <description>mylib</description>
  </metadata>
</package>

executing nuget pack mylib.nuspec -Version 1.9.0.0-SNAPSHOT generates this .nuspec:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>mylib</id>
    <version>1.9.0-SNAPSHOT</version>
    <authors>author</authors>
    <owners>author</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>mylib</description>
    <dependencies>
      <dependency id="dependency" version="3.16.0" />
    </dependencies>
  </metadata>
</package>

As we can see:

• Version is 1.9.0-SNAPSHOT instead of 1.9.0.0-SNAPSHOT (even if we specifically overrode it via the command line) - same happens with a non-SNAPSHOT version
• Even worse, the dependency version is 3.16.0 instead of 3.16.0.0
• Funnily enough, the filename has the full version: mylib.1.9.0.0-SNAPSHOT.nupkg

[emgarten]

@timotei would you share more about your scenarios here? Are you expecting the version string to be an exact match as part of other scripts or tools?

[timotei]

Well, we have a ton of internal libraries that are used throughout the end products (private NuGet repo based on Artifactory). Some scripts/tools generate them directly via the nuget pack, others are using some more legacy build systems and we managed to make some bridge between them (not possible/feasible to migrate them to the newer NuGet-based process).

The issue comes from the fact that newer NuGet automatically normalizes the version, making the interop between the two types of projects hard to almost impossible. Basically, we have interop between Java and .NET libraries (via IKVM), and thus we have a specific format for the versions, which we must respect.

That's why, it's very important (for us and I think in general) that tools that generate stuff, do it in a expected way, based on the .nuspec, because that impacts the entire dependency tree. So, I'd expect the nuget pack step not to change anything that's already set in the nuspec. Or, at least, provide a way to skip that behavior.

[emgarten]

I'd like to understand how this is working, are your scripts resolving nuget dependencies without nuget? Does IKVM consume nupkgs directly? Which parts of your build process read the nupkgs without nuget?

[timotei]

Yeah, up until NuGet 3.4, we had in place a simple mechanism which would translate 1-to-1 from Gradle dependencies, to NuGet packages (which are basically IKVM-compiled assemblies - Java to .NET). We were able to create the packages.config file with the mapped dependencies + versions, then do a nuget restore, use the restored deps in a custom processing step which does the final .NET compilation.

Once NuGet 3.4+ went in, we found the issue that the versions wouldn't be mapped properly, all depending on whether the restored NuGet packages were created with NuGet 3.4+ or not. The thing is, nuget places the restored packages in folder with the full version, not the normalized one, which means, if a package was created with NuGet prior to 3.4 it will have 1.0.0.0, while with NuGet 3.4+, it will be 1.0.0. This means there's suddenly no reliable/deterministic rule for the location of the packages. This issue comes from the fact that nuget pack will normalize the generated version.

[emgarten]

The thing is, nuget places the restored packages in folder with the full version, not the normalized one

This has been fixed in 3.5.0, you can find it here: https://dist.nuget.org/win-x86-commandline/v3.5.0-beta2/NuGet.exe

there's suddenly no reliable/deterministic rule for the location of the packages

With normalization there is only one possible string, making this deterministic now. Paths can be determined upfront without opening every nupkg to check if it is the expected version.

Here is how it works:

1. Versions have three parts to match SemVer
2. If the version contains a 4th non-zero digit, then it is also added since NuGet allowed 4 part versions at the start.
3. Leading zeros are always removed

Examples:
1.0.0.1 -> 1.0.0.1
1.0.0.01 -> 1.0.0.1
1.0 -> 1.0.0
1.0.0.0 -> 1.0.0
1.0.0 -> 1.0.0

[timotei]

Hmm, yeah, but AFAICT, the 3 or 4 digits folder name will depend whether the NuGet package was generated with NuGet <3.4 or NuGet 3.4+, no?

About the normalization part, isn't possible to make nuget pack not normalize the version if explicitly not wanting? Or is the whole 3.4+ Normalized Versions Saga about normalizing the behavior (pun intended) of all NuGet behavior so it uses normalized versions everywhere? (pack + install + restore)

How I see this is, that NuGet is kinda forcing people to use normalized versions even if they don't want it, right :) ?

[emgarten]

Normalizing solves many issues within NuGet. I think your scenario is valid, but less common than the problems normalizing fixes. You can work around this problem by finding the normalized version from the package after packing and using that for your mappings instead of the -Version input. You could also modify the nuspec in the package after pack to un-normalize it.

[timotei]

Does this mean that a PR that enables the NuGet Pack to generate the NuGet package "as is" from the NuSpec (without normalizing versions) would not be accepted into the client?

IIRC, there was once a warning when doing a pack for 4 digits (that the version doesn't satisfy SemVer 2?), but that warning does not show up now, and instead it directly normalizes the version (not sure why normalize it only when ending in '0', since SemVer2 is 3 digits only nevertheless the last 4th digit)

[rrelyea]

@timotei - yes, 3.4.* does this behavior during pack. Earlier versions didn't.
We'd prefer to not add more command line options to not normalize.
Is it possible for you to adapt your mapping file to understand ".0" as the 4th version number will be dropped? We'd prefer to keep everybody normalizing.

[timotei]

Well, in the end we managed to get a post-NuGet pack step that denormalizes the version, to keep the previous behavior. So far it seems to work.

But this is not a very satisfactory solution :( I'd rather have NuGet not change things I specifically set in the nuspec.

[slaneyrw]

This is now causing problems downstream as tools now start upgrading their nuget versions. Octopack now uses 3.4.x which is dropping the 4th part of the version number if it's zero, and breaking templated builds in our TeamCity. Please consider command line options

[emgarten]

@slaneyrw can you explain how TeamCity uses this? Which part of the build breaks due to this change?

[slaneyrw]

Octopack generates an nupkg file which we are expecting to be in a specific format {package name}.{version number}.nupkg that we are publishing to our internal ProGet feed.

The version number is TC's system build number which is a 4 part number. The file name we expect is not present when it has been normalised. The TC configuration template is shared by many individual build projects.

All build projects run off a dedicated nuspec file in each project, this file is patched with TC's system build number before an Octopack build.

[mistachkin]

Today, I received a report from a user about NuGet mangling the version within the nuspec file that gets embedded into the nupkg file. Apparently, this "normalization" completely broke the package restore feature, with an error complaining about the package "not being found" until he manually re-added the final trailing zero.

Linked to ticket [https://system.data.sqlite.org/index.html/info/e9573e2d12387877].

[emgarten]

@mistachkin restore could only break if the user is changing the package and re-packing it using an existing id/version. That scenario is not supported since packages are cached based on the id/version, having different copies of the same id/version will cause other problems.

[mistachkin]

Sorry, but no. The user stated the package would not restore until he manually "un-normalized" the version to add the trailing zero. To be perfectly honest, this is going to be a huge headache. The packages I manage have a total of around 3.3M downloads at this point. If even a small fraction of the users end up with issues like this, that will be a major problem. Looks like the solution is to stick with NuGet version 2.x until this is fixed. Meanwhile, I'll have to look into manually fixing the current production packages. Of course, that won't work either because I cannot ever delete anything to re-upload them with the same version. I'm very frustrated now.

[emgarten]

@mistachkin would you open a new issue for what you are seeing with restore along with some repro steps?

This issue is around package creation, it sounds like you are seeing issues on the consuming side.

[mistachkin]

Respectfully, I disagree. The real issue is that what ended up in the embedded nuspec file was not what I originally authored. Subsequently, this "normalization" ended up breaking package restore (and other features?).

[emgarten]

@mistachkin NuGet supports all both normalized and non-normalized versions during restore. The question is what part of the process is treating the nuspec version as a string instead of a version.

[mistachkin]

That part of the process, whatever it is, by definition, does not "support" the version normalization. Meanwhile, some of the most popular non-Microsoft packages on NuGet.org are effectively broken.

[emgarten]

That part of the process, whatever it is, by definition, does not "support" the version normalization. Meanwhile, some of the most popular non-Microsoft packages on NuGet.org are effectively broken.

@mistachkin please open a new issue for the restore issues you are seeing. Any extra details on how this affects existing packages would be extremely helpful and I would like to understand it better to help you out.

Normalizing the version at pack time only, as this issue covers, only affects new packages from what I can see. Users installing those new packages will have the version form used in the nuspec file from the start, so there will be no changes on the consuming side.

[mistachkin]

I did not discover the issue myself and I do not know the reproduction steps. I will contact the original user and ask them to file a new issue. Done, email sent to original user.

[slaneyrw]

Do we have any direction on what NuGet will be doing in regards to normalization. Are we doing to be able to turn OFF this behaviour during pack ?

[mistachkin]

I agree with @slaneyrw, there needs to be a -verbatim option (or similar) to the NuGet pack command that retains the originally authored nuspec file exactly as authored.

[dougbw]

We are having issues as a result of this too - We use a build counter version number throughout our build, deploy, and test pipelines and having a mismatch causes failures when using the package version in a file system path.

Example scenario:

• TeamCity build is version 1.0.100.0
• Nuget pack creates version 1.0.100
• Deploy application
• Deploy the post-deploy tests to a test server, the package version is part of the target directory path Eg "c:\tests\post-deploy\1.0.100"
• The post-deploy tests fail to run as they use the build version number in the path Eg "c:\tests\post-deploy\1.0.100.0"

If a version is specified in the nuspec file then the output package should contain that exact version. We have had to roll back to a specific older version of NuGet because of this.

The behaviour of using nuget pack should remain the same as previous versions, and a -Normalize switch should be present if you want Nuget to automatically normalize the version number.

[emgarten]

I like the idea of -verbatim or possibly -verbatimversion for pack to keep nuget from trying to help out here.

Making non-normalized opt-in instead of by default makes sense for most users I think. It helps avoid duplicates having in the same folder on disk and the confusing builds that can result from that.

[chris1248]

Nuget version 4.0.0.2283 is also removing leading zeros:
Converting: 27.0001.02220-alpha
to this: 27.1.2220-alpha

Do I have to fork my own repo to fix this? I hope not.
[Edit]
This is happening on versions 3.5, 4.0 and 4.1
Version 3.4 doesn't do the name mangling, so that's what I'm going back to...
Sorry Microsoft, looks like you dropped the ball on this: Forcing normalization when no one wanted it.

[InteXX]

Microsoft's response to this problem they've created and maintained has been a clear lesson on how NOT to behave (toward one's customers).

[rrelyea]

Just asked on the powershell issue:
"Is there somebody from powershell gallery that could engage with the nuget team to discuss the variety of options to fix this?"

[StingyJack]

@rrelyea - "put it back the way it was" should cover the necessary work for all of those affected instead of just for one group of your choosing. Then you can add your nuget.org specific restrictions

[InteXX]

@rrelyea

@StingyJack is correct.

[txwizard]

@InteXX , I may have misinterpreted the labels shown in the upper right corner of the issue page, where I see a Priority 1 label. In any case, has anyone in this thread reviewed #8159?

[InteXX]

@txwizard

Yes, the meaning of that label is unclear.

#8159: The hits just keep on coming.

[SeanLively]

@rrelyea Is the forced SemVer behavior going to be rolled back or not?

[StingyJack]

Also tagging @aortiz-msft since this item has had a Triage discussion

[txwizard]

I see this issue has dragged on for nearly 4 years, without an end in sight. There must be some powerful forces wanting to ram SemVer down everyone's throats.

[Grivus]

I'm also interested in resolving this issue in 2021.

[StormRider01]

Another year and change, and a new NuGet roadmap: #11571
@anangaur, @rrelyea, how about being honest and close this ticket as won't fix? Clearly someone waved the big stick around and declared SemVer as the only versioning strategy.

[StingyJack]

Clearly someone waved the big stick around and declared SemVer as the only versioning strategy.

@StormRider01 - ...and the stick was so big and heavy (caber toss?) they didnt have enough strength left to increment the major version when releasing this breaking change. Or for some other reason. The irony of coercing everyone* to use a specific versioning scheme and at the same time violating the most important part of that same versioning scheme should have been enough to revert the change or actually bump the major version.

They dont even think that they did something wrong by trying to dictate version numbers, or that saying something like "we will only take a PR for it if the result requires the breaking behavior to be default and a cmd arg to be used to get previous behavior" is salting the wound.

* everyone - the intended audience for this was only users of nuget.org, but it affected everyone that uses private package feeds

[uladz-zubrycki]

The only thought that I have after reading this thread is best represented as an emoji, here you are — 🤡.

SemVer is an unnecessary overkill for me and I'd like my package manager to not change the version of the package that I'm publishing without me asking it to do so.

This issue is still applicable 6 years after it was firstly submitted, please stop ignoring it.

[gopannabd]

Versions are strings that can be Semantic Versions.

This remains an active issue in 2023.