Enhance the experience for the testing of packages in development

[anangaur]

https://twitter.com/Mpdreamz/status/965325828455321600

[nkolev92]

I guess the intent here is to use this for testing?

For example in my day-2-day flow I have some extra targets that let me add a new source and new global packages folder, so each time I have a new iteration of the package restored.

Personally, I'm not confident that "teaching" NuGet PR to work with individual nupkgs is a good idea, but there's definitely some improvements we can do here.

[nkolev92]

I updated the title a little bit.
We can come up with a better out of the box experience for this scenario.

The easy workaround here for customers facing this is to define a new global packages folder, by potentially adding a new nuget config file, in the root of the project where they are working.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="MyCoolLocalSource" value="F:\ServerCommon\artifacts" />
  </packageSources>
   <config>
        <add key="repositoryPath" value="LocalGlobalPackagesFolder" />    </config>
</configuration>

Ofc, you can always use some msbuild goo, but lots of people would be more comfortable with the above.

[anangaur]

I don't think it would solve the problem. I guess the intention is to replace the same version package in the project on every restore.

May be if we had the nuget locals all -clear for individual or set of packages then that might have helped?

[nkolev92]

Yeah, I forgot to mention it...they'll just do locals clear -globalpackagesfolder when ready for a new version.

That's the best you can do without any work and msbuild tricks.

[StingyJack]

define a new global packages folder,

HintPath's get broken by trying to use this unless you can get everyone to use the same local folder path,

The way I figured out to get around this OP issue in development was to use a powershell script to build the package I'm testing. It takes the current epoch, splits it in about half, and supplies that as the Rev and Build elements of the version number when packing. So long as I dont run the script twice within the same second, there isnt a problem with caching.

Consuming the package for testing is just setting up a local folder as source, within VS's sources and not in the nuget.config for the solution. The latest one us right on the Update tab of the NuPMUI

Yeah, I forgot to mention it...they'll just do locals clear -globalpackagesfolder when ready for a new version.

Who are these people you refer to? I only know nuget locals all -clear (or is it clear -all? that command is always weird) after having to try and fix a few problems. I suppose a package author would need to know pack and maybe push, and those are kind of obvs, but locals is not. I'm sure that more than one developer has trawled through the filesystem looking for cached packages to delete.

[nkolev92]

My comment about the usage of the global packages folder is meant for Package Reference.

The global package folder is the de facto installation directory for PR, while in Packages.Config it's merely used as another source.
The packages.config packages folder works differently as it's only used by the current project.

packages.config is fundamentally broken for the above scenario because of hint paths.

[StingyJack]

The content in packages.config cannot really be materially different from package ref. How its implemented in VS/Nuget is what is creating the incorrect hint paths, etc. Why you bothered with that file and created additional work instead of just addressing the hint pathing is still a mystery to me, but it seems no amount or delivery manner for feedback makes a lick of difference. Nevertheless, have some more...

Having a global packages folder sounds nice, but the command to clear them is not (use the "locals" to clear the "globals", no mention of "cache" in the command names).

Then again, I dont think I necessarily want a single folder with all the packages (in development and known good). If so, I sure wouldn't want to clear everything just to refresh a development package, or clear everything just to trim out a few unreleased development packages.

Defacto is not the same as default, so what you are saying is there is and will only be one global folder, and that by doing so you will again be creating additional burden for those who try to use this system to author packages.

[nkolev92]

The content in packages.config cannot really be materially different from package ref. How its implemented in VS/Nuget is what is creating the incorrect hint paths, etc.

The hint paths are a bad design, it's as simple as that. Creates a lot of management overhead and it's very error prone. That's why package.config is not a priority for the future.

If so, I sure wouldn't want to clear everything just to refresh a development package, or clear everything just to trim out a few unreleased development packages.

That's what this issue is for.
Improving that workflow.

Defacto is not the same as default, so what you are saying is there is and will only be one global folder,

That's exactly how Package Reference works. You can call it default if you will.
There's always one global packages folder, that's a common thing among package managers nowadays.
It minimizes the need for deduplication. The build system then knows how to copy those files into the output folders when needed.

and that by doing so you will again be creating additional burden for those who try to use this system > to author packages.

In the packages.config case, the global packages folder is still used, rather as a priority source.
Meaning if I'm developing PackageA 2.0.0, and do a restore, it'll extract it to the global packages folder and copy it to the packages folder of the project that I'm testing.
Then I'd need to delete the packages folder and run another restore, but that'd pick up the extracted package from the global packages folder.
Basically packages.config = 2 actions required, package reference = 1 action required, so it's not an additional burden.

[StingyJack]

Hint paths are part of the project system and not of nuget or packages.config. The PackageRef implementation could have used them too, and instead of fixing the packageCfg implementation to not use them, you all decided to do what is generally the most expensive and disruptive choice in software -
a total rewrite.

Npm can use a global folder, but I'm not required to do so. I'm pretty sure Bower is the same. And the difference in word choice between "Defacto" and "Default" is important. The former means "this is what it is, there are no choices", but I think you are saying the truth is that its the latter and there is a choice to be had, and by the individual developer and not governed by a config that is checked into version control and locking everyone in.

I hope its not going to be as infuriating as the GAC is in regards to loading assemblies, where the GACced asm is loaded before one adjacent to the program's location.

PackageCfg could use a global folder, but I found it was broken due to hint path.

[nkolev92]

Hint paths are hard coded values.
Fixing it is nearly impossible. Only the VS project systems knows how to do these problem, so NuGet will never have a consistent experience when working from commandline.
There's lots of other flaws with the whole approach, merge conflicts etc.
Not to mention that moving projects from 1 location to another is impossible.
The whole concept is fundamentally flawed.

That's why in package reference, the knowledge of where exactly do dependencies come from is decoupled from the knowledge about "which" dependencies I want.

Again, packages.config uses global packages folder as an extra source.

Regarding the defacto vs default.
There's no other options in PackageReference.
Packages are "effectively" installed in the global packages folder. That's the only place where a nupkg will ever be extracted.
With the help of the project.assets.json, the build knows how to find these files and copy them around during build.

[StingyJack]

Fixing it is nearly impossible. Only the VS project systems knows how to do these problem, so NuGet will never have a consistent experience when working from commandline.

PackageCfg works just fine from the command line, be that devenv.exe or MSBuild.exe or nuget.exe. so that statement is just not correct. Nuget is the package manager for visual studio projects, so I dont think it matters how well it works outside the project system. Other uses for the nupkg use it differently anyway. (This argument slightly reminded me of the bad old days when there was only classic asp and web site projects - a bunch of unrelated files in a folder that you hoped could be made to stay working.)

There's lots of other flaws with the whole approach, merge conflicts etc

Merge conflicts from nuget happen because of two things...

• HintPaths when you use a global folder, and they are easy to resolve - last checkin wins. If you recognize that even the best and most attentive developer can checkin code that breaks the build, you would use gated checkin which prevents that from even being a merge conflict.
• the constant adding/editing of app.config files for assembly projects. It is a constant effort to undo that so its not creating a merge conflict.

I can't use a single source folder all the time, and with the example given of authoring a package, I wouldnt want to. Other package managers dont coerce this either. Stop trying to make me do things that have no value to me. And Stop doing things that create work for me that also have no value.

Not to mention that moving projects from 1 location to another is impossible.

You... are kidding right? They are files. You Copy Them.

The whole concept is fundamentally flawed.

"The whole concept" is nuget package management, but even at a smaller scope, package ref and package config are fundamentally conceptually the same. If one is fundamentally flawed, both are. However the major flaw here is the idea that writing an entirely new replacement system is going to somehow be easier and have less ripple effect and cost than addressing a set of problems in the existing one. I try to keep pointing out that you (MSFT in general) are consistently making things difficult with no added benefit or value. You all (including the netcore group) need to look at the overall change you are trying to make a bit more objectively.

packages.config and/or nuget.config is replaced with project.assets.json

... is how that reads. Again, change with no benefit.

Back to the OP,
I want that package (points at a package in a source), not that one (points at a package of the same name in a different source). The program needs to do as asked, and not give a package from a source that I don't want, even if it looks the same. There isnt any visible benefit to decoupling what and where to that degree.

The easier and safer authoring workflow is going to require either naming specific packages or multiple local sources. The idea that of a single "local-global" one is not going to work.

[nkolev92]

"The whole concept" is nuget package management, but even at a smaller scope, package ref and package config are fundamentally conceptually the same. If one is fundamentally flawed, both are.

That's not correct.

Package Reference is transitive, readable, provides a consistent experience from commandline and VS.
Packages.config has install scripts which only execute in VS, writes to your csproj which is only done correctly in VS. It has the concept of an install/uninstall action, which on it's own can get out of hand, if you modify file/content that was added by the install action.

is how that reads. Again, change with no benefit.

It's not a change with no benefit.
It allows you to reuse the same dlls, so you don't hold 15 copies of the same package in your 15 solutions.

Regardless, let's stay on topic.

The local/global trick is just a workaround/good practice for testing right now.
I don't have a slam-dunk approach for the new feature right now.
Binding a package to a specific source is an idea. (Then if a hash is different from currently installed one, then just update or something).

That'd likely require a new gesture.

[StingyJack]

Concept and Implementation are different things. You are describing the implementation. The concept of both is to distribute packaged software components for use in visual studio projects.

The choice of an entirely new implementation (with all new bugs being created in both implementations) over correcting the issues with the existing one seems both dumb and extremely costly.

I'd rather have 15 locations instead of being coerced into one that is subject to disagreement by different running instances of visual studio. File locking isnt handled right when there is only one VS instance touching a packages folder, I can't imagine how problematic it will be where all of them are competing and disagreeing.

Back on topic then...
Until you have figured out how most of the existing functionality will be available in the new implementation, stop calling the existing one "legacy", stop closing issues as "wont fix", and start actually fixing the software that you are responsible to support and fix.

[nkolev92]

@joacar Trying to continue the discussion from #9366, here.

Would an explicit gesture to update a package that's in active development be satisfactory?
What about its dependencies? How often do you test packages with many local dependencies?

[khellang]

This issue just keeps coming up, spread all over the place. Here's a thread (with a myriad of links) on this very topic going back years; dotnet/sdk#1151. And here's a year-old thread from Twitter with a bunch of comments and feedback; https://twitter.com/natemcmaster/status/1099021447920406529.

[khellang]

Would an explicit gesture to update a package that's in active development be satisfactory?

What would that look like? Sounds like what you could do today by pointing a local NuGet feed to the bin folder of a project and manually updating version numbers and build packages.

What about its dependencies? How often do you test packages with many local dependencies?

I would say it's pretty common to have at least a few dependencies, at least in the cases I've needed it.

[nkolev92]

Thanks for the links. @khellang

What would that look like? Sounds like what you could do today by pointing a local NuGet feed to the bin folder of a project and manually updating version numbers and build packages.

An idea would be to mark a package with certain metadata and then we can maybe overload previous gestures like -force and rebuild in VS.

Given that restore runs on every build, it'd be unperformant to check if the packages are up to date.

So a solution would require the user saying something like "I changed the package".

The Package/Project Reference is certainly one way to solve this. Don't have a concrete proposal for that though, given that it involves more than just NuGet.

[joacar]

Don't know that much of the inner details of the NuGet magic, but here's an idea.

Extend child element of <packageSource> in nuget.config with additional attribute development="true" to indicate that a path have special meaning. The presence of this attribute signals to NuGet. upon probing sources for package to install, two things:

1. The found package must not be added to the cache (global-packages);
2. The version should be ignored, only match by name.

In this case it's adding it to the cache that is part of the problem and we wan't to increase development and testing of nuget, hence caching that particular package is not needed. Step 2 let use move away from versioning issues. Using nuget.config give the options to find it using nuget locals all -list for a given directory.

Edit Add script from closed issue
Edit Update script

#!/bin/bash

if [[ -z $1 ]]
then
  echo "Version is required"
  exit
fi

# Check if nuget local packages should be cleared
if [[ -x `which nuget` ]]
then
  echo "Using nuget at `which nuget`"
  # Format: 'global-packages: C:\Users\<username>\.nuget\packages' (take note about the two :)
  PACKAGES=`nuget locals global-packages -list`
  # Replace forward slash with backslash
  PACKAGES=$(echo $PACKAGES | cut -d ':' -f3 | sed 's/\\/\//g')
  # Add missing drive letter
  PACKAGES="/c$PACKAGES"
  echo "Delete cached NuGet directory '$PACKAGES'"
  rm -rf ${NUGET_GLOBAL_PACKAGES}<package>
else
  echo "No nuget.exe found in path"
  exit
fi

ROOT=`pwd`
dotnet pack -c Debug <proj> -p:RepoRoot=${ROOT}
# Since pre-release version is required
dotnet add <proj> package <package> -v $1

[Denis535]

I tried to clear the folder PackageOutputPath, but even here I ran into a problem: BeforePack target is invoked after new package is created!

<Target Name="BeforePack" BeforeTargets="Pack">
    <ItemGroup>
        <FileToDelete Include="$(MSBuildProjectDirectory)\$(PackageOutputPath)*.nupkg" />
    </ItemGroup>
    <Delete Files="@(FileToDelete)" />
</Target>

I understood what the problem is. Let's look at "Pack" target. This target does almost nothing. Everything happens in dependencies. So, I should run my "BeforePack" target before those dependencies.

<PackDependsOn>$(BeforePack); _IntermediatePack; GenerateNuspec; $(PackDependsOn)</PackDependsOn>

<Target Name="Pack" DependsOnTargets="$(PackDependsOn)" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <IsPackableFalseWarningTask Condition="'$(IsPackable)' == 'false' AND '$(WarnOnPackingNonPackableProject)' == 'true'" />
</Target>

[Denis535]

You can use DateTime.UtcNow.ToString("yyyy-MM-dd-HH-mm-ss") to get readable package version.

<VersionPrefix>2.0.0</VersionPrefix>
<VersionSuffix>alpha.$([System.DateTime]::UtcNow.ToString(yyyy-MM-dd-HH-mm-ss))</VersionSuffix>

In consumer project you can use floating version. It will automatically update your package.

<PackageReference Include="NuGetLibrary" Version="2.0.0-alpha.*" />
<PackageReference Include="NuGetLibrary" Version="*-*" />

In consumer project you can also make new package on every rebuild event. Only I'm not sure if I specified the properties correctly or not.

<Target Name="__BeforeRebuild__" BeforeTargets="BeforeRebuild">
    <MSBuild Projects="..\NuGetLibrary\NuGetLibrary.csproj" Targets="Pack" Properties="Configuration=$(Configuration);Platform=$(Platform)" />
</Target>

[LazZiya]

@Denis535 I like the idea, but after saving the .csproj file I see many repeated errors in Package Manager output window, even if the pacakge is created successfuly.

Error occurred while restoring NuGet packages: Invalid restore input. Missing required property 'Name'. Input files: D:\MyWorks\MyNugetPackage.csproj.
Time Elapsed: 00:00:00.0039303
========== Finished ==========

Error occurred while restoring NuGet packages: Invalid restore input. Missing required property 'Name'. Input files: D:\MyWorks\MyNugetPackage.csproj.
Time Elapsed: 00:00:00.0029827
========== Finished ==========

Error occurred while restoring NuGet packages: Invalid restore input. Missing required property 'Name'. Input files: D:\MyWorks\MyNugetPackage.csproj.

Those errors are not showing with standard naming like:

<VersionPrefix>1.0.0</VersionPrefix>
<VersionSuffix>alpha4</VersionSuffix>

[StingyJack]

Making a package every time you build the project is a waste of time for non-trivial packages. You also lose the ability to update the nuspec and try rebuilding the package without having to wait for the whole project to rebuild. Having the things separate means at most an Alt+Tab, Up-Arrow, Enter when you need to build, then pack. Not having them together means waiting for things that take longer than those keyboard keystrokes.

I mention nuspec because historically, packing without a nuspec is inviting weird, unwanted, and inconsistent behaviors from nuget.

[Denis535]

@LazZiya
Here is my example: NuGetLibrary.zip. Everything works.
Updated. It looks like Id, Tags, Icon must ber renamed to PackageId, PackageTags, PacakgeIcon.

[Denis535]

@StingyJack
I came to the conclusion that the .nuspec file should be generated by dotnet.exe. Otherwise, it only leads to problems.

[LazZiya]

@Denis535 many thanks for the package, it helped me and I created another version formatting, sharing it below for those who may want a short and readable version:

<TotalSeconds>$([System.DateTime]::UtcNow.TimeOfDay.TotalSeconds)</TotalSeconds>
<VersionPrefix>1.0.0</VersionPrefix>
<VersionSuffix>alpha.$([System.DateTime]::UtcNow.ToString(yy)).$([System.DateTime]::UtcNow.DayOfYear).$([System.Math]::Ceiling($(TotalSeconds)))</VersionSuffix>

The output package name is something like:

MyNugetPackage.1.0.0-alpha.20.292.52544

It requires more processing, but it looks nice :)

[StingyJack]

@Denis535 what's the dotnet command for creating the nuspec file that you can edit/tweak and then later feed into the pack command?

[joacar]

@StingyJack dotnet pack doesn't support nuspec files, you'll have to use nuget for that. As for generating nuspec you can use nuget spec <project> as described here. Unfortunately it doesn't evaluate MSBuild properties. So if you use $(AssemblyName) the nuspec creation will fail.

[nkolev92]

@StingyJack dotnet pack doesn't support nuspec files,

@joacar
You can pack nuspecs with dotnet pack as well, the difference is that you still need a project file.

https://docs.microsoft.com/en-us/nuget/reference/msbuild-targets#packing-using-a-nuspec

I mention nuspec because historically, packing without a nuspec is inviting weird, unwanted, and inconsistent behaviors from nuget.

If you getting any of these scenarios do let us know by creating an issue. dotnet pack as a project file pack should be consistent, so if something is off we'd love to hear about it.

[joacar]

@nkolev92 Perhaps that was just when doing it from Pipelines. The documentation says either csproj or nuspec which I can see how that doesn't work if you need the csproj file.

searchPatternPackPath to csproj or nuspec file(s) to pack	Pattern to search for csproj or nuspec files to pack. You can separate multiple patterns with a semicolon, and you can make a pattern negative by prefixing it with !. Example: /*.csproj;!/*.Tests.csprojArgument aliases: packagesToPack

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#pack

[nkolev92]

Maybe the simpler way to think about is that through dotnet.exe you always call pack on the csproj.

Then the project file itself can use the conventions for csproj packing for PackageReference projects, or itself it can point a nuspec file like in the docs I linked.

[StingyJack]

If you getting any of these scenarios do let us know by creating an issue. dotnet pack as a project file pack should be consistent, so if something is off we'd love to hear about it.

@nkolev92 - i have used nuget pack task following the VSBuild/MSBuild task for the nuspec files in the branch for a few years. dotnet was not working to handle mixed project types (solutions with netfx legacy projects+package.config, netfx sdk projects + package.config, netcore with packageref, etc) at the time but that's probably been fixed. Since dotnet is just a wrapper for common calls to other utilities, I'd still expect nuget.exe to operate correctly.

The biggest example I run into somewhat often is where packing a csproj doesnt work in any setup is when there are two assemblies in a branch that produce packages, and one of them depends on the other. An example would be a common objects and interfaces library, and a services library that implements the interfaces and uses the objects in the common library and whose proj file consumes the common proj as a project reference. The common library could be installed into a project, or the service library could be installed and that should install the common library package of the correct and corresponding version. I tried to explain a facet of this before , but apparently did not explain it well enough. The service library package doesnt include the common library package as a dependency when packing the proj, and in some versions of nuget.exe packing the service library's proj file has not packaged the common library file as part of the service library package. I thought I entered that as its own issue but I cant find it at the moment.

I'm not sure how often this comes up in other's development environment work, but it has come up for me and my colleagues often enough that we had to draft a wiki page for this topic.

[nkolev92]

I thought I entered that as its own issue but I cant find it at the moment.

This is the issue: #4491.

In general, dotnet pack & msbuild /t:pack are equivalent and they should be used for PackageReference.
nuget.exe is to be used with older style scenarios where the projects are packages.config and scenarios where only a nuspec is used for packing.

[ericsampson]

@nkolev92 I would really love to see some solutions for improved/faster inner-loop experience when developing packages.
It would fit very well with one of the main .NET 6 themes: dotnet/core#5510

My first thought would be to have a mechanism to switch quickly between package reference and project references in VS/VSCode/etc, maybe along these lines: dotnet/sdk#1151

[stemoser]

Just another note: If the package you are testing contains assemblies that are going to be loaded by MSBuild, this environment variable may be needed to not having MSBuild locking files, thus preventing clearing the cache:

MSBUILDDISABLENODEREUSE=1

Take this into account if this issue will ever be addressed...

[nickhoeferpickpro]

I have been spending the last couple days trying to figure this out. I did something like this to achieve a similar effect.
I borrowed the idea from @thesushil from #9891

<Target Name="DeleteLocalCache" BeforeTargets="Pack">
  <RemoveDir Directories="$(NugetPackageRoot)$(PackageId.ToLower())\$(version)" />
  <Message Text="Cleaning $(NugetPackageRoot)$(PackageId.ToLower())\$(version)" />
</Target>

<Target Name="UpdateLocalCache" AfterTargets="Pack">		
  <Message Text="creating diretory $(NugetPackageRoot)$(PackageId.ToLower())\$(version)" Importance="high" />
  <MakeDir Directories="$(NugetPackageRoot)$(PackageId.ToLower())\$(version)" />
  <Message Text="copying file $(PackageOutputPath)$(AssemblyName).$(version).nupkg" Importance="high" />
  <Copy SourceFiles="$(PackageOutputPath)$(AssemblyName).$(version).nupkg" DestinationFolder="$(NugetPackageRoot)$(PackageId.ToLower())\$(version)" />
		
  <Exec Command="tar -xf $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\$(AssemblyName).$(version).nupkg -C $(NugetPackageRoot)$(PackageId.ToLower())\$(version)" />
  <Exec Command="certutil -hashfile $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\$(AssemblyName).$(version).nupkg SHA512 &gt; $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\temp1.txt" />
  <Exec Command="more +1 $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\temp1.txt &gt; $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\temp2.txt " />
  <Exec Command="powershell Get-Content $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\temp2.txt -First 1 &gt; $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\$(AssemblyName).$(version).nupkg.sha512" />
  <Exec Command="del /f $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\temp1.txt" />
  <Exec Command="del /f $(NugetPackageRoot)$(PackageId.ToLower())\$(version)\temp2.txt" />
</Target>

If anyone knows a more elegant way to handle this, I'm all ears. Changing version numbers each time just for local development is just silly. semver is great and everything but we shouldn't be forced to adhere to it just for the sake of purity because doing so results in dirty hacks or workarounds.