Cannot install prereleased tool package from dotnet-install-tool without specifying the exact version

[natemcmaster]

To install a prerelease version of a CLI tool, users must know the exact version they want to install. This means they have to open a browser to nuget.org or myget.org to find the package.

For example, this is the console output for dotnet-watch

> dotnet install tool --global dotnet-watch --source https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json
Install failed. Failed to download package:
NuGet returned:

Failed to restore package.
WorkingDirectory:
Arguments: restore C:\Users\namc\AppData\Local\Temp\0rc552nv.kmy\wjh4elju.lgh.csproj --source https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json --runtime win10-x64 /p:BaseIntermediateOutputPath=\"C:\Users\namc\.dotnet\tools\dotnet-watch\gx5ifack.dyr\"
Output:   Restoring packages for C:\Users\namc\AppData\Local\Temp\0rc552nv.kmy\wjh4elju.lgh.csproj...
C:\Users\namc\AppData\Local\Temp\0rc552nv.kmy\wjh4elju.lgh.csproj : error NU1103: Unable to find a stable package dotnet-watch with version

C:\Users\namc\AppData\Local\Temp\0rc552nv.kmy\wjh4elju.lgh.csproj : error NU1103:   - Found 124 version(s) in https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json [ Nearest version: 2.1.0-preview1-27934 ]
C:\Users\namc\AppData\Local\Temp\0rc552nv.kmy\wjh4elju.lgh.csproj : error NU1103:   - Found 1 version(s) in C:\Users\namc\.dotnet\x64\sdk\NuGetFallbackFolder [ Nearest version: 2.1.0-preview1-28124 ]
  Restore failed in 680.98 ms for C:\Users\namc\AppData\Local\Temp\0rc552nv.kmy\wjh4elju.lgh.csproj.

Usage: dotnet install tool [options] <PACKAGE_ID>

Arguments:
  <PACKAGE_ID>   NuGet Package Id of the tool to install.

Options:
  -g, --global        Install user wide.
  --version           Version of the tool package in NuGet.
  --configfile        The NuGet configuration file to use.
  --source <SOURCE>   Specifies a NuGet package source to use during installation.
  -f, --framework     The target framework to install the tool for.
  -h, --help          Show help information.

This is the console output when trying to specify --version 2.1.0-*

> dotnet install tool --global dotnet-watch --version 2.1.0-* --source https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json
System.ArgumentException: Illegal characters in path.
Parameter name: path
   at System.IO.Path.GetFullPath(String path)
   at System.IO.Directory.CreateDirectory(String path)
   at Microsoft.DotNet.ToolPackage.ToolPackageObtainer.EnsureDirectoryExists(DirectoryPath path)
   at Microsoft.DotNet.ToolPackage.ToolPackageObtainer.CreateIndividualToolVersionDirectory(String packageId, PackageVersion packageVersion)

   at Microsoft.DotNet.ToolPackage.ToolPackageObtainer.ObtainAndReturnExecutablePath(String packageId, String packageVersion, Nullable`1 nugetconfig, String targetframework, String source)
   at Microsoft.DotNet.Tools.Install.Tool.InstallToolCommand.ObtainPackage(DirectoryPath executablePackagePath, DirectoryPath offlineFeedPath)
   at Microsoft.DotNet.Tools.Install.Tool.InstallToolCommand.Execute()
   at Microsoft.DotNet.Cli.DotNetTopLevelCommandBase.RunCommand(String[] args)
   at Microsoft.DotNet.Tools.Install.InstallCommand.Run(String[] args)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

Ideas

• Support wildcards --version 2.1.0-*
• Support a --prerelease flag

cc @KathleenDollard

[natemcmaster]

cc @DamianEdwards

[KathleenDollard]

--prerelease will be passed to NuGet

[peterhuene]

I may have a fix for the wildcard issue with my changes related to the uninstall command work.

I propose not creating a version sub-directory (this is the source of the illegal path exception) under the staging directory when restoring and using the version number from the nuget assets file when renaming from staging to final location.

[peterhuene]

I've confirmed that --version can now support NuGet version ranges and wildcards if we omit the explicit version number (and the need for a "placeholder" too) from the staging directory path.

$ dotnet install tool -g --version '[1.0.4,)' foo
If there were no additional instructions, you can type the following command to invoke the tool: foo
Tool 'foo' with version '1.0.4' was successfully installed.

I'll include this fix with the upcoming uninstall command implementation.

[KathleenDollard]

Are we/will we only install preview with the --preview switch?

[peterhuene]

That's my understanding.

[wli3]

The reason of not support 2.* is here https://github.com/dotnet/cli/blob/master/src/dotnet/ToolPackage/PackageVersion.cs#L10

If version is not null, we assume it is a concrete version. However, in 2.* case, it is not. So it all goes wrong after that.

And I don't think we should guess which is concrete version to nuget 2.*, 2.1.1, [2, ). We should just always assume it is not concrete and let nuget restore with it(put 2.* in temp project). After that, we rename and use the version nuget restored(this logic has been there already in no version case)

[peterhuene]

That's what I've implemented in the uninstall command work; I got rid of PackageVersion entirely and now just restore directly to the random staging directory (i.e. with no "placeholder" subdirectory). We then detect the version that was installed and use that version when renaming from staging to the final package directory.

I've confirmed this works for wildcards and ranges, as well explicit versions and '*'.

[peterhuene]

I'm going to resolve this as fixed with the merging of dotnet/cli#8615 now that it properly supports version ranges that enable installing pre-release packages:

$ dotnet install tool --global dotnet-watch --source https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json --version 2.1.0-preview2\*
/var/folders/rl/m1rv1rxd643gpwspqrbxy76r0000gn/T/mhmqoe5u.gh0/u22v3akx.yw1.csproj : warning NU1603: u22v3akx.yw1 depends on dotnet-watch (>= 2.1.0-preview2) but dotnet-watch 2.1.0-preview2 was not found. An approximate best match of dotnet-watch 2.1.0-preview2-28177 was resolved.
If there were no additional instructions, you can type the following command to invoke the tool: dotnet-watch
Tool 'dotnet-watch' (version '2.1.0-preview2-28177') was successfully installed.

I don't think a --prerelease option for dotnet install tool makes sense since the package is being installed via a restore operation and the proper way to enable installing pre-release packages via this mechanism is with a version suffix wildcard.

Please re-open this issue if you feel the fix is inadequate. Thanks!

On a side note: I hope to clean up the warning / error output to try to eliminate the clutter coming from the use of the temp project soon.

[natemcmaster]

What will be the output when calling dotnet install tool --global MyTool where the only versions of MyTool are pre-release?
How hard would it be to create an experience like this?

> dotnet install tool --global MyTool
Installation failed. Unable to find a stable package version for MyTool. The following verisons were found:

 - 0.1.0-alpha
 - 0.2.0-beta 

To install a specific version, use `dotnet install tool --global MyTool --version $(version)`

I know our current implementation is dependent on dotnet-restore for the restore error messages...but that seems to be the tail wagging the dog. Have we considered other ways to implement package fetching that would allow us to do this?

[peterhuene]

The output would be along the lines of (with the fix from dotnet/cli#8707):

$ dotnet install tool -g --source https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json dotnet-watch
error NU1103: Unable to find a stable package dotnet-watch with version
error NU1103:   - Found 264 version(s) in aspnetcore-dev [ Nearest version: 2.1.0-preview1-27934 ]
error NU1103:   - Found 0 version(s) in /Users/peterhuene/src/cli/bin/2/osx-x64/dotnet/sdk/NuGetFallbackFolder
The tool package could not be restored.
Tool 'dotnet-watch' failed to install.

Unfortunately, as you stated, we have limited control over the experience because we currently have to rely on a generic restore operation to install the tool package. Hopefully in the future we can improve the UX via more direct control over the package install, but that's not the plan for the 2.1 release.

[natemcmaster]

Ok. Well, supporting wildcards in --version is a good step forward. Thanks!

[wli3]

Since dotnet-watch is bundled in SDK as well. I guess --prerelase has even lower priority.

[natemcmaster]

@wli3 This is about more than just dotnet-watch. It applies to all tool authors releasing tools with pre-release package versions.

[DamianEdwards]

I don't think a --prerelease option for dotnet install tool makes sense since the package is being installed via a restore operation and the proper way to enable installing pre-release packages via this mechanism is with a version suffix wildcard.

I'm not sure I understand this reasoning. The NuGet cmd line experience (and UI) has always had the concept of "please install whatever is the latest version of this package, including pre-releases". That is no less applicable in the case of global tools, and while the wildcard support is good, it's in no way the idiomatic way to do this in NuGet. Very few people know about (and even less use) version ranges or wildcards in NuGet.

[peterhuene]

Because we use a package reference under the covers, we have to give a version range / wildcard. Perhaps we could have a --prerelease option that implies a certain wildcard that accepts prerelease packages? * means latest stable; is there a value that implies "any latest"?

[natemcmaster]

There isn't a NuGet wildcard syntax in .csproj for this AFAIK, but there is a way to do this in NuGet's API. This kind of limitation is part of what I meant by this:

I know our current implementation is dependent on dotnet-restore for the restore error messages...but that seems to be the tail wagging the dog. Have we considered other ways to implement package fetching that would allow us to do this?

The implementation we have chosen gives us access to a very small set of the broad capabilities NuGet already has baked in. I recommend we consider using the NuGet API programmatically instead of generating a .csproj file and shelling out to dotnet-restore.

[peterhuene]

My understanding is that it was done this way due to a technical limitation on NuGet not supporting xplat for the package management API. I think NuGet/Home#6150 is a tracking issue. Thus we're working within the confines of these limitations. I'd love to see this implementation removed in favor of an experience that is more familiar to NuGet users, including a --prerelease option that can mean "install latest".

[natemcmaster]

Take with grain of salt: I don't know what is in the NuGet.PackageManagement API that you might need, but I know you can implement a programmatic restore with existing xplat API. MSBuild did this using RestoreRunner and I've written similar code in our build tools to download and fetch packages.

[peterhuene]

I don't know if @wli3 looked into programatic restore vs. just invoking the restore target, but just casually looking at the implementation you provided it just seems like it'd perhaps improve tool install performance but not necessarily solve the --prerelease problem.

My guess is that it was done this way in the hope that NuGet would (sooner than later) provide a xplat API for managing packages in a manner similar to the NuGet command line experience and not reinvent the wheel for what is essentially a workaround.

If there is a way to accomplish this with the current xplat APIs, then I absolutely think we should investigate doing so.

[wli3]

NuGet/Home#4699 CLI don't support prerelease for dotnet add which is very similar

[wli3]

recommend we consider using the NuGet API programmatically instead of generating a .csproj file and shelling out to dotnet-restore.

We asked NuGet as well
NuGet/Home#6629

Please follow the issue. I raised the similar question
NuGet/Home#5919 (comment)

Here is NuGet's answer for why we should not use existing C# API cc @nkolev92

@wli3 As I mentioned in the other issue, I'm not sure this is the appropriate direction for the dotnet tools.
I'm not a fan of expanding the scope of a "download API", to fit reqs that specifically require restore. (compat checking, runtime graph eval)
This API is not supposed to replace restore.
The C# API thing is a also a tricky one, because that'd effectively introduce a new public surface area for NuGet, and that surface area is not something that is written with that in mind.
If logging/language is the only issue, we can certainly look into fixing that.
The error code in question, is a DotnetTool specific error code.

[wli3]

@DamianEdwards I'm not sure I understand this reasoning. The NuGet cmd line experience (and UI) has always had the concept of "please install whatever is the latest version of this package, including pre-releases

nuget.exe does have this opinion(windows only), however, dotnet add package has never supported this. NuGet/Home#4699 is still in progress. I think we should support --prelease for tools after right after we support --prerelease for dotnet add package

[nkolev92]

Additionally, related work.
NuGet/Home#912

//cc @rrelyea

[DamianEdwards]

@wli3 honestly, I didn't even realize we didn't have --prerelease for dotnet add package. That is... severely sub-optimal. We really need to peel this onion and figure this out.

[natemcmaster]

Related: NuGet/Home#6658

[KathleenDollard]

Confirm if this is the correct summary of current thinking:

• the lack of --prerelease is not a great experience for customers, but there has been little comment on it for add package
• this exposes some of the negative aspects of using the temp project for dotnet install tool and shortcomings in the current .NET Core NuGet CLI
• fixing this is a priority for another CLI release (hopefully 2.1.400), but it is too late for the kinds of changes that we would need, and at best it would be a hack
• to mitigate for this release, we will point out to people the closest match which is displayed on failure. We will also display a different static message on failure that will mention the --version switch (Kathleen to provide)
• we need closest match to work. There may be a bug with it as we believe there are preview2 versions available for the @peterhuene 's sample above

Were there any other actions items?

[nkolev92]

As a side-note.
The closest match works, but it's only the error message that's incorrect, as per NuGet/Home#6658

[wli3]

@tmds the fix only goes to 2.1.4xx dotnet/cli#9525

[wli3]

@aventurella do you mean when adding a package using dotnet add package Xxx.Yyy.Zzz --version 2018.8.3-dev* but you get explicit version Version="2018.8.3-dev0001 as result in csproj? And you are expecting Version="2018.8.3-dev*" ?

[theladyjaye]

Ok, well maybe it's a non-issue? I just did:

dotnet nuget locals all --clear

And I rebuilt all of our local packages.

Now running

dotnet add package Xxx.Yyy.Zzz --version 2018.8.3-dev*

Seemed to work without issue. Lemme try adding the package from Visual Studio too just to see.

¯_(ツ)_/¯

Seems to all be working as expected now. Dangit! hahahaha sorry for bugging you!

[yufeih]

What is the status of this issue? Installing latest pre-release tool is a badly needed scenario.

[DamianEdwards]

@yufeih you can use a wildcard in the version to make it simpler, e.g.

dotnet tool install -g dotnet-ef --version 3.0.0-preview*

[yufeih]

@DamianEdwards , that works if you already know the exact major.minor.revision, it does not express the intend to install the latest pre-release. Something like dotnet tool install --pre or dotnet tool install --version *-* is helpful.

[wli3]

unfortunately, this issue is still being blocked on NuGet/Home#912

[heaths]

While I'm not a fan of long command line options, dotnet list package --include-prerelease already exists. Should the parameters be consistent with dotnet, or with nuget (or both)?

[DamianEdwards]

FYI NuGet/Home#912 is being fixed right now. This could be revisited for an upcoming SDK update.

[wli3]

Thanks! This should be an easy change once the code flows to SDK. And we would add it once it is available since it is highly requested as well. @KathleenDollard

[aolszowka]

Is there no other work around today than knowing the complete major.minor.revision-dev?

Our use case is to try and give developers some tooling while they're working on updating the tooling[0]. I suspect our usecase is similar to what @yufeih was attempting to describe above. Having to have them edit the batch file to contain the current version is kinda a bummer, the next step is for us to write a PowerShell script to extract out the version from the CSPROJ File.

Here's what we're doing (with a batch file), is there a better way?

@ECHO OFF

SET PACKAGE_NAME=REDACTED

REM Clear any packages out of the local nuget package cache
REM This is because `dotnet tool install --no-cache` appears to be broken?
SET PACKAGE_CACHE_FOLDER=%userprofile%\.nuget\packages\%PACKAGE_NAME%
ECHO Attempting to Clean Existing Package Cache From %PACKAGE_CACHE_FOLDER%
IF EXIST "%PACKAGE_CACHE_FOLDER%" (
RMDIR /Q /S "%PACKAGE_CACHE_FOLDER%"
)

ECHO.
ECHO Delete Existing Packs
IF EXIST nupkg (
RMDIR /q /s nupkg
)

ECHO.
ECHO Building / Packing
dotnet pack --configuration Debug --version-suffix "dev"

ECHO.
ECHO Uninstall Existing Tooling
dotnet tool uninstall %PACKAGE_NAME%

ECHO.
ECHO Install the Latest Prerelease
dotnet tool install --add-source=nupkg --no-cache %PACKAGE_NAME% --version=2.0.0-dev

[0] I heard you like your tooling in your tooling

[nkolev92]

@aolszowka

Try --version "*-*"

It's basically what #9037 (comment) enabled.

[aolszowka]

@nkolev92 Solid this works, any reason to keep this open still then? or does this not work for most others?

[nkolev92]

I think the issue is open because the switch has not been added yet.

I'm not the owner though, so it's just a guess :)

[wli3]

Yes. The switch has not been added yet.

[wli3]

fixed in #19646