Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local Tools Early Preview Documentation #10288

Open
wli3 opened this issue Nov 6, 2018 · 33 comments
Assignees
Milestone

Comments

@wli3
Copy link
Collaborator

@wli3 wli3 commented Nov 6, 2018

Local Tools Early Preview Documentation

This document is for checkin build of dotnet SDK. The feature is still work in progress. We hope to gather more feedback especially on user experience by this documentation. If you think the experience need improvement, you can file issues at Github

Motivation

Enable high cohesion between tools and corresponding code repositories. The repository maintainer could check-in a list of dotnet tools with certain version. And other contributor can use these tools within the repository with certain gestures. Local tools use the same package as global tools.

Walk through

Install the latest dotnet SDK "checkin build"

  1. You can find the installer at here under the Master (3.0.x Runtime) column.
  2. Most of the existing dotnet tools package(global tools) are targeting runtime netcoreapp2.1. You need to install it in order to run the existing command. You can find it at here under Run apps - Runtime column.

Create tool manifest file

Run dotnet new tool-manifest command. It will create a tool manifest file dotnet-tools.json under the directory .config.

Restore command

In order to use the command specified in tool manifest file. You need to run command dotnet tool restore and you should see the following success message:

Tool 'dotnetsay' (version '2.1.4') was restored. Available commands: dotnetsay


Restore was successful.

Run the command

Run local tool would require prefix dotnet. So it will not be ambiguous most of the time. And when there is a global tools with the pattern "dotnet-*" CLI will find the tool with the following rule

global tools command invoke local tools command invoke
dotnetsay dotnet dotnetsay
dotnet say (aka dotnet-say) dotnet say
dotnetsay and dotnet say both exist dotnet dotnetsay and dotnet dotnet-say

You could also call it by dotnet tool run dotnetsay hi. "dotnet tool run" is not ambiguous.

If you see the following error, please install netcoreapp2.1 runtime. At https://www.microsoft.com/net/download/dotnet-core/2.1. Since dotnetsay and most of the global tools are targeting 2.1 runtime while preview dotnet SDK only contains preview netcoreapp3.0 runtime.

It was not possible to find any compatible framework version
The specified framework 'Microsoft.NETCore.App', version '2.1.0' was not found.
  - Check application dependencies and target a framework version installed at:
      C:\Users\name\Downloads\dotnet-sdk-latest-win-x64-localtools\
  - Installing .NET Core prerequisites might help resolve this problem:
      https://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409
  - The .NET Core framework and SDK can be installed from:
      https://aka.ms/dotnet-download
  - The following versions are installed:
      3.0.0-preview1-27017-04 at [C:\Users\name\dotnet-sdk-latest-win-x64\shared\Microsoft.NETCore.App]

Manage local tools

Install new local tool

Run dotnet tool install <PACKAGE_ID> (dotnet tool install dotnetsay) to install new command. It will add the tool entry to your manifest file. And at the same time it will restore the tool so you can invoke it right away.

The message will show which manifest file it added to

You can invoke the tool from this directory using the following command: dotnet tool run dotnetsay
Tool 'dotnetsay' (version '2.1.4') was successfully installed. Entry is added to the manifest file /Users/name/tools/sub/dotnet-tools.json

Uninstall local tool

Run dotnet tool uninstall <PACKAGE_ID>

Update local tool

Run dotnet tool update <PACKAGE_ID>. If the new package version is higher. SDK will find the first manifest file that contains the package id and update it. If there is no such package id in any manifest file (that is in the scope), SDK will add a new entry to the closest manifest file.

List local tool

Run dotnet tool list to list all available tools in the current directory. It will also list the origin manifest of the tool. It can be used to diagnose unexpected tools in the current directory

$ dotnet tool list
Package Id            Version      Commands             Manifest
----------------------------------------------------------------------------------------------------------------------
dotnet-dbinfo         1.3.1        dotnet-dbinfo        /Users/name/tools/sub/dotnet-tools.json
dotnet-depends        0.2.0        dotnet-depends       /Users/name/tools/sub/dotnet-tools.json
amazon.ecs.tools      3.0.0        dotnet-ecs           /Users/name/tools/sub/dotnet-tools.json
dotnet-encrypto       1.0.5        dotnet-encrypto      /Users/name/tools/sub/dotnet-tools.json
t-rex                 1.0.53       t-rex                /Users/name/tools/.config/dotnet-tools.json

Feature Detail

The scope of the tool manifest file

If there is a tool manifest file or a tool manifest file under folder .config in the parent/current directory of the current directory. The current directory has the access to the tools specified in the tool manifest file. For example. If there is a tool manifest with a tool dotnetsay in the root of the repository, the user could run dotnet tool run dotnetsay hi in any sub directory of the repository.

isRoot flag

The flag in the manifest file means "stop looking up parent directories for manifest file when the first one is found". This entry in manifest file is required.

Invoke local tools

Global tools use $PATH and shell to resolve command via shim. However we could not use the same mechanism due to the limitation of the operation system shell.

Example of multiple tools

{
    "version": "1",
    "isRoot": true,
    "tools": {
        "t-rex": {
            "version": "1.0.53",
            "commands": ["t-rex"],
        },
        "dotnetsay": {
            "version": "2.1.4",
            "commands": ["dotnetsay"]
        }
    }
}

Requirement of commands

We require listing out the command in the tool package. Note many dotnet tool packages do not have matching package id and the command.

Term

Local tool scope: the working directory that has the access to the tool specified in the dotnet-tools.json stored in current or parent directory.

Tool manifest file: dotnet-tools.json

Repository maintainer: A dotnet tool consumer. Programmers have the write access to the repository and decide which dotnet tools should be used in this repository.

Repository contributor: A dotnet tool consumer. Programmers have the read access to the repository and use dotnet tools decided by repository maintainer.

Local tools restore: Restore all tools that is possible to be accessed by current directory and generate cache.

Resolver cache: Technical, not customer facing. Cache to speed up the local tool resolution.

@wli3 wli3 added this to the 3.0.1xx milestone Nov 6, 2018
@wli3 wli3 self-assigned this Nov 6, 2018
@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Nov 6, 2018

I will close this issue after preview is done

@jmezach

This comment has been minimized.

Copy link

@jmezach jmezach commented Nov 6, 2018

Am I correct in assuming that this would allow different versions of a tool within different repositories? For example, in the npm world we can have different versions of say typescript within different repositories. What happens if a global tool is also available? Does the local one basically override the globally installed one, provided that you'd run it by using dotnet ?

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Nov 6, 2018

Am I correct in assuming that this would allow different versions of a tool within different repositories? For example, in the npm world we can have different versions of say typescript within different repositories.

Yes. That is one of the problem we want to solve.

What happens if a global tool is also available? Does the local one basically override the globally installed one, provided that you'd run it by using dotnet ?

Outdated

Run local tool would require prefix dotnet. So it will not be ambiguous most of the time. And when there is a global tools with the pattern "dotnet-*" CLI will find the tool with the following rule

| global tools command invoke | local tools command invoke |
| ------ | ------ |
| dotnetsay | dotnet dotnetsay |
| dotnet say (aka dotnet-say) | dotnet say |
| dotnetsay and dotnet say both exist | dotnet dotnetsay and dotnet dotnet-say |

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Nov 16, 2018

Update:
To invoke local tools you need to use dotnet tool run dotnetsay hi.

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Dec 14, 2018

Added Manage local tools session

@amis92

This comment has been minimized.

Copy link

@amis92 amis92 commented Dec 14, 2018

After latest change (a move back from dotnet tool run fulltoolname to dotnet fulltoolname), shouldn't that:

global tools command invoke local tools command invoke
dotnetsay dotnet dotnetsay
dotnet say (aka dotnet-say) dotnet say
dotnetsay and dotnet say both exist dotnet dotnetsay and dotnet dotnet-say

Be rather

global tools command invoke local tools command invoke
dotnetsay dotnet dotnetsay
dotnet say (aka dotnet-say) dotnet dotnet-say
dotnetsay and dotnet say both exist dotnet dotnetsay and dotnet dotnet-say

For some reason this single case "infers" the dotnet part. I think it shouldn't as well?

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Dec 14, 2018

@amis92 sorry that was a mistake in the document. We still use dotnet tool run dotnetsay hi today. And we are still discussing if we need a short hand version or not. Corrected in the issue

@GerjanOnline

This comment has been minimized.

Copy link

@GerjanOnline GerjanOnline commented Jan 20, 2019

I am not liking the verbosity and unintuitiveness of dotnet tool run dotnetsay

Would it be possible to "auto detect" which tool should be run? So I think checking from working directory?

For example when I am inside my repository (which has dotnet-tools.json) it should pick the one listed. When I am outside the repository, it should pick the global one.

D:\dotnetsay (global one, ie version 1.1)
D:\myrepo\dotnetsay (local one, ie version 1.0)

But probably you need to do some weird magic/tricks, right? Because you only can have one in your PATH, so you need to create shims and all that kind of stuff.

@Thraka

This comment has been minimized.

Copy link

@Thraka Thraka commented Jan 31, 2019

What about compromising on that with dotnet tool dotnetsay? Basically, any command after dotnet tool that isn't recognized is parsed as if dotnet tool run was used.

@kzu

This comment has been minimized.

Copy link

@kzu kzu commented Mar 15, 2019

having local + global automatic fallbacks with shorthand names would be awesome.

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Mar 20, 2019

bring back dotnet <Toolname> to run a local tool after #10980

@amis92

This comment has been minimized.

Copy link

@amis92 amis92 commented Apr 11, 2019

Seeing as there is dotnet/announcements#107, and the "documentation" is currently this issue, can we have some info about how to author these local tools? I'm quite interested in testing them in preview.

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Apr 11, 2019

@amis92 authoring is the same as global tools which is available already https://docs.microsoft.com/en-us/dotnet/core/tools/global-tools-how-to-create

@poke

This comment has been minimized.

Copy link

@poke poke commented Apr 14, 2019

Just out of curiosity: Where are local tools installed to? Are they installed into a local folder relative to the dotnet-tools.json? Or are they still installed into a global location so projects that depend on the same version of the same tool version can reuse the binary? What is the uninstall story in that case? Is there a way to list all local tool installations and uninstall them properly?

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Apr 15, 2019

added "Update local tool"

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Apr 15, 2019

Note the following are current implementation detail, that is subject to change.

Where are local tools installed to?

The asset are installed to nuget global package folder.

uninstall story

It will be removed from the manifest file. No change in nuget global package folder since it is maintained by a different set of command(detail in previous link).

Is there a way to list all local tool installations and uninstall them properly?

If you mean "install" by downloading asset. There is no easy way to do so, however, you could browse your nuget global package folder if there is no override of that folder. At the same time, you could clean that folder using nuget command. However, that will also clean your other nuget packages like newtonsoft.json. (and cause a longer review next time a project uses it)

@tebeco

This comment has been minimized.

Copy link

@tebeco tebeco commented May 18, 2019

@wli3
Update is a great idea and would be really effective with a full update option :

dotnet tool update --local --all

asking developper to manually list + update tool one by one is kinda harsh

I'm super lazy and if would add a nightly build on my CI just for that one

so every night the tooling is updated

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented May 20, 2019

@tebeco there is an existing issue #10860 and I'd like to add this feature. However, this is unlikely to make in 3.0.100 release due to this is not a trivial feature.

@dotnet dotnet deleted a comment from tebeco May 20, 2019
@dotnet dotnet deleted a comment from restrellasosa May 20, 2019
@dotnet dotnet deleted a comment from tebeco May 20, 2019
@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented May 20, 2019

@tebeco and @restrellasosa I moved the other issue to #11386

I hope this thread only focus on bigger design of local tools

@hknielsen

This comment has been minimized.

Copy link

@hknielsen hknielsen commented Jul 16, 2019

I like that its part of the dotnet tool, and local tools. But why not part of the csproj? We got away with the nuget file, but now adding a new file for this feels weird? 🙂

@jnm2

This comment has been minimized.

Copy link

@jnm2 jnm2 commented Jul 16, 2019

@hknielsen To me it's a relief that it's not part of a csproj. I make use of this in a PowerShell build script where there is no one csproj that is more particularly connected to the build script than any other csproj.

@tebeco

This comment has been minimized.

Copy link

@tebeco tebeco commented Jul 16, 2019

@hknielsen
dotnet tool should not have any relation with csproj.
dotnet tool are CLI adding it to a csproj would force you to restore all the dependency tree for your business code.
This means :

  • the restore time will blow up
  • that transitive of the CLI will impact your code (or fail the restore)
  • transitive of your code can make the resolution fail

there are scenarios where you want to use a CLI on a repo without restoring the nuget from any

a great example is powershell core
if you take a look at the Dockerfile for the SDK 3.0-previewX you will see that it's now doing a
dotnet tool install to install powershell

there are also scenario where you need the tooling in order to be able to restore.
this is how fake and paket works for fsharp and this is pretty neat

The good news about the file you want to get rid of is that

  • the version of the tools are under source control
  • if you have multiple repo / multi version it is clear
  • if a new teamate arrives, all info are together
  • you don't need to use powershell to remember the custom --tool-path (this was what you had to use since 2.1 because global tool is insane when working with a team or multi repo
@JeremyMorton

This comment has been minimized.

Copy link

@JeremyMorton JeremyMorton commented Oct 18, 2019

Does this hook into dotnet restore or dotnet build, at all?

We have a DotnetCliTool that is a code generator that we run before compiling C#, dotnet restore gets the package and the DotnetCliTool, so someone building a project that uses the generator just needs to run dotnet build and it gets what is needed.

If I understand this, switching to a local tool means that every developer needs to run dotnet tool restore every time they don't have the version of our generator that's checked into the .config\dotnet-tools.json file in our repo. They can't just have build do the right thing, correct?

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Oct 18, 2019

If it is a generator that is part of and always part of the build, writing a Task and deliver it via nuget would be more appropriate.

Local tools operate on a directory level while "dotnet restore" and "dotnet build" operate on project level. Although you could add "dotnet tool restore" as part of your build by configuring your csproj.

@JeremyMorton

This comment has been minimized.

Copy link

@JeremyMorton JeremyMorton commented Oct 18, 2019

We also have a Task in an assembly in a NuGet package for some build number logic. The big drawback of that is that the dll is held open by VS, so restoring when the project is opened in the IDE fails because it can't overwrite the assembly with a new version.

We get around this by having an init1.ps in the package (which only runs nowadays if you manually open the Package Manager Console in VS ☹️) to rename the in-use assembly. This is obviously a bit ugly, and only works if someone manually opens the Package Manager Console in VS.

# Copy task binary file
$taskBinaryFileName = "CreateLocalPackages.dll"
$oldTaskBinaryPath = $taskBinaryFileName + ".old"
$olderTaskBinaryPath = $taskBinaryFileName + ".older"
$destTaskBinaryPath = Join-Paths $solutionDir.Directory $taskBinaryFileName
$packageTaskBinaryPath = Join-Paths $installPath "lib" "netstandard2.0" $taskBinaryFileName

if (Test-Path $olderTaskBinaryPath)
{
  Write-Host "Deleting $olderTaskBinaryPath"
  Remove-Item $olderTaskBinaryPath -ErrorAction:SilentlyContinue
  if (Test-Path $olderTaskBinaryPath)
  {
    Write-Host "Can't delete $olderTaskBinaryPath as it is in-use"
    Write-Host "Deleting $oldTaskBinaryPath"
    Remove-Item $oldTaskBinaryPath -ErrorAction:SilentlyContinue
  }
  else
  {
    Write-Host "Successfully deleted $olderTaskBinaryPath"
    if (Test-Path $oldTaskBinaryPath)
    {
      Write-Host "Moving $oldTaskBinaryPath to $olderTaskBinaryPath"
      Move-Item $oldTaskBinaryPath -Destination $olderTaskBinaryPath
    }
  }
}
else
{
  if (Test-Path $oldTaskBinaryPath)
  {
    Write-Host "Moving $oldTaskBinaryPath to $olderTaskBinaryPath"
    Move-Item $oldTaskBinaryPath -Destination $olderTaskBinaryPath
  }
}

if (Test-Path $destTaskBinaryPath)
{
  Write-Host "Moving $destTaskBinaryPath to $oldTaskBinaryPath"
  Move-Item $destTaskBinaryPath -Destination $oldTaskBinaryPath
}

Write-Host "Copying $packageTaskBinaryPath to $destTaskBinaryPath"
Copy-Item $packageTaskBinaryPath -Destination $destTaskBinaryPath
@AndersMalmgren

This comment has been minimized.

Copy link

@AndersMalmgren AndersMalmgren commented Oct 23, 2019

Hi, we are migrating our enterprise application from 2.2 to 3.0. I found this post telling us to move to local tools if we use dot net CLI

dotnet/announcements#107

So I found my way here. This does not seem like an replacement for donetcli tool at all? With dotnet cli we made a nuget package. And could reference it from project like

<DotNetCliToolReference Include="dotnet.cqsgen" Version="1.0.22" />

And run it like

  <Target Name="BuildContracts" AfterTargets="Build">
    <Exec Command="dotnet cqsgen $(OutputPath)IC.Eko.Core.Contracts.dll cqs.contracts.ts IC.Eko.Core.Contracts.Commands.Command;IC.Eko.Core.Contracts.Queries.Query" />
  </Target>

When a developer pulled latest version of our project from git and build the tool would invoke and do its code generation magic. I dont see how this can replace that?

Thanks

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Oct 23, 2019

@AndersMalmgren as the previous comments stated. If the tool is only used during build, could you try to make it a msbuild task?

At the same time, you could restore a local tool and run it with the similar fashion.

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Oct 23, 2019

We used local tools for our tests. Here is a sample. Although you can start from "dotnet tool restore" instead of "dotnet new tool-manifest" https://github.com/dotnet/sdk/blob/062d9a036ea335833e472d3b3c17174ba62c5c19/src/Tests/Directory.Build.targets#L37

@AndersMalmgren

This comment has been minimized.

Copy link

@AndersMalmgren AndersMalmgren commented Oct 23, 2019

@wli3 I wasnt aware you could restore msbuild tasks using nuget?
I solved it like this for now,

  <Target Name="BuildContracts" AfterTargets="Build">
    <Exec Command="dotnet tool restore" />
    <Exec Command="dotnet SignalR.EventAggregatorProxy.AspNetCore.GlobalTool $(OutputPath)SignalR.EventAggregatorProxy.Demo.AspNetCore.dll $(MSBuildProjectDirectory)\wwwroot\js\events.js SignalR.EventAggregatorProxy.Demo.AspNetCore.EventTypeFinder" /> 
  </Target>

though it feels a bit like a step back from CLI tool that you are forced todo a restore even though its fast becasue of nuget cache

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Oct 23, 2019

@AndersMalmgren we acknowledge this short coming. And we do want to remove this step once some nuget feature is ready

@AndersMalmgren

This comment has been minimized.

Copy link

@AndersMalmgren AndersMalmgren commented Oct 24, 2019

@wli3 Good to hear. I have found a showstopper sadly. We have private feeds. These interfere with dotnet tool restore. Even though the tool is published to nuget.org it tries to use the private feeds and fails

@wli3

This comment has been minimized.

Copy link
Collaborator Author

@wli3 wli3 commented Oct 24, 2019

@AndersMalmgren --ignore-failed-sources should solve it

@AndersMalmgren

This comment has been minimized.

Copy link

@AndersMalmgren AndersMalmgren commented Oct 25, 2019

@wli3 Thanks, that works. But I cant help to think the old CLI solution was more straight forward :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.