Global tools are not found on zsh

[baronfel]

The below example is for FAKE but I can repro with other global tools as well, such as weeknumber. I tried to remove any profile customizations from my profile by running zsh without them and the problem persists. I've provided my PATH as well to show that the global tools dir is in fact in there.

Steps to reproduce

0. Have the 2.1 preview SDK installed (2.1.300-rc1-008673)
1. run zsh with no profile via zsh --no-rcs
2. dotnet tool install -g fake-cli --version '5.0.0-*'
3. fake

Expected behavior

The FAKE cli help should display

Actual behavior

The shell reports that fake could not be found.

Known workarounds

• run the command in bash, or
• run the fake command explicitly, via ~/.dotnet/tools/fake

Environment data

dotnet --info output:

.NET Core SDK (reflecting any global.json):
 Version:   2.1.300-rc1-008673
 Commit:    f5e3ddbe73

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.13-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.1.300-rc1-008673/

Host (useful for support):
  Version: 2.1.0-rc1
  Commit:  eb9bc92051

.NET Core SDKs installed:
  1.0.4 [/usr/local/share/dotnet/sdk]
  2.0.0 [/usr/local/share/dotnet/sdk]
  2.1.4 [/usr/local/share/dotnet/sdk]
  2.1.101 [/usr/local/share/dotnet/sdk]
  2.1.200 [/usr/local/share/dotnet/sdk]
  2.1.300-preview2-008533 [/usr/local/share/dotnet/sdk]
  2.1.300-rc1-008673 [/usr/local/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0-preview2-final [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.0-rc1-final [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0-preview2-final [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.0-rc1-final [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0-preview2-26406-04 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0-rc1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

$PATH:

/Users/chethusk/.rbenv/shims:/Users/chethusk/.nodenv/shims:/Applications/Postgres.app/Contents/Versions/latest/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands

[livarcocc]

I don't believe this will work without a profile, because in that case, we won't be able to load the PATH env variable with a path to the shim folder.

cc @peterhuene @wli3

[baronfel]

That's fair, I just added that in to sidestep instructions to remove any path modifications from my own profile. I have very little in there, though, just some hand-rolled command completions.

[peterhuene]

That's correct, the installer only modifies PATH for bash. The instructions printed when dotnet tool install doesn't find the tools directory on PATH are also for bash.

For zsh, you'll need to modify your .zshrc manually to append ~/.dotnet/tools to PATH.

[peterhuene]

Oh also, when I mean add ~/.dotnet/tools, I mean with an expanded tilde (when sourced from .zshrc it would expand). zsh does not support ~ on PATH like bash does, i.e. it does not expand ~ when searching it.

[baronfel]

So would it be better to use $HOME then? Or is the problem no sort of expansion at all?

[peterhuene]

Should be the same depending on how you set PATH. Would you mind sharing how it was set?

[baronfel]

I didn't set it at all, the ~/.dotnet/tools directory just appeared in there :D From what I can tell, the /etc/paths.d/dotnet-cli-tools file was written with that value by the installer?

[peterhuene]

Ah, gotcha, that is from the installer. Unfortunately macOS is reading paths.d and setting the environment variable for zsh, which doesn't respect the tilde. Using $HOME won't work either because you get a literal $HOME on your PATH.

I don't know if path_helper (the macOS utility doing this) supports this in a way we can enable this universally for all shells.

Having the extraneous entry from paths.d shouldn't hurt things (other than the unlikely event you have a local directory named ~/.dotnet/tools...), but I think you'll still need to manually modify PATH for zsh to work.

[baronfel]

alternatively, is this something where at install-time we know where the tools root is and could write the full path?

[peterhuene]

By default, dotnet tool install -g installs "globally for the user" into ~/.dotnet/tools, so there isn't a system-wide location known at install time we can set with paths.d. If we expanded it for the current user at install time, it'd only work for that user.

Personally, I think we should remove this behavior from the installer and make the first run experience for the user "smarter", where it can detect the shell being used and modify the user's profile accordingly. It would require users to source the profile again or restart their shell after the first time dotnet is run, though.

I looked over the source to path_helper (https://opensource.apple.com/source/shell_cmds/shell_cmds-162/path_helper/path_helper.c.auto.html) and it doesn't evaluate variables or expand ~, so there doesn't seem to be a way to accomplish this with paths.d for shells that won't expand ~ when searching PATH (bash is the only one I know of).

[baronfel]

I don't think that's too much to ask, honestly. Thanks for digging!

[dasMulli]

the oh-my-zsh installer has made some problems already (TL;DR there is no real way to reliably work around how oh-my-zsh sets up a user's environment to perform the PATH modification):
https://github.com/dotnet/cli/issues/8466
https://github.com/dotnet/cli/issues/3063
and maybe a few others

[wli3]

Set PATH is tricky on macOS due to path_helper is simple and there is no paths.d. Since in unix, the user are responsible for the PATH configuration. There should be less magic. Maybe a better way is to detect current shell is zsh and print a tailored message that the user can copy paste like the current fallback bash output

[txchen]

Why not let dotnet tool create symlink under /usr/local/bin, like what npm does? Then you don't need to worry about different shell environments.
And by doing that, you can sudo install tool for every user on the same machine. The current dotnet tool implementation cannot do that.

[wli3]

@txchen do symlink will require sudo for install and uninstall. It blocks user who don't have sudo access. And there is also security implication as the result of that.

I am curious about "install tool for every user on the same machine" case. What is your scenario?

[txchen]

@wli3 At least on my MacOS, npm install which does the symlink does NOT require sudo. I have write permission on /usr/local/bin, I believe this is the default setting.

As for the install tool for every user on the same machine, what I mean is on linux box when you do sudo npm i -g something, it will create symlink at /usr/bin/something, then every non-root user can run something, this is really the "global tool", like apt get install.

What dotnet tool offers now is actually user level tool, not "global".

IMO, you probably should take a look at npm's implementation because that's mature proven solution. If you only modify .bash_profile, of course zsh users like me will not be happy, so why not using symlink on linux/osx?

[txchen]

@wli3 you are right that creating symlink under /usr/local/bin or /usr/bin, will sometimes require sudo permission. I think that's okay because apt-get or npm have the same behavior, I don't think it has security concerns at all.

To solve this maybe you can create a "user-level switch", but by default, dotnet tool install -g should do system global level installation, IMO.

[wli3]

@txchen
The following are several posts in other echo systems discouraging running under sudo.

https://givan.se/do-not-sudo-npm/
https://stackoverflow.com/questions/2119064/sudo-gem-install-or-gem-install-and-gem-locations
https://github.com/calabash/calabash-ios/wiki/Best-Practice%3A--Never-install-gems-with-sudo

[txchen]

@wli3 thanks for the information. I think it is easy to find articles to support your opinion on internet on any topics, I can easily find some counter links but I will not do that:)

Let's go back to the actual issue: the current bug is, dotnet only append its path ~/.dotnet/tools in .bash_profile. This is a wrong assumption because a lot of ppl are using non-bash.

And your proposal is to print some warning in the end of sdk installation, to let user manually tune their PATH settings. I think this is a bad idea, because I never need to do that for other programming languages, it is just not user friendly.

So, are you going to add handling for zsh (.zshrc / .zshenv)? I also think that's not perfect because there are other shell envs. That's why my proposal is to add tools to /usr/bin or /usr/local/bin for osx/linux because that will support every shell env.

It is up to you MS guys to decide what to do. But I am pretty sure if you keep the current implementation, then every zsh user who install dotnet core and wants to try dotnet tool, will end up having issue, then they will google, then they will find this github issue. They will be disappointed, but they can workaround by adding ~/.dotnet/tools to their PATH. Is that what you want? I am not sure :)

[peterhuene]

I'll chime in to add that it isn't completely unheard of for users to have to explicitly add something to PATH to get things like this to work.

For example, $GOPATH/bin for go install, $HOME/.cargo/bin for cargo install (if you didn't use rustup which will append to the profile for your current shell), etc.

That said, I think we can do better here. I'd like to remove the installer logic of adding something to paths.d on macOS and profile.d on Linux entirely. There are meant to be system-wide and we install tools relative to $HOME by default; the fact that bash expands a literal ~ in PATH is limiting this design.

Instead, I think the first-run experience for dotnet should check if the tools directory is already on PATH. If it is not, detect the shell currently used by the user. If bash, append export PATH=$PATH:<tools_dir> to ~/.bash_profile. If zsh, append to ~/.zprofile. If ksh/sh, append to ~/.profile. Inform the user that they need to source their profile for the current shell or open another one. If we couldn't determine the shell or it's unsupported, print out generic instructions.

One caveat is that the first-run sentinel file would then need to be shell-specific, since if I'm using bash and run dotnet the first time, I'd like dotnet to also setup PATH when I run it under zsh for the first time.

[wli3]

@peterhuene we discussed this in early design. Directly edit Unix user's profile is not "idiomatic", even dangerous. And asking user to copy paste adding to path is a good middle ground.

Also that is why we need extra config for auto complete

[dasMulli]

@txchen the actual issue is in the oh-my-zsh installer that users use to install OMZ: ohmyzsh/ohmyzsh#4317, ohmyzsh/ohmyzsh#1359, ohmyzsh/ohmyzsh#3960

by [@]robbyrussell

Happy to discuss this further, but am not convinced yet (as we're assuming a PATH was already defined to their liking and we're just retaining that)

If someone installed zsh using oh-my-zsh AFTER the dotnet CLI set up the global tools, then everything would be fine.

[txchen]

@dasMulli I am pretty sure that on all my machines, I install zsh first, then install OMZ. (I think without zsh, you cannot install omz). So it is not using omz installer to install zsh, OMZ is just a set of zsh config, and all my boxes (ubuntu, debian, macos) have same issue, dotnet tool path is not in PATH.

Please double confirm, and don't blame OMZ too fast.

[dasMulli]

@txchen yes of course, I blame Friday ^^

There's really two issues here:

• Older versions of OMZ fixed the PATH. newer versions don't but this is still an issue for everyone who has been using OMZ for a while.
• Zsh doesn't do tilde expansion which needs some setup here (I think I forgot about this before).

[bilal-fazlani]

It seems it not working on fish shell too.
Workaround: it works after this command:

set --universal fish_user_paths $fish_user_paths ~/.dotnet/tools/

[gabrielbarceloscn]

I´ve ran into the same problem today.
Got resolved editing ~/.zshrc, and putting a new line with:

export PATH=$HOME/.dotnet/tools:$PATH

[peterhuene]

Thanks, @bilal-fazlani and @gabrielrb! That would be the recommended way to get this to work for fish and zsh, respectively, for now.

[bmsantos]

This also has another annoying side effect, even if you are using bash.
Calling a dotnet tools cli app from another C# app will not work with ~/.dotnet/tools.

        Process.Start(
          new ProcessStartInfo(cmds[0], cmds[1]) {
            UseShellExecute = false
          }
        )?.WaitForExit();

Replacing "~" with fully qualified $HOME path, works just fine.

Don't think I need to open another bug for this since it is another manifestation from this same issue. If you still want me to open another issue, let me know.

[lshearer]

I'd be glad to file a separate issue for this as well, but it seems relevant to the discussion:
My global tools (within a bash shell on macOS) are not discovered by either which or /usr/bin/env, and it would seem to be because these tools are also not expanding the ~ added to PATH from /etc/paths.d/dotnet-cli-tools.

I believe this is causing dotnet-script/dotnet-script#409. Manually adding ~/.dotnet/tools to PATH in my ~/.bash_profile fixes this issue, but it's not an obvious workaround given that calling tools directly like dotnet-script works without it.

[ikeough]

I had the same problem as @bmsantos. My workaround is the following:

            // Expand the default paths for tools on different platforms.
            var home = Environment.ExpandEnvironmentVariables("%HOME%");
            if(System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
            {
                home = Environment.ExpandEnvironmentVariables("%USERPROFILE%");
            }

            var process = new Process()
            {
                StartInfo = new ProcessStartInfo()
                {
                    WorkingDirectory = $"./{functionId}",
                    RedirectStandardOutput = true,
                    RedirectStandardError = true,
                    UseShellExecute = false,
                    CreateNoWindow = true,
                    FileName=$"{home}/.dotnet/tools/hypar", // Use the full expanded path to your tool.
                    Arguments="init"
                }
            };

Of course, this only works for dotnet global tools installed in the default location.

[SimonCropp]

@ikeough would 'which dotnet' help you resolve that location ?

[ikeough]

Possibly. I didn't realize that the global tools were installed in the same location as dotnet. And, I think that would only work on *nix right?

[wli3]

In this case manually append $HOME/.dotnet/tools to PATH like adding it to .bashrc is appropriate. Since in the end, unix expect the user to maintain the PATH, and it has constrain for what we can automatically do

[sajidali2444]

I´ve ran into the same problem today.
Got resolved editing ~/.zshrc, and putting a new line with:

export PATH=$HOME/.dotnet/tools:$PATH

its work for me thank you dear

[baronfel]

This is going to become a greater usability concern with the next version of macos, where bash is being replaced by zsh.

[wli3]

Here is my plan:

We won’t have the same experience as bash shell.

We will print the following instruction.

Tools directory '/USERNAME/home/.dotnet/tools' is not currently on the PATH environment variable.
If you are using zsh, you can add it to your profile by running the following command:
 
cat << \EOF >> $HOME/.zprofile
# Add .NET Core SDK tools
export PATH="$PATH: /USERNAME/home/.dotnet/tools "
EOF

Detail reason:
Mac don’t have a good solution to put user path for all user (/USERNAME/home/.dotnet/tools is different for different user). We currently adding literal ~/.dotnet/tools for all users. And that works for bash. But it no longer works for zsh.

At the same time, directly adding new lines to .zprofile is not “idiomatic” for unix. So we can only use existing fallback experience. (bash will have the same experience is you install it via tar.gz).

I find there are 3 ways you can detect the application is running on zsh. (a)$SHELL (b)$ZSH_NAME and use (c)ps -p $$

Only (a) can work in a dotnet program. If you run Console.WriteLine(Environment.GetEnvironmentVariable("ZSH_NAME", EnvironmentVariableTarget.Process)); you will get nothing, even when echo $ZSH_NAME can give you the value in the same session. I don’t know why at this moment. (c) won’t work since $$ is “the current process”. It can only work for scripts. However, the downside of $SHELL is, it will only give you the default shell. That means if you run zsh on top of bash, we will not detect that. I verified in macOS 10.15 VM if it has $SHELL set. (after running chsh -s /bin/zsh)

[DatXN]

I´ve ran into the same problem today.
Got resolved editing ~/.zshrc, and putting a new line with:

export PATH=$HOME/.dotnet/tools:$PATH

This is the answer! Thank so much.

[simonech]

On Catalina, the default shell is zsh... maybe now time to find a way to fix the installation issue? not just writing how to do it manually :)

[wli3]

it is done. the latest sdk will print out instruction to ask you to copy paste scripts in order to add the path to your .zshrc

path_helper no longer can put it on path for zsh magically.

[simonech]

@wli3 the problem is that the message is not visible enough. It says that the installation was successful, so none really care about reading through the info printed.
It should be more visible, maybe in yellow as warning.

[wli3]

I see. I created this bug to track it #3992

[0xced]

Have you considered simply creating the /etc/paths.d/dotnet-cli-tools file with /Users/username/.dotnet/tools (with the current user name) instead of ~/.dotnet/tools?

Pros:

• Works on all shells (bash, zsh, etc.)
• No need to print instructions on how to make cli tools available for your shell, works out of the box

Cons:

• Only works for the user who installed the .NET Core SDK

In my opinion, the pros largely outweight the cons. Not having .NET cli tools work out of the box is a shame.

Even @RickStrahl got bit by this today:

I can't get dotnet tools to launch on my Mac. They're installed & show in dotnet tool list. ~/.dotnet/tools is properly in $PATH, but running any of the Tool commands gets: Command not found.

Going into the tools folder and typing command doesn't work. From Finder it works???

[0xced]

Yet another casualty: waf/CSharpRepl#36

[maxpavlov]

We are a few macOS's in with zsh as a default shell.

Installer still writes
export PATH=$PATH:~/.dotnet/tools
while it should be writing
export PATH=$HOME/.dotnet/tools:$PATH
as zsh does "translate" $HOME, and does not translate tilde.

[jrdodds]

We are a few macOS's in with zsh as a default shell.

Installer still writes export PATH=$PATH:~/.dotnet/tools while it should be writing export PATH=$HOME/.dotnet/tools:$PATH as zsh does "translate" $HOME, and does not translate tilde.

zsh does understand and expand the tilde (~). export PATH=$PATH:~/.dotnet/tools will produce a correct path to the current user home directory.

The dotnet installer is not 'writing' an export statement. It is adding a file in /etc/paths.d named dotnet-cli-tools that contains ~/.dotnet/tools. The files in /etc/paths.d are read by the path_helper tool. path_helper doesn't expand environment variables and it doesn't expand the tilde.

[maxpavlov]

The dotnet installer is not 'writing' an export statement. It is adding a file in /etc/paths.d named dotnet-cli-tools that contains ~/.dotnet/tools. The files in /etc/paths.d are read by the path_helper tool. path_helper doesn't expand environment variables and it doesn't expand the tilde.

Thank you @jrdodds for clarification. I recently installed dotnet on Raspberry Pi 4 and the same problem was observed. However, I failed to find anything at /etc/paths.d, so I wonder if it is written and then "picked up" and erased by path_helper tool, or on arm Debians the process is different (most likely not). Could you elaborate more on how the "path_helper" tool works further after the installer has written to /etc/paths.d? Thank you in advance.

[jrdodds]

@maxpavlov My understanding is that the path_helper tool and the /etc/paths.d directory are unique to Apple macOS.