git prompt causes FISH to start very slowly on OS X

[duganchen]

dugan ) fish --version
fish, version 3.1.0
dugan ) echo $version
3.1.0
dugan ) uname -a
Darwin MacBook-Pro.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64
dugan ) echo $TERM
xterm-256color-italic
dugan ) sh -c 'env HOME=$(mktemp -d) fish'

I had the common OS X issue where FISH would take seconds to load. It would only happen once per reboot. The first time I tried to start FISH after a reboot, it would take seconds to start. If you tried to start it again, it would start instantly. At least, until the next reboot.

I profiled it: and the cause seems to be having a prompt with VCS information in it? This absolutely jumped out at me:

356	7369742	--> fish_vcs_prompt
360	379	---> source /usr/local/Cellar/fish/3.1.0/share/fish/functions/fish_vcs_prompt.fish
19	19	----> function fish_vcs_prompt --description "Print the prompts for all available vcsen"
    # If a prompt succeeded, we assume that it's printed the correct info.
    # This is so we don't try svn if git already worked.
    fish_git_prompt
    or fish_hg_prompt
    or fish_svn_prompt
...
394	5204357	---> fish_git_prompt

I tried switching to a prompt without VCS information, and now FISH starts instantly. Even right after a reboot.

Both profile results ("FISH Profile with git prompt" and "FISH profile with minimalist prompt" are here:

https://gist.github.com/duganchen/7467164add3bcfa6cae0e2f2741ca8bc

This, more precisely, is what's taking so much time on startup:

277	5194027	----> set -l repo_info (command git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD 2>/dev/null)
5193750	5193750	-----> command git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD 2>/dev/null

And no, I'm (obviously) not in a git repository when FISH starts.

If I change fish_prompt_git to call git with its absolute path: well, that makes no difference:

327	4283632	----> set -l repo_info (/usr/bin/git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD 2>/dev/null)
4283305	4283305	-----> /usr/bin/git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD 2>/dev/null

An as for why git is so slow the first time it starts on OS X: I don't know. This takes several seconds, the first time after a reboot it runs:

dugan@MacBook-Pro ~ % GIT_TRACE=1 /usr/bin/git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD
15:25:54.711033 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Applications/Xcode.app/Contents/Developer/usr/bin/git
15:25:54.727587 exec-cmd.c:236          trace: resolved executable dir: /Applications/Xcode.app/Contents/Developer/usr/bin
15:25:54.739887 git.c:419               trace: built-in: git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD
fatal: not a git repository (or any of the parent directories): .git
dugan@MacBook-Pro ~ %

[duganchen]

It's a lot better (but not perfect) after I edit fish_git_prompt and have it launch git from its absolute path in Xcode.app:

277	95933	----> set -l repo_info (/Applications/Xcode.app/Contents/Developer/usr/bin/git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD 2>/dev/null)
95656	95656	-----> /Applications/Xcode.app/Contents/Developer/usr/bin/git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD 2>/dev/null

[duganchen]

After seeing this:

491 4192685 ---> fish_svn_prompt

I managed to solve it.

First, I did "funced fish_vcs_prompt" and deleted the following:

 or fish_hg_prompt
 or fish_svn_prompt

I then copied /usr/local/Cellar/fish/3.1.0/share/fish/functions/fish_git_prompt.fish to ~/.config/fish/functions, and edited the copy, replacing "command git" with "/Applications/Xcode.app/Contents/Developer/usr/bin/git"

FISH now starts up instantly. Even with a VCS prompt.

[floam]

Yeah, it's the svn (and git?) prompt. xcrun has to find the Xcode-provided tools for the full path. This is really slow the first time you run after an Xcode update (or reboot? I don't know). We could maybe figure out a hack to resolve it to the full path ourselves, else fall back to using the /usr/bin path. In my mind git and svn should both take the same amount of time, not sure why svn would be slower unless it's just ran less by a user or something.

Or maybe just disable svn by default on macOS.

[floam]

Potential hack: most people do not have homedirs that are VCS repos. On fish launch, skip doing the vcs prompt but launch no-op SVN and git commands in the background to prime it for the next prompt.

[faho]

How about checking if the xcode path exists and using it then?

[floam]

The Xcode path is going to vary - quickly resolving it ourselves is kind of tricky. Using xcrun to ask for the path is likely to be just as slow the firs time. We could maybe cache it. That still will be slow the very first time.

[faho]

The Xcode path is going to vary

Sure about that? "/Applications/Xcode.app/Contents/Developer/usr/bin/git" looks quite stable.

[floam]

absolutely sure, you could install Xcode.app anywhere - and we don't know which Xcode.app is active, Xcode.app or Xcode-beta.app or.. "my Xcode 8.app"

[floam]

"install" is the wrong word - you can put Xcode.app anywhere, it's portable-ish. It's several gigabytes so people do move it around to save space on / and stuff.

[floam]

It remains to be seen if xcode-select -p which outputs /Applications/Xcode-beta.app/Contents/Developer is slow in that first run circumstance. Annoying to test.

[floam]

(The fact that the xcrun machinery exists for the wrappers installed to $PATH and xcode-select exists and the reason that the first run is slow is precisely because you can put Xcode anywhere and you can have multiple toolchains and versions of Xcode on a system.)

[floam]

Oh, and big one - some people do not install Xcode at all (it's huge) and just install the command line tools package by running xcode-select --install or downloading a .pkg from Apple. That can be at /Developer/CommandLineTools/usr/bin/git or /Library/Developer/CommandLineTools/usr/bin/git or installed to a custom path.

[Darkshadow2]

xcrun -f COMMAND will give you the path to that command. You could use that to cache the path, though there'd be no way to pick up if someone uses xcode-select to change the Xcode being used.

[floam]

Does it short circuit caching? My concern would be if it blocks for a few second the first time you run it anyhow. I'll do some testing

[Darkshadow2]

Yeah, it blocks the first time you use it. I don't think there's any way around that using Apple's tools.

[floam]

I think there's a way around it by taking advantage of something else most likely cached previously and synthesizing the path.

[floam]

e.g. xcrun -f clang | string replace -r clang git compared to how long xcrun -f ls | string replace -r ls git takes. (ls obviously not being provided by the SDK so it's taking as long as a first-run git or anything else.)

[Darkshadow2]

That specific case won't work, as the clang executable is in a different place than the git one.

Maybe xcrun -f xcodebuild? Although I'm not sure if that's only working because it got cached earlier.

[floam]

Whoops, busted example.

[Darkshadow2]

Huh, interesting. If you give xcrun -f an invalid command it gives you this error:

xcrun: error: sh -c '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk macosx -find idonotexist 2> /dev/null' failed with exit code 17664: (null) (errno=No such file or directory)
xcrun: error: unable to find utility "idonotexist", not a developer tool or in PATH

So it looks like it's not even actually xcrun that's looking up the paths...

[floam]

Nice!

[Darkshadow2]

Ah, but if you use xcodebuild -sdk macosx -find COMMAND directly, there's no caching. It always takes a bit to find something.

[floam]

the dylib that causes the wrapping takes SDKROOT into account as well, so it should be possible once we have a way to quickly get the SDK path regardless of cache (xcrun --show-sdk-path?) we can just export something and a bare git or anything else should work. I think. Or maybe not.

[duganchen]

These paths aren't going to change often, so I'd propose baking the xcrun results into "set -U" universal variables instead of running xcrun each time FISH starts.

And for cache invalidation: I'd just check if the path still exists, either on startup on when you display the prompt.

The following script, which is meant to be executed with an ampersand, is what I have in mind:

if not test -x $GIT_PATH
    set -U GIT_PATH (xcrun -f git)
end

[drewish]

I've noticed another issue that seems to be slowing down the prompt is locating svn (I removed my prompt since it had some unrelated information displayed):

$ fish -p FILE -c 'fish_prompt'
$ sort -n FILE | tail -n 8
6655	6655	---------> read -lz key value
10674	10674	-----> command git rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD 2>/dev/null
10903	10903	---------> command git diff --staged --name-status | string match -r \\w
12420	12420	---------> command git rev-list --count --left-right $upstream...HEAD 2>/dev/null
12935	12935	--------> command git symbolic-ref HEAD 2>/dev/null
18197	18197	---------> command git diff --name-status 2>/dev/null | string match -r \\w
23973	23973	---------> command git ls-files --others --exclude-standard | wc -l | string trim
594636	594636	-----> command svn info 2>/dev/null
$ svn info
svn: error: Failed to locate 'svn'.
svn: error: The subversion command line tools are no longer provided by Xcode.
$ time svn info
svn: error: Failed to locate 'svn'.
svn: error: The subversion command line tools are no longer provided by Xcode.
        1.02 real         0.35 user         0.40 sys

[drewish]

Oh looks like my SVN issue was fixed by #6681

[peterstory]

I think this issue can be closed, based on the fix in #6681 having been incorporated into 3.1.1 and later. Fish is speedy again! 🎉

[ridiculousfish]

Cool! Thanks!

[faho]

It's... not fixed?

The svn prompt was one issue, the initial problem however was that macOS does weird things with git, and I don't think that's changed?

I mean I'd personally be entirely okay with calling that an OS bug, but then I'm not using macOS.

[floam]

Yeah this isn't fixed.

[floam]

Update since the last time I commented here: this happens specifically after each reboot, because that clears $TMPDIR. libxcselect.dylib is regenerating the cache at $TMPDIR/xcrun_db.

It specifically takes so long because syspolicyd is spending a lot of time in the security framework checking code signing across the Xcode bundle.

[floam]

So there is a practical workaround: if $TMPDIR/xcrun_db doesn't exist, we run something like git --version& backgrounded. That'll kick off the cache being rebuilt without making fish appear to hang in the meantime. Likely the user won't even notice that a lack of VCS status in their prompts printed during those ten seconds or whatever, since most home directories are not git repos.

[floam]

I addressed this in PR #9345.

Even if fish is launched with a CWD of a git repo, the behavior is pretty pleasant now, with the prompt and command line updating as soon as the cache file exists, the VCS component just kind of pops in late.