git path completions are slow

[mqudsi]

fish, version 2.6.0-77-g5e94650

Path completions when completing git commands under WSL are very slow (the shell hangs when I <tab> to complete a local path; I'm not sure why since normal path completions autocomplete just fine and git commands execute quickly with no apparent issues.

Any hints on how I can help debug this issue?

[faho]

Which command exactly are you trying?

The git completions are rather smart, and e.g. for git add, it will try to show only files that are modified or untracked, in the entire repository with the path changed so it works from anywhere in that repo.

That pretty much requires calling git, which is pretty much guaranteed to take a bit longer than normal path completion.

I'm writing the rest assuming it's about git add.

After you've tried to complete any git command (which loads the completions), can you try

__fish_git_add_files
echo $CMD_DURATION

?

Note that on linux (where git is faster in general IIRC), this will benefit massively from caching, so you might want to invalidate the cache between runs (e.g. the first call might actually take a second, the next one will take 12ms).

It is also possible that this just isn't fixable - I've just tested the performance of __fish_git_add_files, and 90% of the time there is spent in a call to git ls-files. Unless there's a faster way to ask git for the same information, the only way to speed it up is to remove the feature.

[faho]

Okay, sorry, of course there's a nicer way to profile this!

# Note: The arguments to `complete -C` are weird - they pretty much need to be given in that exact form
fish --profile /tmp/some_file -c 'complete -C"git add "'
sort -nk 2 /tmp/some_file

[bennoleslie]

FWIW, I was having the same problem on OS X. Using the profiling above, I found the issue was shelling out to git config 62 times was the root cause of the problem.

I worked-around this be removing the alias support in __fish_git_using_command.

[faho]

Using the profiling above, I found the issue was shelling out to git config 62 times was the root cause of the problem.

@bennoleslie: Obviously that's a bit slow, but on my system it's essentially not an issue. However, I have 3 aliases. I'm guessing you have a bunch more.

I worked-around this be removing the alias support in __fish_git_using_command.

Yes, that'll do it. The thing is to complete aliases like the commands they are aliasing (which isn't ideal - they should be completed like they were given in full, but that requires #1976), we need to somehow map the string that is currently in command position to the command.

I have a solution that should be a bit faster (one git config call to rule them all), but it's a bit hacky - it requires global variables.

I'll make a PR later.

[bennoleslie]

@faho no, I only have 3 (which is why disabling was not a big deal). The number of shell outs doesn't seem to be related to the number of aliases though. I tested this with all my aliases removed.

I mentioned on the mailing list a solution would be something along the lines of calling git config --get-regexp alias once at the start of the completions scripts, parsing the output and then using that data structure to do lookup in __fish_git_using_command rather than shelling out each time.

I'm surprised how 62 shell outs can be so quick for you though. I'm interested, if you do:

fish --profile /tmp/some_file -c 'complete -C"git add "'
grep "set -l aliased" /tmp/some_file | wc -l

Do you see 62 executions? Or is calling out to git that much faster on Linux for some reason? (I'd be surprised).

[mqudsi]

Thanks for the speedy input. This is what it came back with:

profile.txt

[faho]

I mentioned on the mailing list a solution would be something along the lines of calling git config --get-regexp alias once at the start of the completions scripts, parsing the output and then using that data structure to do lookup in __fish_git_using_command rather than shelling out each time.

I had the same idea - that's pretty much what #4118 does 😄.

Do you see 62 executions? Or is calling out to git that much faster on Linux for some reason? (I'd be surprised).

Yes, 62 executions totalling about 58ms (grep "set -l aliased" /tmp/git4.prof | string replace -r '\d+\s+(\d+)\s+.*' '$1' | paste -sd+ | bc).

@mqudsi, @bennoleslie: Please try #4118.

[bennoleslie]

@mqudsi FWIW, that profile looks very similar to mine. 982ms total, 858ms resolving 64 aliases.

I don't think this is a WSL specific issue.

[mqudsi]

@faho that branch exhibits the same delays.

[faho]

@mqudsi: That's surprising. Have you started a new shell? If so, can you upload a new profile?

[mqudsi]

whoops, I kind of forgot to make install ;) Let me try again.

[faho]

@mqudsi: It's a script-only change, so you don't actually need to make install. You can simply grab the file and save it as ~/.config/fish/completions/git.fish.

[mqudsi]

Yeah, that's much faster.

[faho]

Okay. I'll let the PR sit for a week or so to figure out any issues, then I'll merge.

@mqudsi: It would still be nice to get a new profile - maybe there's some low-hanging fruit there.

[mqudsi]

Here's the latest:

profile.txt

[bennoleslie]

@faho Yeah, that fixes it. I'm still a bit surprised that my sub-shell takes 80ms, and yours less than 1ms!

[faho]

Okay, the good news is that there's no more low-hanging fruit in that path - the only way to get it faster would be to refactor the entire completions script to go like this:

function __fish_complete_git
    if set -l cmd (__fish_git_command) #hypothetical function that returns the current de-aliased command
        switch $cmd
            case add
                 __fish_git_add_files
            case checkout
                 __fish_git_reflog
                 __fish_git_branches
                 # and so on....
         end
     end
end

complete -c git -a '(__fish_complete_git)`

which works around much of our completion system (option completion would have to be done manually etc), but avoids checking the command more than once.

I don't really want to do that.

[faho]

I'm still a bit surprised that my sub-shell takes 80ms, and yours less than 1ms!

I'm actually not surprised that git is fastest on linux, considering the author.

In fact, #4118 is actually slightly slower here (by about 7ms). Edit: Sorry, it actually only takes half the time. Still a lot better than a tenth, but absolutely an improvement.

Note to myself: Dude, your configuration only loads test/completions if the shell is interactive!

[faho]

I'm closing this since it seems #4118 is enough.

I still have a few more ideas to speed the git completions up, but all of them would make the code uglier, and there doesn't seem to be a great need to do so.

[unthingable]

New fish user, 2.7.1 on MacOS, assuming the fixes mentioned here are in and probably improved upon. Yet, git completions are very slow, exact same problem as OP described (git add REA<tab> taking forever to find README.md on a newly init'ed repo).

"The git completions are rather smart" — maybe that's not a good thing? One of the first things I had to do to make it bearable to use was to change fish_prompt:

#    printf '%s ' (__fish_vcs_prompt)
    printf '%s ' (parse_git_branch)

...
function parse_git_branch
	/usr/bin/git branch ^ /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ (\1)/'
end

I.e., just the branch name is good enough, I'm not too lazy to type git status when I need to. The lag was annoying enough on its own, an active Python VE (via sourcing activate.fish) seemed to make it considerably worse.

Why is bash-completion from brew able to complete instantly (and in several years of use it's never been not smart enough)? Can we configure fish's git completions to be stupider but faster (or maybe even fall back on bash-completion)?

Sorry for the rant/frustration report from the field. Still having problems with git completions and having to fall back to bash in a pinch.

[faho]

One of the first things I had to do to make it bearable to use was to change fish_prompt:

I find this rather confusing. First of all, the git prompt is different from the git completions. It can be slow or fast, and the completions can be slow or fast.

And by default, the git prompt does two git calls:

• rev-parse --git-dir --is-inside-git-dir --is-bare-repository --is-inside-work-tree HEAD

and

• symbolic-ref HEAD

These are pretty much the cheapest possible - the first one is necessary to figure out if we should even do anything else. On my system these take about 5ms each. That's about as long as your parse_git_branch takes, and it works in more circumstances.

If it does anything more, then that's because you've enabled more! The git-prompt has a whole bunch of configuration variables - e.g. $__fish_git_prompt_show_informative_status. And enabling those will do more, and under certain circumstances it might take a while. That's why these are options.

---

Okay, now on to the completions.

assuming the fixes mentioned here are in and probably improved upon

Not just these, but a bunch more - see e.g #4673.

Why is bash-completion from brew able to complete instantly

Because it mostly just falls back on completing files. E.g. git add <TAB> often just completes files in the current directory. Fish tries to offer files where that operation actually makes sense. E.g. git add unmodifiedfile is useless, but git add newfile or git add changedfile isn't.

Can we configure fish's git completions to be stupider but faster (or maybe even fall back on bash-completion)?

Sure. Create a file called "git.fish" somewhere in $fish_complete_path (e.g. ~/.config/fish/completions) ahead of where ours is, and make your own complete script.

[unthingable]

@faho, thank you for a balanced response to an unbalanced rant.

The prompt mention is indeed not related to completions, that was more to add weight to the suspicion that by default the git machinery tries to do too much. Then again, I simply selected the "Informative Vcs" prompt (which, I'm guessing, uses different defaults) and didn't know to look at $__fish_git_prompt_show_informative_status, next time I will.

FWIW, I don't see bash-completion completing unmodified files. Also, it only searches in the current directory and does not descend into subdirectories, whereas fish completion seems to make a full traversal from the root (where, in my example, I currently have 27487 untracked files I don't really care about).

This is not to say that one behavior is wrong and the other is right, but to present a use case where the default behavior is boldly inefficient (over 10s on my machine in this case) and to challenge the design assumptions while suggesting for an easier way to opt out than "write your own git.fish". Yours is 1000+ LoC, of which ~120 is __fish_git_files, with 41 contributors to which an average new user is no match. :)

FWIW, this new user is enjoying fish immensely and appreciates the work that went into it.

[dracos]

Hi, I'm just trying out fish (thanks!) and it's all nice, apart from typing git log -p per and then hitting tab takes 3-4 seconds before completing to perllib. I've run the profile command suggested above (as time fish --profile /tmp/some_file -c 'complete -C"git log -p per"') on my installed fish (2.7.1, macports) with/without putting the current git.fish in completions, doesn't appear to make any noticeable difference.

Here is the output with the current master git.fish: some_file.txt - hopefully that might be helpful in finding out what is making it so slow? It looks like a git branches call takes 1.5s and the completion calls it twice – first __fish_git_refs is called directly, and then it is called again from inside __fish_git_ranges.

It looks like it is this line of git.fish:

fish-shell/share/completions/git.fish

Line 755 in 0f4126c

complete -c git -n '__fish_git_using_command log; and not contains -- -- (commandline -op)' -a '(__fish_git_refs) (__fish_git_ranges)'

which calls both refs and ranges, which then each call branches, doubling the call time.

A git branch --no-color -a on my repo only takes 0.1-0.2s, so it is the rest of __fish_git_branches that is expanding it to 1.5s, and then doubled.

[faho]

Here is the output with the current master git.fish

@dracos: Not quite true - going by your output, that is running something before f3f2d2d - which should speed just that case up. Please check your $fish_complete_path for another git.fish.

A git branch --no-color -a on my repo only takes 0.1-0.2s, so it is the rest of __fish_git_branches that is expanding it to 1.5s, and then doubled.

There's something seriously wrong. Please check type string.

[dracos]

Thanks for the really quick response! Two separate things:

1. Yes, somehow I wasn't using the master git.fish, though I really thought I was, hohum! Looks like it was at 6c0f31d. Using the actual master, it is quicker but still around 2 seconds, here's the new output profile: some_file.txt

2. string is a builtin, but definitely appears to be slowing everything down:

~/P/m/F/fixmystreet (master|…) § type string
string is a builtin
~/P/m/F/fixmystreet (master|…) § function timeit
                                     perl -e'use Time::HiRes; use v5.14; say Time::HiRes::time();'
                                     eval $argv
                                     perl -e'use Time::HiRes; use v5.14; say Time::HiRes::time();'
                                 end
~/P/m/F/fixmystreet (master|…) § timeit "command git for-each-ref --format='%(refname)' refs/heads/ refs/remotes/ >/dev/null"
1527695535.47777
1527695535.68287
~/P/m/F/fixmystreet (master|…) § timeit "command git for-each-ref --format='%(refname)' refs/heads/ refs/remotes/ \
                                         | string replace -r '^refs/heads/(.*)\$' '\$1\tLocal Branch' \
                                         | string replace -r '^refs/remotes/(.*)\$' '\$1\tRemote Branch' >/dev/null"
1527695557.92773
1527695558.77275

[faho]

Using the actual master, it is quicker but still around 2 seconds

Either you have an absurd number of branches - thousands of them (I'm regularly testing this with 1600 branches), or, more likely, you're not actually running fish from git master, meaning you don't have 2de38ef. That should speed this up by a factor of 4 or so (i.e. reduce the time taken to 25% of what it was).

[dracos]

git branches -a returns 3358 rows – the repo itself has 600 odd, but the checkout does currently have 12 overlapping remotes, and therefore a number of 'duplicate' branches.

No, I was only using the master git.fish not all of fish, I'm using 2.7.1 as mentioned. I'll try a recompile if I have time, thanks for the pointer. Otherwise will have to cope and await 3, thank you :)

[dracos]

I can confirm it is much quicker on the current master of fish, yes.

[masukomi]

Either you have an absurd number of branches - thousands of them (I'm regularly testing this with 1600 branches)

just wanted to provide some level setting here. I work with ~40 developers. I have 26 of them set up as remotes on my repo, because over the years it has been prudent to check out the topic branch of 26 coworkers and collaborate with them. We follow a pretty standard process of making a topic branch for each pull request. People tend to not delete their topic braches after merge, because ...why bother? Branches are a teeeeny little file with nothing but a treeish in it.

git branch -a | wc -l returns 25,516 branches.

so I don't think "1600" is anywhere near a valid upper bound for testing. 26 people asking you to comment on their code / help with their work over the course of a single employment is not a lot.

I've been using fish for years now and the slowness of tab completion in git repos is so painful that i'm seriously considering uninstalling fish and going back to bash. I keep having to remind myself to use arrow instead of tab then delete and retype the end because otherwise i've got literally about 10 seconds worth of delay every time i accidentally hit it to tab complete file paths. I'm working through debugging it based on the stuff above but a) i've got better things to do b) i had some pretty sweet git integration going in bash and it was never painful.