Git prompt performance

[quicknir]

Decided to try out this theme, was very impressed by a number of things. However, a deal breaker is the performance of the git segment. Even in a pretty small git repo, there is noticeable lag after each time I hit enter until it is repainted (not present outside of git repos). On the other hand, sorin theme from prezto (which I use a very small fork of) is 100% instant. I'd suggest adopting some of the implementation from here: https://github.com/sorin-ionescu/prezto/blob/master/modules/prompt/functions/prompt_sorin_setup.

[quicknir]

As an update, I was basically able to get this to work by: 1) copying and pasting all the code from sorin theme, except deleting the lines of code after # Define prompts. 2) I took the hello world segment example, and I simply changed it so to output _prompt_sorin_git instead of content. That's it. And with this modification, the git info in powerlevel9k is super fast.

Obviously, it would be nicer to integrate this properly into powerlevel9k so you get the nice icons, customizability, etc.

[bhilburn]

@quicknir - Thanks so much for the feedback! Yes, the vcs segment has been a weak point of P9k for a while, now. It sounds like you have done some really great work, though!

Would you mind sharing what you've done? Feel free to either just post a git diff here, or even file a PR so I can see a changeset that you have tested and verified is working.

I would actually like to get this integrated as soon as possible - if we have a clear path to improving the vcs segment, I'd like to get it done.

[quicknir]

@bhilburn Okay, I'll give this a shot, it may take a few days before I can look at it. You're giving me too much credit, I didn't really do anything but notice that another person (sorin) had done good work :-) I also want to think about customizability for git; traditionally with these things, they only show e.g. * to indicate that there are modified files, but I much prefer e.g. *2 to indicate there are two modified files. I can send a screenshot of my setup if you like, but probably best would be to make it easy to customize (also, being able to customize color especially is rather crucial).

[quicknir]

Okay, so I sat down to look at the code today. There are two non-trivial complications. The first is that the approach used by the sorin theme in the prezto package, uses a fair bit of code from another prezto module (prezto has a dedicated module for retrieving git info, it's pretty nice. However, that code might be very similar to the code you already have for extracting various bits of information, so I don't know to what extend you'll want to duplicate that in this repo (or even how that works in terms of license, etc). The thing that really drives the performance in sorin is being async. So in principle that can be fused with some of the existing functions.

The second is that the current approach supports both git and mercurial, whereas again, the sorin stuff only supports git.

Basically, while getting a nice git segment up and running for me personally as a prezto user is trivial, getting git+mercurial segment that works in powerlevel9k independently of prezto is quite a bit less trivial. At its most hacky, you could basically copy in all the code you need from prezto and just bifurcate the git and and mercurial implementations, but I doubt this is what you would want.

Anyway, I'm curious to hear your thoughts generally. I may, to be honest, not be the guy for the PR if it is something non-trivial: I love zsh for interactive stuff but actually really dislike writing scripts in it (I'm one of those just do it in python guys), and my skills are not very sophisticated, so in the past whenever I've done things to get my setup working it's been very time consuming (e.g. I was skimming your code just now and saw (( ahead )) && gitstatus+=( " $(print_icon 'VCS_OUTGOING_CHANGES_ICON')${ahead// /}" ), I really have trouble understanding exactly what's going on even in this relatively simple code).

Alright, let me know your thoughts!

[shibumi]

I use the git prompt for over a year now. I never had any performance issues.

[ncolton]

I'm in the camp of having experienced performance issues. If I cd into a particular git repository, the shell appears to hang. If I use Ctrl-c, then type a command, I eventually get response but if I do not Ctrl-c then nothing is output from the shell even if I wait, say, 15 minutes (literal terms; I was trying to troubleshoot from another ssh connection to the host to hopefully find what was hanging the shell).
If I disable the vcs portion of the prompt, responses are close enough to instant.
I wish I knew how to provide more useful information for this, but given the history so far in this issue it may not be necessary.

[diraol]

I do see a performance decrease while under a git versioned directory.

[bhilburn]

So, there's a bit of a long history here. I'll try to explain where we currently are and what needs to get done. The fact of the matter is that the vcs segment is arguably the most important segment in P9k, and it is also the poorest performing.

The issue, fundamentally, is that querying the local git repository is expensive. There are two types of git code in P9k. Some is provided by ZSH's VCS_INFO module, and some is stuff we have coded. In both cases, the issue is that some git commands can take a very long time to return results - especially if the repo is large or has a long history. If you take the git commands that P9k and VCS_INFO are using and run them yourself, you'll see similar results.

What many other themes have resorted to doing is forking these git commands off into another process, drawing the prompt before it's ready, and then updating the prompt when the information shows up.

We've known that this needs to get done for a while now. In fact, @ayyjayess and @Tritlo first started working on P9k async stuff in 2016, in #263. @ayyjayess pointed out that the zsh-async library exists, and we even had some discussion with its creator, @mafredri, about integrating it with P9k. This work continued in #331.

Around this time, @dritter began a massive update effort in which he was converting all existing segments to be asynchronous using a self-built async functionality - you can see this work in #344. This was a massive undertaking, and @dritter poured tons of time into it through mid 2017. But, life happens, everyone was busy, and things slowed.

In April of last year, @onaforeignshore also started hacking on the zsh-async integration, picking up #331 and continuing it in #504. He got it to the point that it worked with experimental stability, and asked for my help. To be honest, I just haven't gotten to it.

Your findings, @quicknir, that the Sorin theme performs better using an async capability totally lines up with what we would expect.

We very much need to move the vcs segment, at the very least, to some sort of async library. One of the top goals of P9k is to keep things speedy, and right now the vcs segment absolutely kills that. The major issue, as you probably surmised from the above, is simply manpower. I'm insanely busy, and it's difficult for me to find time to sit down and just do something like this. @dritter is in a similar boat. Both @dritter and @onaforeignshore have done a ton of work on async, but it's quite likely that their work has bitrotted to the point, now, where it would need some time to stand it back up again.

I think in the near-term, our best solution is to go back to the zsh-async library and implement it for the vcs segment. It doesn't need to be for every segment - just vcs. And we need to not include a ton of other work in that PR. Just the port of vcs to zsh-async. Getting that done would have a tremendously beneficial impact for P9k.

Back to the issue of time, realistically, I just need help getting this done. Right now I simply don't have the time to do it. If anyone is up for taking this on, I am happy to help as I can, and I'm sure plenty of others here will volunteer to help solve difficult hurdles and provide reviews - but the bulk of the development will need to be done by you.

So, anyone up for it? =)

[robobenklein]

Can I request an interim patch?

Just add a 10-second timeout to the git prompt and after that just draw it with timeout as the text?

I've been sitting at this blank prompt for long enough to search through this repos issues and post this comment :/

(RPROMPT shown because I tried to ctrl-c, but to no avail. )

[dritter]

@quicknir @robobenklein which git version do you use? The VCS segment relies heavily on git, and old git versions seem to be quite slow on some queries.

Btw. Did you know, you could disable some of the checks to see if one of them is causing significant lag? I mean, the segment as a whole is slow -> this should be fixed with async help some day. But, for instance, having a lot of untracked files slows down the prompt significantly:

The single callbacks for git are defined via POWERLEVEL9K_VCS_GIT_HOOKS=(vcs-detect-changes git-untracked git-aheadbehind git-stash git-remotebranch git-tagname).

Regarding the untracked files:
It might be worth a shot to pick up @tupton 's suggestion from #60 (comment)

You see three iterations of each command. The first three are with git status, the latter ones with git ls-files.

The submodule problem could be tackled with git 2.11.

[robobenklein]

The specific case that I had hit was on a 4.6GB repo of binary files with all files tracked and unmodified. The real slowdown was reading that 4.6GB from the disk at usb-drive speeds. Once it was in memory disk cache, there wasn't a problem.

[yeganer]

@bhilburn Hey, I just stumbled onto this issue. I'm using this theme inside WSL on windows and I'm facing the same problem (~5-30s prompt load time). If I had some spare time (I really don't have it but this issue is really annoying), where would I start? Continue one of the many PRs you linked or start from scratch?

[quicknir]

@dritter I use a reasonably new git, but at any rate these are all band aid solutions, because fundamentally: a) git just doesn't stay fast as repos get even moderately large, and b) it only takes a very small slowdown for it to be noticeable. At any rate I've been staying on sorin for now; it looks like lots of people have jumped in and started looking at async and performance more seriously so I'll see how things evolve. I'm worried that even with async changes, all of the looping and checking of variables is still causing too much slow down, because even with the git prompt completely gone this prompt is very noticeably slower than sorin, but that's a topic for another issue/day.

[bhilburn]

Hey @yeganer - Sorry, I somehow missed the notification when you posted here.... four months ago =(

@ryanwhocodes @yeganer - I super appreciate you guys offering to dig into this. So, we are working on making the v0.6.6 release ASAP. This release will be the final release using the existing codebase.

Once that release is done, we will migrate to what we are calling the v0.7.0 codebase, for lack of a better name. It is heavily refactored & reorganized, and has a lot of new features. One of these features is the concept of "generators", which @onaforeignshore came up with, which will basically allow us to ship different backends in p9k to use for rendering the segments. The default one in that branch, currently, is the same as the current codebase. But, the next one should be async. In fact, if you check out the branch onaforeignshore-code_separation, you can see I've already pulled the zsh-async library into p9k as a subtree.

So, the next thing that needs to happen is that we need to actually use zsh-async (there is a "generator" for it in that branch, but it's the same as the default generator right now). If you're up for it, making a new generator using zsh-async is the next critical step.

This would be a really significant contribution to p9k, if you're up for taking it on. Me, @dritter, and @onaforeignshore are here to help and guide you along the way, but we're spending most of our time right now trying to push v0.6.6 out the door and then start to prepare v0.7.0, and simply don't have the manpower to also take that on ourselves at the moment. Let us know if you're up for it, and if you can, be sure to join the Gitter and chat with us =)

Tell your co-worker to hang in there just a little bit longer - we're almost there =)

[onaforeignshore]

@bhilburn Correction. 070-registerSegment already contains a working zsh-async generator. It just needs testing and tweaking.

[ghost]

@bhilburn Great to hear there's so much progress with P9k in terms of structure and a start has been made with adding the async feature to improve performance. It sounds like there is a async generator in progress already, so let me know if you still need help with anything.

[phcerdan]

Hello, this is a great goal, just to want to check how is it going, I would be glad to help testing it. Thanks!

Meanwhile, changing the default:

POWERLEVEL9K_VCS_GIT_HOOKS=(vcs-detect-changes git-untracked git-aheadbehind git-stash git-remotebranch git-tagname)

to:

POWERLEVEL9K_VCS_GIT_HOOKS=(vcs-detect-changes)

speed up things quite a bit (missing features, but still giving the most important info). Maybe this helps somebody else waiting for the ongoing PR solving this issue.

[bhilburn]

@phcerdan - We're still making progress, here! The v0.7.0 codebase now exists on the next branch! If you wouldn't mind checking out next and letting us know how it works for you, that would be SUPER helpful.

The next major step is that @onaforeignshore is implementing the zsh-async based segment generator, which will allow for asynchronous segment processing. We also expect this to have a significant impact.

[thromera]

@bhilburn I've been experiencing the same exact slowness on all the VCS code, and while the next branch is a bit better, it's still super slow.

I can confirm that @phcerdan change in VSC Git Hooks actually fixes the issues (but misses some features, I'm kind of fine with that.)

Like, it went from 1 second to load the prompt to almost instant. It's a huuuuge gain of time (Imagine each time you hit enter, you have to wait 1 second to see the prompt properly. So annoying.)

[bhilburn]

@phcerdan @Erowlin - Wow, really? It had that much of a difference?

That's interesting, because those changes don't have anything to do with P9k -- those are specific to OMZ. That's super helpful information.

I would be curious to know if you see similar delays with those OMZ turned on with other P9k themes that also invoke ZSH's VCS_INFO?

[dritter]

Thanks for that numbers @Erowlin ! So the biggest impacts here are checking for untracked files and stashes.

Btw. just to be sure: I recently made some speed improvements to the VCS segment - especially when checking untracked files - You checked 5200a1b right? And how does this compare to the version before I did the speed improvements (next@ fa4e336) ?

And I prepared a PR with better performance when checking stashes. Can you give #1058 a shot?

[thromera]

@dritter I wasn't actually, just checked out, I will redo the benchmarks :). Just by feelings, it seems to be way faster. Let me run some benchmarks :)

[thromera]

Edit: Removed the screenshots, they were inaccurate because I was using the P9K_xxx variables instead of POWERLEVEL9K_xxx. The correct timings are here: #732 (comment)

[onaforeignshore]

Looks like vcs-detect-changes is the bottleneck

[thromera]

@onaforeignshore I've updated the screenshots to use the big repository instead of powerlevel 9K.

Very weird: The changes to the P9K_VCS_GIT_HOOKS seems to be ignored, as all the timing are similar. Any idea?

Edit: P9K_VCS_GIT_HOOKS is actually on the next branch. On master branch, we have to use POWERLEVEL9K_VCS_GIT_HOOKS.
I switched back to master, redoing the benchmarks.

[onaforeignshore]

All timings are similar except for vcs-detect-changes

[thromera]

Last commit on master in powerlevel9k repository:

• 78df520 (HEAD -> master, origin/master, origin/HEAD) Merge pull request Speedup vcs master2 #1055 from dritter/speedup_vcs_master2

(Nothing)

[thromera]

Git stash is actually a bottleneck here.
With git-stash: 1.71s (user)
Without git-stash: 1.04s (user)

git-untracked is also a small bottleneck
With git-untracked: 0.98s (user)
Without git-untracked: 0.68s (user)

[onaforeignshore]

True

[dritter]

Thanks @Erowlin . Btw. 1.76 is system. User is 1.50s.
So you are down from 1.50 secs (old master) to 1.35 (78df520).

Could you try #1058 as well? There I optimised git stash.

[dritter]

I just merged #1058 into next. Didn't want to wait longer as this contained a Bugfix..

@Erowlin could you update and redo the tests?

[anuj-modi]

I am also experiencing slow prompt reload times when I am in a git repository. Is there any work still being done on this?

[shibumi]

@anuj-modi well, you will never reach the same prompt reload times as with zsh, because of a lot additional checks for stashes, etc... powerlevel9k staff can just work on making it more efficient. (in before: writing backend in C, rust oder golang :D )

[robobenklein]

I've come to realize that git information is too much work to be done in between prompt redraws, it's far too dependent on disk read or access time that it's inconsistent and sometimes (like in your cases) awfully slow.

I think the only true solution is to have it async or have the info made available in a different way. Personally I've had success moving everything heavy (requiring disk, network, binary calls, etc) into a worker using zsh-async.

You can see it's a common theme for development:

#344
#504
#331

I've gotten zsh-async prompt segment generation working, but I don't believe there's much hope for getting it to work on Windows such as Msys or MinGW. ( mafredri/zsh-async#26 )

I don't know if Powerlevel9k intends to support that or not... (I'm not an active dev here - I just pop in once in a while), but I will answer any questions about how I got zsh-async working.

[dritter]

I don't know if Powerlevel9k intends to support that or not... (I'm not an active dev here - I just pop in once in a while), but I will answer any questions about how I got zsh-async working.

Yes, we would love to have async capabilities in P9K. In the past we tried to change the whole prompt into async rendering, but I am not sure if this is a good idea for every segment..
Regarding Windows, we could try to detected that and use process substitution combined with a callback handler to work around this as agkozak/agkozak-zsh-prompt does.

@anuj-modi Same questionnaire for you:
On which version of P9K are you? Was the prompt faster before you updated P9K? How big is your repo? Could you quantify "slow" (time ( repeat 10; do; prompt_vcs "left" 1 false > /dev/null; done;))? Do you use a lot of git submodules? Do you have a lot of untracked files?

[diraol]

P9k version: dcd7718

python-mode repo:
Repo size:

> git-sizer
Processing blobs: 4126                        
Processing trees: 2821                        
Processing commits: 990                        
Matching commits to trees: 990                        
Processing annotated tags: 25                        
Processing references: 99                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest checkouts            |           |                                |
| * Maximum path depth     [1] |    10     | *                              |
| * Maximum path length    [1] |   109 B   | *                              |

[1]  767cb1ccd9f0a077062a9d0931c26744ab86bb0d (ce7882b16282ce38e8d219cb79946b3599416316^{tree})

"time": 0.36s user 0.31s system 103% cpu 0.644 total

p9k repo:
Repo size:

Processing blobs: 4291                        
Processing trees: 4073                        
Processing commits: 2397                        
Matching commits to trees: 2397                        
Processing annotated tags: 12                        
Processing references: 32                        
No problems above the current threshold were found

"time": 0.37s user 0.36s system 105% cpu 0.695 total

[romkatv]

If anyone is using the master (stable) version of Powerlevel9k and seeing slow prompt rendering, give #1170 (comment) a try. There are instructions at the end of the comment for how to try it in your current shell without any permanent changes. Just copy-paste the command and see if your prompt becomes more responsive.

[mcfiredrill]

I think the only true solution is to have it async or have the info made available in a different way. Personally I've had success moving everything heavy (requiring disk, network, binary calls, etc) into a worker using zsh-async.

Do you have any resources for how to do this? I'd love to try it on WSL. :)

[dritter]

@mcfiredrill there is already #1176 for that. It is based on the next branch, so it might be quite different, if you are coming from master. At the moment it is in beta, so don't expect everything to work. Besides migrating your config to the new format, you have to do two things:

• Install the git submodules, so that zsh-async gets installed at the right location (not yet configurable).
• Flag the segment you want as async, by suffixing it with ::async.

And, if you are on next and don't want to try #1176 , you could get rid of the vcs segment and replace it with gitstatus. That segment makes way less IO work, and is deeply optimised for performance, but you have to install gitstatus first. As the name suggests, that works for git only.

[romkatv]

With the latest improvements in gitstatus performance there is no practical difference between sync and async mode but only if you are using Linux (and maybe MacOS and BSD; I haven't benchmarked them in a while). At some point it was also reasonably fast on WSL but then I fucked it up. WSL masquerades as Linux, so it triggers linux-specific code path in gitstatusd. This code path is optimized to the brim based on the timing of system calls in Linux kernel. WSL has the same system calls but with very different performance characteristics (obviously, because it doesn't have Linux kernel). So when I booted Windows yesterday and fired up WSL, I was appalled to realize gitstatusd is now as slow as plain git status. Ironic, given that I started hacking on zsh and git because I couldn't stand how slow things were on WSL. Oh well.

This means that as of right now there is no fast ZSH theme for WSL. There are only tolerably slow and kill-me-now slow. And you absolutely have to use an async git prompt to maintain semblance of sanity. The most popular theme with async git prompt is definitely Pure. It's the one that has pioneered async workers and it's truly a marvel to study. The most practical theme at this point is IMO Powerlevel10k, but I'm obviously biased.

[thromera]

With the latest improvements in gitstatus performance there is no practical difference between sync and async mode but only if you are using Linux (and maybe MacOS and BSD; I haven't benchmarked them in a while). At some point it was also reasonably fast on WSL but then I fucked it up. WSL masquerades as Linux, so it triggers linux-specific code path in gitstatusd. This code path is optimized to the brim based on the timing of system calls in Linux kernel. WSL has the same system calls but with very different performance characteristics (obviously, because it doesn't have Linux kernel). So when I booted Windows yesterday and fired up WSL, I was appalled to realize gitstatusd is now as slow as plain git status. Ironic, given that I started hacking on zsh and git because I couldn't stand how slow things were on WSL. Oh well.

This means that as of right now there is no fast ZSH theme for WSL. There are only tolerably slow and kill-me-now slow. And you absolutely have to use an async git prompt to maintain semblance of sanity. The most popular theme with async git prompt is definitely Pure. It's the one that has pioneered async workers and it's truly a marvel to study. The most practical theme at this point is IMO Powerlevel10k, but I'm obviously biased.

OMG. I wasn't aware of PowerLevel10K, thank you for your work. Just switched to it, 💥 , lightning fast in a sec.

[romkatv]

@Erowlin Are you trying to get me banned here? Jesus...

[robobenklein]

I guess I'm lucky that I never have to deal with WSL, so I have no idea if my zsh-async stuff works there at all or not.

I wonder if it's possible to get the same zsh-async API on WSL without using zpty? (Which is still the root cause, right?)

The pty issue seems to have been closed though: microsoft/WSL#323 (I don't own Windows so I can't test it...)

[romkatv]

@robobenklein zpty works fine on WSL. In fact almost all Linux software just works.

There are three sources of prompt slowness on WSL. The first is that some people insist on using ridiculously slow terminal emulators. Stuff like ConEmu or Hyper. These terminals have typing latency in the order of tens of milliseconds when your screen is full of text. Absolutely hideous. The only terminal with decent performance on Windows is Command Prompt -- the default one. It's ugly as sin and has no configuration options to speak of but it's the only thing you can use without having an urge to murder someone.

The second reason is that forking has 10x the cost it has on Linux. You know how you can make your prompt really slow if you use stuff like $(echo | grep | wc) in it? 10 times so on WSL.

The first two reasons don't affect me personally. My terminal is Command Prompt, and Powerlevel10k has no forks in the generic code or the prompt segments I use. This brings us to the third reason -- git is incredibly slow. Git makes a lot of stat() calls, which are cheap on Linux but very expensive on WSL (some people say it's IO but it isn't; this is all CPU in kernel code). They are expensive on WSL because rootfs -- the filesystem WSL uses -- is built on NTFS, and NTFS has all those advanced features no one cares about that make stat() slow. So it's not really the fault of WSL. If you install "Git for Windows", it'll be slow too. Or you can mount an NTFS partition on Linux and enjoy the slowness without owning a copy of Windows. How convenient!

It is possible to make git (or gitstatusd) faster on Windows but this requires dedicated effort and decent setup with a low-level profiler that doesn't lie too much. The set of people who can do it is rather small. If you intersect it with the people who care, it becomes empty.

[romkatv]

P.S.

You can totally do async without zpty. Powerlevel10k has async git prompt without zpty. It's faster this way.

[Syphdias]

In fact almost all Linux software just works.

If you don't want to use some network features or gpu or usb...

[thromera]

@Erowlin Are you trying to get me banned here? Jesus...

I am sure people in here are not censoring nor dictators 😄.

[bhilburn]

I am on a issue-cleaning spree. This issue hasn't had motion in quite some time, and so I am closing it out due to inactivity. If anyone would like to re-open it because they feel it is un-resolved and is something they care about, please feel free to re-raise the discussion!