counsel-git-grep frequently hangs

[midsbie]

When running counsel-git-grep on a fairly large mono-repo, I get frequent hangs that block emacs, making it completely unusable, and which can only be resolved by killing the running git process via pkill git. Only once I kill git can I issue C-g to finally stop counsel and get input back in emacs. In addition to the hangs, I have seen memory consumption increase greatly since I started using ivy + counsel + swiper and suspect the lock ups I've been experiencing are the cause. For instance, my current emacs session has a total of 110 buffers open and a RSS memory consumption of over 1GiB:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
miguelg  20500  0.6  3.7 1623880 1232824 ?     Sl   Jun26  34:08 emacs

Perhaps because of the high memory consumption, I've noticed that sometimes emacs itself freezes momentarily for anywhere .5s to 2s, which I don't really know how to explain. Running (garbage-collect) does nothing to the overall memory consumption and returns the following:

((conses 16 5984385 271635) (symbols 48 56871 79) (miscs 40 26186 5365) (strings 32 212150 85729) (string-bytes 1 10114138) (vectors 16 127401) (vector-slots 8 7050126 164634) (floats 8 428 1684) (intervals 56 60473 11778) (buffers 992 166))

Lastly, I have also observed the exact same behaviour when running counsel-ag, perhaps even more often than with counsel-git-grep.

I'd be happy to help further nail down the causes of the hangs and suspected resultant higher memory consumption.

[abo-abo]

I think the hangs might be related to a large amount of output from either git or ag. Let's try to reproduce it.
Open a shell and enter a command equivalent to what counsel-git-grep does and see how Emacs reacts. In my experience, if you're getting megabytes of output, Emacs is likely to hang.

[midsbie]

@abo-abo: I attempted your suggestion over the course of the last week and can confirm that the sort of queries I run do not produce megabytes of output. There's definitely something else at play here and I don't think git is to blame. counsel-ag is also afflicted by the same hanging bug. My suspicion right now is on the IPC communication channel, which under certain conditions that I haven't been able to understand and frustratingly don't even know how to debug, seems to somehow hang causing Emacs to block.

Any other steps you recommend I take to try and debug this?

[abo-abo]

What's your system and Emacs version? Is there a public repository to reproduce it with, and a set of steps that are likely to demonstrate the bug?

[midsbie]

@abo-abo System is Ubuntu 18.0419.04 and running Emacs 26.1. Unfortunately the repositories I've been experiencing this issue on are private and not even owned by me, but I'm pretty sure that any sufficiently large enough repository would produce the same result. Unfortunately I've not been able to eliminate the possibility that this is the result of some weird interaction with some other package, however unlikely this may be -- could you please recommend a way of testing for this?

I'm more and more inclined to think that this issue does not manifest itself on a fresh Emacs session and it is only after some time has gone by and several uses of Ivy's awesome git-grep search mechanism (how many I cannot quantify as of right now) that it starts hanging. As an additional data point for debugging, it seems Emacs/Ivy causes the child process -- in my case, git, but the exact same thing happens with ag -- to hang and while I can C-g to abort the search when it first starts to hang, eventually after several hangs have occurred it gets to a point where C-g stops working at all and the only way of getting back input in Emacs is to first kill git. Only once I kill git (or ag, for that matter) am I able to switch over to Emacs and flood it with C-g, and only then is input in Emacs is restored.

When C-g finally restores input the following message is consistently produced in *Messages*:

Error in post-command-hook (ivy--queue-exhibit): (quit)

[abo-abo]

could you please recommend a way of testing for this?

This public repo is sufficiently big:

git clone https://github.com/torvalds/linux

To isolate the interference from other packages, use:

git clone https://github.com/abo-abo/swiper/
cd swiper
make plain

[abo-abo]

Any update? If there's no way to reproduce the issue, it's better to close it.

[midsbie]

Unfortunately I keep stepping into the issue every now and then but not in a way that would be reproduceable. Thanks for your assistance with this!

Closing.