Check why running nested consult-ripgreps do not work

[minad]

When starting multiple long running consult-ripgreps in recursive minibuffers it does not work for some reason with Vertico. Display update issue? Some state is not isolated well enough?

[jakanakaevangeli]

I tried and couldn't reproduce (with Vertico and Selectrum). Has this been fixed?

[minad]

No idea, theoretically this should work. But when I tried it quickly a while ago it didn't work. I have to give it another check.

[minad]

I am still seeing the issue. When starting nested long running ripgreps the Minibuf-1 process happily sends its results but the Minibuf-2 process does not send anything until the Minibuf-1 process finished.

[jakanakaevangeli]

Oh sorry, I missed the "long running" part. I can reproduce it now. I looked a bit into this: 1) I started a long running consult-ripgrep 2) Before ripgrep process finished, I started another consult-ripgrep 3) After both processes finished, I quit the inner minibuffer The outer minibuffer is indeed not updated. However, running consult--completion-refresh-hook in this minibuffer will successfully update it without problems. The issue doesn't seem to lie in state isolation, the lexical `candidates' variable in consult--async-sink is updated correctly for both ripgrep sessions. The problem is simply that outer minibuffers aren't refreshed. Note: I tested this with a slowed down fake ripgrep program that outputted a single line per second. When testing with normal ripgrep, observations were the same except for the fact that the output from inner ripgrep wasn't accepted by Emacs until the outer ripgrep exited. I'm guessing that its output isn't accepted because Emacs is too busy accepting outer ripgrep's output. Emacs CPU usage is at 100% during that time after all.

[minad]

The problem is simply that outer minibuffers
aren't refreshed.

Exactly. I think the throttle timer works for the outer session but no ripgrep results arrive in the :filter function.

the fact that the output from inner ripgrep wasn't accepted by Emacs until the outer ripgrep exited.

Okay, you found out the same as I did.

I'm guessing that its output isn't accepted because Emacs is too busy accepting outer ripgrep's output.

I was guessing that too. But with your slowed down program this should not be the case. What kind of inefficient event mechanism is Emacs using for this - polling? There are plenty of efficient mechanisms select, kqueue on bsds, epoll on linux.

However it should be quite common to have multiple async processes running. I guess the problem must lie with the recursive editing session since another event loop is started when another minibuffer is entered recursively. The process handling must happen in this loop and the problem lies there. For a single running ripgrep the load isn't 100%, is it?

[jakanakaevangeli]

I was guessing that too. But with your slowed down program this should not be the case.

Yes, with my slowed down program, Emacs handled output of both processes just fine. The only minor problem I observed was the failure to refresh the outer minibuffer's UI.

I guess the problem must lie with the recursive editing session since another event loop is started when another minibuffer is entered recursively.

I doubt that this is the case. read-event will accept any process's output regardless of which recursive-edit the process was created in and which recursive-edit is current.

For a single running ripgrep the load isn't 100%, is it?

On my machine, Emacs's load is close to 100% and ripgrep's load is close to 0%, which is expected as the process filter does some non-trivial processing, which is slower than ripgrep's very fast output. That is probably why Emacs only accepts the first process's output: After running its process filter it checks whether the first process's output is available and runs its process filter again. As its output is never unavailable due to ripgrep's speed, Emacs never gets the chance to check availability of the second process's output.

[minad]

After running its process filter it checks whether the first process's
output is available and runs its process filter again. As its output is
never unavailable due to ripgrep's speed, Emacs never gets the chance to
check availability of the second process's output.

Sounds like a bug to me. There should be some round robin scheduling here. But this is plausible. Maybe the problem is that I am using my own filter function and the problem would go away if the output is written to a buffer which is then processed? Maybe the filter can also be optimized. consult--async-process is one of the messier parts of Consult, which should be cleaned up (PR welcome ;)

[minad]

I reported the bug: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=48118