Slow when typing ** [condition: shopt -s globstar] #82
Comments
|
Thank you for the report! OK, I think it is related to the setting There are several possible bottlenecks, which I'll later investigate.
To solve the problem, now I'm thinking of running pathname expansions containing |
Yup, unsetting that option makes it fast again
Yup, this stops the issue
This doesn't fix it...
So I guess it's a filename coloring problem, so a quick fix would be to just disable completion highlighting for this case? Idk how you are currently getting the files to serve to the autocomplete, but would it be possible to run it in a sub-shell and continuously update the suggestions based on the filter as soon as it gets more values? For example similar to how fzf handles its file search. If this works it could be applied to all file suggestions, and it would also fix the problem when trying to request completions for a directory with thousands of files. I don't think a timeout is necessary in this case, maybe eventually some indicator that the search job is running would be good, and the first SIGINT could be sent to it if needed instead of aborting the current command. And highlighting could be performed only for the results being displayed instead. Idk if you are doing this already... Idk how polling for these new incoming suggestions would work though... How are you currently displaying the suggestions? I could help you out on testing this but I would need a few pointers to where some of these things are, like generating filepahts, displaying suggestions and filtering them. Maybe I could at least experiment some ideas... and probably hit the same problems you already did xD. Tell me what you think since this certainly brings a lot of issues I'm not seeing. |
akinomyoga
changed the title
akinomyoga
changed the title
|
Thank you for the tests! I think, at least, one of the problems occurs inside the highlighting of filenames. In your quick test, it seems auto-completion didn't suffer from the problem, but I think it can also be blocked by slow
The suggested solution seems to be not so easy to implement soon, so for a quick fix, I think I'll introduce the above workaround (subshell & timeout) for the patterns that contain raw
That is indeed an interesting idea! I have considered that direction. It seems some technical problems should be solved to implement it in the shell. Below are some random thoughts on those problems. Edit: I think Problems 2 and 3 can be handled if we really think this idea valuable and make sufficient efforts. The real problem is Problem 1. Problem 1. Incremental Pathname ExpansionsFirst,
Problem 2. Slow BashNext, Bash is slow. In my expectation, as far as it is implemented in a Bash script, we cannot expect as responsive user experience as
Problem 3. Implementation ComplicationAlso, full support of such a continuous implementation would lead to the complication of the implementation. Bash is a single-thread language and doesn't provide first-class support of alternative mechanisms such like coroutines or fibers. Bash doesn't provide first-class support of semaphore, mutex, or other facilities for the synchronization. These language limitations make it difficult to do asynchronous programming in Bash. We may still implement these mechanisms by hand, but it will require a big rewrite of the existing codes, and also the code would become unnecessarily complicated. We may create background processes by
Misc
I have been also thinking that a timeout or some hard-coded limitation is unnecessary for background tasks, but in the discussion at #64 (comment) it was pointed out that the fact Bash occupies so much CPU usage in the background for any user input may be unpreferable for many users. In particular, the pathname expansions like
I'm sorry that I couldn't get what you mean by this? Would you mean stopping filename highlighting for the words that contain special glob characters?
Currently, in the processing of
Completions are implemented in
|
|
Indeed this brings quite a lot of problems... In the meantime do you think it's possible to do some workaround to prevent it being so slow, because even when scrolling through the command history it halts for a while when it finds a command with ** . |
This is rather related to the syntax highlighting but not to auto-complete. In the previous comment, I have shown the pointers for auto-complete but not for the syntax highlighting. If you want the pointers for the syntax highlighting as well, please ask me again. Also, I guess you'll get further questions on the code of auto-complete, so also please feel free to ask me about them. In these couple of days, I too have thought about how we can manage this problem in the syntax highlighting. Now I'm thinking of performing word highlighting as a background task (but not in a subshell). Maybe it requires a large change of the code, but I believe it is possible. |
In Now I don't notice remaining problems with these changes, but they are a relatively large set of changes (47a3301...1e771d5) and haven't been tested well, so I haven't yet pushed it to the Thank you |
|
Hi there, |
The user-input cancellation and timeouts are independent, but I actually guess the delay you see here is rather related to a timeout. Let me explain the behavior. For example, consider the case you input
The user-input cancellation instantly occurs any time when you input additional character Maybe we want to make the timeout bleopt highlight_timeout_defer=50Thank you!! |
|
I rebased and force-pushed
|
|
Hey there, Also after typing a globstar path and even after this has been correctly highlighted, asking for completion with These improvements are really good, but I'm wondering if it would be simpler to just ignore highlighting on glob patterns since they might not even be accurate and not really relevant. For example in |
Yeah, I actually thought something shorter time is enough for the quick pathname expansions, but unfortunately, the fork cost for subshells can be about a few tens of milliseconds in some systems. Setting a shorter time than the fork cost means that the subshells are immediately killed before it does the actual task. So, I want to set the default value to be a relatively larger value, i.e., a longer time than the fork cost of arbitrary systems. (Actually, in some systems with antivirus software, the fork cost can be about 200 msec, but I think setting the timeout to be that order is noticeably unresponsive, so I chose 50 msec as a moderate value which is likely to be longer than most cases.) Also, I'm wondering if the difference between 50 msec and 20 msec really matters. One may notice the difference of this 30 msec if one carefully compares the behavior, but I don't know if it really makes things different in real use.
If we don't perform sync highlighting for glob patterns, the highlighting of glob patterns will not be performed at all while I am typing but will be performed after I stop the typing, which I don't like. Anyway, now it is configurable, so you can turn off the sync highlighting by setting
Oh, OK. That may be a problem. Did you experience the worsening before the second push to If you could reproduce it, could you tell me the details?
Does the tab completion get slower than
Yeah, the fact it can be inaccurate might be a problem, but the highlighting on glob patterns is really relevant for my usage. The purpose for highlighting is not just eye candies for simple paths like As you have pointed out, highlighting for multiple file paths can be inaccurate, but it is rather obvious that the highlighting cannot show the status of multiple files, so I thought it is an easy guess that the highlighting is just for one path from the glob pattern. In fact, the highlighting is performed based on the first path generated by the pattern.
Is it on the
Maybe this is only for me, but in my case, I feel it very useful in my long use history of Edit: In case you don't want highlighting glob patterns, you can optionally turn it off by setting both timeouts zero:
Yeah, that is the idea that I initially thought. One problem is that it is not so trivial to detect the globstar Also, I don't want to complicate the rule observable to users in a way hard to guess. For example, if we only perform the highlighting on the glob patterns without a globstar |
|
Hi, I tried to show all the problems I mentioned, comparing performance against just executing a command with the globstar. Then performing completions after the globstar pattern is processed and simply typing while the pattern is being run.
Ok, yeah I get your point, I have fucked up before due to globstars not expanding to anything xD Also, on the spike branch the shell crashed on me a couple of times. I believe all of them happened when I hit tab to ask for a completion. I don't know more details but if I notice a pattern I'll let you know. Cheers. |
|
Nice!
|
|
Yeah here you go:
Yeah but I thought that since bash also expands the arguments to the |
Re: Slow even with bleopt highlight_timeout_sync=0
Thank you for your measurements! Hmm, it seems there is no bottleneck in the tested items. As I haven't yet reproduced this delay, I think I'll ask you for more detailed profiling if I cannot reproduce it for several days. Maybe I'll try other UNIX systems if I have time. Re: ls time and highlighting time
Ah, you're right. I measured the time and found that the highlighting time is three times as long as the time of the command execution. Then, I found that the reason is that, for the glob pattern Re: Slow TAB completion with globstar wordsFor this one, I wrote in the previous reply
but it wasn't actually the true cause. It turned out that in my host the slow TAB completion (after the highlighting of globstar patterns finished) doesn't reproduce. I noticed my misunderstanding after I implemented a new timeout So, ... this is another problem that I couldn't reproduce. Maybe I'm going to ask for detailed profiling for this one as well. Re: Crash on TAB completionI haven't yet reproduced this either. I looked into the code but don't have an idea now. |
Maybe don't worry about this, at least for now. It might have been an issue on my side since I noticed it doesn't reproduce every time. it could have been my HDD that is getting dubious read speeds xD
Ok good, so this time can be shortened? Like the full glob can be expanded once, pick one of the results and just syntax highlight the glob according to that result? Ideally we could stop bash on the first expansion result and this would be nearly instant, but sadly we can't as you've noted before...
I just tried the new commits and it does improve TAB completion after globstar patterns. It's still not as smooth as without the pattern... Also I noticed something else that I'll put bellow.
Well, this didn't happen to me again, since then (I think I haven't used it that extensively these last few days either), however as I was typing this answer and about to record the gif bellow it just crashed when I was about to type For the new issues I found:
Glob patterns don't work with paths starting with And it seems that the speed of typing/tab completion is linked with the amount of expanded items in the same command, since it drastically increases cpu usage. Do you have idea what's happening? It shouldn't care for the expression that has already been calculated... |
|
Thank you for the continuing discussions! Re: ls time and highlighting time
Yeah, I thought a similar thing. Would you be interested in contributing to this one? The above mentioned expansions are made in a single function Re: Slow even with bleopt highlight_timeout_sync=0
Maybe I was misunderstanding the speed level that you discuss. I somehow assumed there are severer delays in inputting texts, so I was trying to guess what causes a long delay. But now I came up with the possibility that you are rather focusing on the input experience. In that case, I had some improvement ideas (which I had kept from until the severe delay would be solved). Now I've implemented them in
Could you see if anything changes with this commit (i.e., by comparing before and after the commit)? Re: Crash on TAB completion
May I confirm that it means the shell crashed with just " By the way, I encountered hangs a few times while typing glob patterns in my Linux. I'm not sure if it is related to this crash, but I'll later fix it anyway. Tilde prevents correct pathname expansions
Thank you for the report! It was a trivial mistake. I fixed it in Speed is linked with the number of expanded items
Maybe it is related to the cache size of pathname expansions. In To check if that's the case or not, I added a commit for the experiment (640f32d) that restricts the number of words in a cache to two. Could you see if anything changes before and after this commit? Thank you! |
Yeah ok, I can give it a go 👍
Yeah, this commit with
I think I might be leading yo the wrong direction. The shell doesn't exactly crash, it just hangs there unresponsive... So it can be this problem that you mentioned. I noticed today that when it hangs it has only one subshell running ( as oposed to 2 when typing a glob pattern) and if I kill this subshell the shell works again. It did happen twice today, once when typing the first asterisk in a glob and a second when hitting tab on a normal path after I had a glob in the same command like:
👍
Awesome, this makes a huge difference! Everything seems to be working really well, including Tab completions after the glob pattern |
Re: ls time and highlighting time
OK. The target is the function This function is used inside the highlighting something like in the following way: ble/syntax:bash/simple-word/evaluate-path-spec '**/*tool*/*.sh' / after-sep:cached:stopcheck:timeout=50:timeout-carryFor the present purpose, the following command is enough to check the performance. # Pre-condition: Ensure there is no timeout carry set by "timeout-carry" option
unset _ble_syntax_bash_simple_eval_timeout_carry
time ble/syntax:bash/simple-word/evaluate-path-spec '**/*tool*/*.sh' / after-sep
# Result
declare -p path spec ret
Currently, in my host, it causes three times as longer time as the single pathname expansion. $ time ble/syntax:bash/simple-word/evaluate-path-spec '**/*tool*/*.sh' / after-sep
real 0m9.555s
user 0m4.544s
sys 0m4.853s
$ time ble/syntax:bash/simple-word/eval '**/*tool*/*.sh'
real 0m2.885s
user 0m1.179s
sys 0m1.658sSo, the goal is to shorten this time. Maybe it is not convenient to regenerate and reload # evaluate-path-spec-v2.sh
function evaluate-path-spec-v2 {
local word=$1 sep=${2:-'/:='} opts=$3
ret=() spec=() path=()
[[ $word ]] || return 0
# read options
local eval_opts=$opts notilde=
[[ :$opts: == *:notilde:* ]] && notilde=\'\'
# compose regular expressions
local rex_element; ble/syntax:bash/simple-word/.get-rex_element "$sep"
local rex='^['$sep']?'$rex_element'|^['$sep']'
[[ :$opts: == *:after-sep:* ]] &&
local rex='^'$rex_element'['$sep']?|^['$sep']'
local tail=$word s= p= ext=0
while [[ $tail =~ $rex ]]; do
local rematch=$BASH_REMATCH
s=$s$rematch
ble/syntax:bash/simple-word/eval "$notilde$s" "$eval_opts"; ext=$?
((ext==148||ext==142)) && return "$ext"
p=$ret
tail=${tail:${#rematch}}
ble/array#push spec "$s"
ble/array#push path "$p"
done
[[ $tail ]] && return 1
((ext)) && return "$ext"
return 0
}
# tests
unset _ble_syntax_bash_simple_eval_timeout_carry
time evaluate-path-spec-v2 '**/*tool*/*.sh' / after-sep
declare -p spec pathso that you can edit the file and measure the time by Now I think you can explore the implementation of this function and related codes by yourself, so I skip line-by-line explanations, but please feel free to ask about details if you have any questions.
Re: Slow even with bleopt highlight_timeout_sync=0
OK, nice! I think I'm going to adjust this commit more. Re: Crash on TAB completion
Great! This is actually what I observed in my host!
Thank you! I still don't get the exact condition to reproduce it, so the information is helpful! Re: Tilde prevents correct pathname expansions
Thank you! Re: Speed is linked with the number of expanded items
OK! I think the slow TAB completion even with the finished highlighting is also related to this one. The current commit is just for an experiment, and it breaks some functionality of |
|
I have force-pushed to Re: Slow even with bleopt highlight_timeout_sync=0
I modified this one as the timeout carry only occurs in the sync highlighting (fd3b0c4). If this additional change has worsened the response, please let me know. Re: Speed is linked with the number of expanded items
I've updated the code so that the caches for the first word and all the words are separately saved (a5f22cc). I guess this also solves or improves the problem "Re: Slow TAB completion with globstar words". |
Re: Crash/hang on TAB completionThese few days, I have worked on this problem. It seems to take more time to settle it, so I summarize the current status here. This seems like a problem of the implementation of The following script is a reproducer for the Bash problem. If you are interested, you can just run it as a normal Bash script to reproduce the problem. It should count up to 2000 without stopping. But, in Bash 4.3+, it stops randomly; #!/bin/bash
rm -f a.pipe
mkfifo a.pipe
exec 9<> a.pipe
rm -f a.pipe
for c in {0..2000}; do
(eval "echo {0..$c}" & read -u 9 -t 0.001) >/dev/null
printf $'\r\e[Kok %d' "$c"
done
echoActually, it seems that I have tested in Linux, Cygwin, FreeBSD, Minix with different versions of Bash. In Linux, Cygwin, and FreeBSD, the behavior is consistently reproduced with all the Bash versions whose version is 4.3 or higher including Here the results for different versions in different systems
I've performed Critical Bash change in 201310e784337238e6081ca5f9bdc8c21492f7b89388 is the first bad commit
commit 10e784337238e6081ca5f9bdc8c21492f7b89388
Author: Chet Ramey <chet@caleb.ins.cwru.edu>
Date: Mon Mar 4 08:10:00 2013 -0500
commit bash-20130208 snapshot
diff --git a/CWRU/CWRU.chlog b/CWRU/CWRU.chlog
index 04c629e7..59759f24 100644
--- a/CWRU/CWRU.chlog
+++ b/CWRU/CWRU.chlog
@@ -4502,3 +4502,23 @@ variables.c
but create them as (invisible) exported variables so they pass
through the environment. Print an error message so user knows
what's wrong. Fixes bug reported by Tomas Trnka <ttrnka@mail.muni.cz>
+
+ 2/9
+ ---
+
+builtins/read.def
+ - sigalrm_seen, alrmbuf: now global so the rest of the shell (trap.c)
+ can use them
+ - sigalrm: just sets flag, no longer longjmps to alrmbuf; problem was
+ longjmp without manipulating signal mask, leaving SIGALRM blocked
+
+quit.h
+ - move CHECK_ALRM macro here from builtins/read.def so trap.c:
+ check_signals() can call it
+
+trap.c
+ - check_signals: add call to CHECK_ALRM before QUIT
+ - check_signals_and_traps: call check_signals() instead of including
+ CHECK_ALRM and QUIT inline. Integrating check for read builtin's
+ SIGALRM (where zread call to check_signals_and_traps can see it)
+ fixes problem reported by Mike Frysinger <vapier@gentoo.org>
diff --git a/builtins/read.def b/builtins/read.def
index ea2667bb..7a1dff6f 100644
--- a/builtins/read.def
+++ b/builtins/read.def
@@ -136,27 +136,21 @@ static void ttyrestore __P((struct ttsave *));
static sighandler sigalrm __P((int));
static void reset_alarm __P((void));
-static procenv_t alrmbuf;
-static int sigalrm_seen, reading;
+procenv_t alrmbuf;
+int sigalrm_seen;
+static int reading;
static SigHandler *old_alrm;
static unsigned char delim;
/* In most cases, SIGALRM just sets a flag that we check periodically. This
avoids problems with the semi-tricky stuff we do with the xfree of
input_string at the top of the unwind-protect list (see below). */
-#define CHECK_ALRM \
- do { \
- if (sigalrm_seen) \
- longjmp (alrmbuf, 1); \
- } while (0)
static sighandler
sigalrm (s)
int s;
{
sigalrm_seen = 1;
- if (reading) /* do the longjmp if we get SIGALRM while in read() */
- longjmp (alrmbuf, 1);
}
static void
diff --git a/quit.h b/quit.h
index 8df01e1f..7d447ab2 100644
--- a/quit.h
+++ b/quit.h
@@ -36,6 +36,11 @@ extern volatile int terminating_signal;
if (interrupt_state) throw_to_top_level (); \
} while (0)
+#define CHECK_ALRM \
+ do { \
+ if (sigalrm_seen) \
+ longjmp (alrmbuf, 1); \
+ } while (0)
#define SETINTERRUPT interrupt_state = 1
#define CLRINTERRUPT interrupt_state = 0
diff --git a/trap.c b/trap.c
index d60a7fb4..b030747b 100644
--- a/trap.c
+++ b/trap.c
@@ -88,6 +88,8 @@ static void reset_or_restore_signal_handlers __P((sh_resetsig_func_t *));
extern int last_command_exit_value;
extern int line_number;
+extern int sigalrm_seen;
+extern procenv_t alrmbuf;
extern char *this_command_name;
extern sh_builtin_func_t *this_shell_builtin;
extern procenv_t wait_intr_buf;
@@ -456,18 +459,20 @@ any_signals_trapped ()
return -1;
}
/* Convenience functions the rest of the shell can use */
void
check_signals_and_traps ()
{
+ check_signals ();
QUIT;
run_pending_traps ();
}
void
check_signals ()
{
+ CHECK_ALRM; /* set by the read builtin */
QUIT;
}
#if defined (JOB_CONTROL) && defined (SIGCHLD)I got a stacktrace when When So, before the change, Bash cancels After the change in 2013, a related change has been made in 2018 as follows. This change was introduced after the discussion bug-bash/2018-01/msg00114. It seems that the problem actually has been already reported, but only "a partial fix" has been made at that time. Chet added an additional check on Bash change in 2018 1/31
----
lib/sh/zread.c
- zread,zreadintr: call check_signals() before calling read() to
minimize the race window between signal delivery, signal handling,
and a blocking read(2). Partial fix for FIFO read issue reported by
Oyvind Hvidsten <oyvind.hvidsten@dhampir.no>
diff --git a/lib/sh/zread.c b/lib/sh/zread.c
index 14b66381..8b7ecedf 100644
--- a/lib/sh/zread.c
+++ b/lib/sh/zread.c
@@ -53,6 +53,7 @@ zread (fd, buf, len)
{
ssize_t r;
+ check_signals ();^I/* check for signals before a blocking read */
while ((r = read (fd, buf, len)) < 0 && errno == EINTR)
/* XXX - bash-5.0 */
/* We check executing_builtin and run traps here for backwards compatibility */I've tested for this fix, but it seems this fix actually didn't have much effect on the above reproducer. I also investigated the timing when SIGALRM arrives, but it seems In this case, I don't know what is the proper way to fix Bash. Maybe we should use The problem is present in Bash 4.3, 4.4, 5.0, and 5.1 in any way. I think from the experience, even if this is fixed in |
Re: Crash/hang on TAB completionI pushed a workaround for this in I also reported the issue to the Can you test the latest version of How's it going with the improvement on Thank you! |
|
Hi, I'm sorry for not replying sooner, I haven't had the time to check this out and I'm sorry if I'm delaying things for you.
Thank you for your detailed explanation, as soon as I find some time I'll get into it. But if you don't want to wait for me it's ok, you can probably fix it quite easily
Awesome that you found all this and that you might be able to fix it, I'm gonna update to the latest version, use for a few days and let you know |
|
Thank you for your reply! Re: ls time and highlighting timeIt's not in a rush so please take your time. I usually unset the Re: Crash/hang on TAB completionThank you! I look forward to seeing your feedback! Hope everything works fine. |
|
I have pushed Edit: I have deleted the |
|
Hi there, sorry for not saying anything anymore. I haven't had time/energy to check this and do the changes we mentioned. |
|
OK! Thank you for letting me know that. I'll close this issue after I implement the suggested optimization for |
|
Yeah, no problem 👍 |
Hi there,
I'm having some problems when typing **, specially on large directories. I assume this could be something to do with fzf feature of **, however I don't have this feature enabled (I think) . In this example I'm typing and it takes too long to write what I typed:
In cases of very large directories (with lots of entries) it will hand the shell.
Is it possible to disable this behavior by default ?
List of Discussed Issues
Edit by @akinomyoga
highlight_timeout_synchighlight_timeout_asyncbleopt complete_timeout_autobleopt highlight_timeout_sync=0by async highlightingbleopt complete_timeout_compvarconditional-syncis suspicious because there is always a subshell when it occurred.The text was updated successfully, but these errors were encountered: