-
-
Notifications
You must be signed in to change notification settings - Fork 76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Long processing times when pasting large amounts of text #45
Comments
The example paste size was 500~ lines, but anecdotally I've noticed if it's around 2000 it becomes very difficult to tell if the terminal is processing the input at all. There's a quick "Pasting" message and then nothing quite a long time. I initially thought the session had crashed but it does start do decode/process eventually. |
Thank you for the report!
I recognize this performance issue. If you really want to edit the pasted contents, unfortunately it's unavoidable since all the command line contents are processed by shell scripts which is slow. I have been thinking of this issue for long time, but not yet found a perfect solution. The following are some ideas which might partially solve the issue. Abort ProcessingIf you just mistakenly pasted the large amount of texts and want to cancel the processing, you can press C-\ to abort the process and discard the inputs. You need to press C-\ as soon as possible when you noticed the mistake. (The key which can be used to abort the processing can be configured by Set upper limit (not implemented)For this issue, once I thought about setting upper limit of the number of lines, or the number of characters in a command line. But I don't know the appropriate value for the upper limit as it depends on the computational power of each host. This is a matter of degree. For example, even if you use raw Bash (without ble.sh), I didn't try but it should become slow if you paste 1 M lines. Also I didn't think there is a case that users wants to edit so many lines of command. In fact the line editing in original Bash doesn't work properly with lines more than the terminal height. So I haven't set any upper limit so far. Store the pasted contents into a shell variable (not implemented)Another solution might be to store the pasted texts into a shell variable (e.g., Launch real text editor (not implemented)Or if you want to edit so many lines before executing commands, maybe we can launch a real text editor (written in real programming language but not shell scripts) just like the binding C-x C-e when the length of the pasted texts exceeds some limit value. I would like to ask for your comments. What is the use case for pasting such a large amount of texts? Is it just a mistake? Or do you want to pass long texts to commands without editing the pasted contents? Or do you want to edit the pasted contents? Do you have any other ideas to deal with this issue? |
Thanks for the response!
I think I'd like to change the command to be
This sounds reasonable. Although it's hard to set arbitrary limits because each host is different like you said, maybe if only a few lines are pasted in it's fine and the normal multi-line editor kicks in. Otherwise an editor boots up. I'm not positive if I understand the
Usually something has been printed to standard out that I don't really want to run again and I need to reuse the output. For example, I might have performed a |
Thank you for your feedback! I have implemented the above ideas. Sorry, it took some time to design the detailed behavior, and to implement and test them. But still the performance issue is not completely solved. Value of bleopt decode_abort_char
I think you can see the two leftmost columns in the following table in Wikipedia: For example, if you want to know the code for C-c, you can look for the row with bleopt decode_abort_char=3 By the way I found that this Upper limit & truncate / discard / editor
I implemented the setting of upper limit and the settings to control the behavior on the case the command line length exceeds the limit. The following options are newly implemented 2f9a000: Please refer to each link for details. The upper limit is turned off by default for now. For example, if you want to switch to bleopt editor=vim
bleopt line_limit_type=editor line_limit_length=2000 However the performance issue has not yet been solved even if a real text editor is used. |
Thanks! Tested and this works great.
Tested this out and it works as well. The performance issue still does seem to be pretty prohibitive, so I'm interested to hear what you come up with. Feel free to close this issue or leave it open. |
Thank you for testing!
I added another optimization 0d9d867.
For this one, I also added progress bars for the input reading phase and character insertion phase. Now there are four progress bars corresponding to the four phases: violet, blue, green and pink phases.
Maybe the bottleneck phase depends on the environment. When I pasted 2000 lines (obtained by repeating JSON from your link four times) in my environment, each phase equally took about ten seconds with the latest Actually, it should depend on whether your terminal supports Bracketed Paste Mode (Mode But I noticed that if one intentionally pastes a large amount of text, actually one could launch the text editor by C-x C-e before pasting the text. So now I'm kind of thinking this issue is not so serious. |
Cool! I did notice faster paste performance after pulling in the latest update. I tried pasting the 500~ lines in the link I sent you in iTerm2 and kitty. Apparently kitty is more performant than iTerm2, but I didn't notice much of a difference in this case. I confirmed that my iTerm paste settings do have bracketed paste selected. Here are the approximate stages of the paste job: Stages 1 - 3 were 5~ seconds each I tested out 2k lines in iTerm Stages 1 - 3 were 15~ seconds each
Yes, that's probably true. Here's some hardware specs for my machine in case it's helpful.
Huh, yeah that's a good point. That's probably the workflow I'll use if I'm pasting in a large amount of text and can remember to do so. There may be some small hacks to speed up detecting whether the Small aside question. Some paste operations out there are very quick - say opening a text editor with |
Thank you very much for providing me the results of such detailed measurements!
It's interesting to see that stage 4 suddenly becomes slower when one goes to 2k lines from 500 lines. Because you are seeing Stage 4, the Bracketed Paste Mode is turned on. In this case, Stage 4 performs the conversion of Unicode code points to strings which are all performed in Bash process. So the terminal is irrelevant for Stage 4.
Hm, it seems like the cache effect (though my guess for performance is usually incorrect...). Usually, the sudden change of the performance is a signal of a cache overflow. Also, the L2 cache size is four times different in your environment and my environment (see below). The size of 2k-line test data was about 53kB, which results in 53k elements of an array in
Yes, it should have been more efficient if it was possible. Actually, this is the difficult point. The problem is that the input stream from the terminal is a mixture of pasted texts and terminal sequences, so someone has to decode the input stream and detect the end of paste. Generally, the program which detected the start of paste takes care of the detection of the end of paste. Unless there is a text editor which can start in the middle of the paste and only handles the end of the paste,
For the same reason with the previous paragraph, once But, even if It is difficult to reach a performance similar to real text editors, but nevertheless I think it is still possible to improve the performance several times keeping the current structure of |
I added several optimizations 3f33dab. I think the performance has been significantly improved by this commit (yet it decodes the input and constructs texts inside ble.sh). |
That's great stuff! I'd say since opening this thread the time it takes to perform one of these large copy paste jobs has been reduced by about 1/2. |
Thank you for all your help! Actually I was thinking of further optimizing the processing by specially bypassing the decode/encode phases for bracketed pastes, but it needs refactoring. Maybe I will implement it someday but not now. Anyway, thank you! |
For future reference if others run into the same issue - when pasting large amounts of text that I don't need to edit before pasting, I've just been using |
Thank you for the information! Yeah, that is one way to circumvent the problem. |
On a related topic: I don't often need to edit text before pasting, but if I do I'm actually a bit more comfortable in a GUI code editor when working with larger amounts of text. Do you think it be difficult to extend the If this sounds feasible/desirable on your end I can create a separate issue/feature request. |
Thank you for the suggestion. I think you can basically assign |
I'm on macOS and your suggestion seems to work perfectly :)
When I read this I overlooked the "e.g." bit and thought that only these three editors were supported. I also tried |
Oh, thank you! I think I should spell out "for example" instead of "e.g.". |
I have updated blerc |
ble version: 0.4.0-devel2+420c933
Bash version: 5.0.11(1)-release,x86_64-apple-darwin17.7.0
Pasting to the terminal when there's a lot in the clipboard is rather slow when
bleh.sh
is loaded.The multi-line editor works great when writing writing multi-line commands or pasting in small amounts of text. However, when you paste in large amounts of text it takes quite a while to process. When I use a terminal without
ble.sh
copying and pasting in large amounts of text is relatively fast (1~ second). To reproduce, try copying the random JSON that was generated here into a terminal. Maybe this issue is unavoidable given the syntax highlighting tools, but I wanted to check first.P.S I can record a video if this doesn't reproduce well.
The text was updated successfully, but these errors were encountered: