Support jump scrolling

[digulla]

When a software outputs a lot of text, the output in mintty can slow the software down.

In my case, the numbers are:

Output directly to the terminal: ~20 minutes
Redirect to file: ~10 minutes
Pip through tee: ~10 minutes

i.e. cmd > log and cmd | tee log are twice as fast as cmd alone.

The log file is 30 MB. Printing it with cat takes ~40 seconds. In the Ubuntu terminal, the same command takes roughly 2 seconds.

Cygwin's xterm(1) needs ~13 seconds. Cygwin's urxvt(1) needs ~5 seconds.

The scrollback buffer doesn't seem to have an effect. Since writing to pipe or file will introduce a 4 KB buffer, my assumption is that "render line" code in mintty is somehow slowing things down when the output is line buffered.

xterm also had this problem. It was solved by "jump scrolling" which means that the terminal reads as many lines as possible and renders them all at once: If you get 8 lines out output, scroll once for 8 lines, then print 8 lines.

This becomes especially useful when the software prints more lines than the console is tall.

See the option JumpScroll in the xterm(1) manual and source code.

[mintty]

For me, cat of a 30MB file to xterm with jump-scrolling takes 45s (and looks nasty because output virtually stalls repeatedly), the same in mintty takes only 38s, so I think mintty is pretty fast already.
Mintty has a kind of on-demand screen refresh, that may be effectively like (or even better than) jump-scroll.

[digulla]

I've added the numbers for xterm and urxvt on the same computer. xterm needs 13 seconds and urxvt only 5.

I'm wondering why your cat is so slow. I have a lot of long and short lines in my output. xterm output never stalls for me, it just looks ripped apart.

[mintty]

If you think your test file makes a difference, you should upload it somewhere (maybe not here...).

[digulla]

If you have Apache Maven installed, you can clone the Maven repo at https://github.com/apache/maven/tree/maven-3.6.0 and build it with

mvn -DdistributionTargetDir="C:/dev/apache-maven-3.6.0" clean package -X | tee ../mvn.log

Note: The folder C:\dev must exist and you have to use forward slashes.

That gives me a 4.6M file that takes ~5 seconds to cat in mintty and ~0.7s in urxvt.

If you can't then I can try to clean that file and send it to you.

[digulla]

I did some more tests. In the output are 26 lines with 3500 and 7800 characters. Removing those saves roughly one second in mintty.

Maybe the rest happens in escape sequence processing.

[mintty]

If you compress the test file it may be small enough to upload it here.

[digulla]

I stripped the escape sequences and tested with a pure text file. That doesn't change much. Right now, it looks like a mix of huge file and ~150 long lines between 1000 and 7500 characters.

[digulla]

Times for time cat mvn-cleaned.log:

mintty: 5.1s
urxvt: 0.8s

mvn-cleaned.log.gz

Versions:
mintty 2.9.5
urxvt 9.22-3
xorg-server 1.20.2-1
cygwin 2.11
Windows 10 Professional Ver 10.0 Build 17134
Font: Source Code Pro from https://github.com/adobe-fonts/source-code-pro/releases/tag/2.030R-ro%2F1.050R-it

[mintty]

My times (Windows 7):
mintty: 12s
xterm -j: 11s (with a hang after the initial page until output is finished)
urxvt: 2s

[digulla]

I get similar numbers but xterm outputs without any noticeable hang.
What's the window size?

I'm on an Intel GPU.

[digulla]

I can slow down xterm to 21s by clicking maximize (319x86 characters).
urxvt changes from 0.8s to 1s in this case (211x59).
mintty (271x81): 8.3s.

Running minnty with 80x10, Lucida Console, 9pt takes a mere 2.1s.
urxvt 80x10 takes 0.66s

[avih]

Other than for comparison sake, do you have a real-world use case where dumping many MB of text to the terminal is useful in any way?

[avih]

Also, AFAIK mintty already does support a sort of "jump scroll". It only updates the screen at 16ms intervals (theoretically, in practice it could be more, depending on output complexity). So it doesn't actually render all the output, and during a big dump it's highly likely that most of the text never appears on screen - try to record the screen at 60fps and see for yourself.

[mintty]

It is in fact possible to double the output speed by resetting the output timer on new output, see the patch below. However, output looks somewhat raggy now. Maybe with some more specific timer tweaking, a good trade-off could be found...

wintext.c:

@@ -1226,7 +1226,7 @@ win_update(bool update_sel_tip)
 void
 win_schedule_update(void)
 {
-  if (update_state == UPDATE_IDLE)
+  if (update_state == UPDATE_IDLE || update_state == UPDATE_PENDING)
     win_set_timer(do_update, 16);
   update_state = UPDATE_PENDING;
 }

[digulla]

What does mintty do when the timer runs out? Refresh the screen? In that case, there is a problem with the code which fetches the stdout. Or maybe rendering will block fetching/preprocessing more output.

As I said in my original post, when I run a large build job, printing the text in mintty takes 50% of the time (20 minutes instead of 10). That's a bit too much of my lifetime to ignore.

The build will either print a few lines or dump a whole lot in quick succession. My gut feeling is that mintty gets congested when a lot of output happens.

[mintty]

This is a typical use case for tee as you already mentioned yourself, while the task of a terminal is not really to skip output... So the whole issue is of lower priority, I'd say.
But yes, we can try to find a solution. It's about recognizing the flood and adapting the refresh rate to it.
This is a spot in mintty I've never touched. Suggestions welcome.

[digulla]

urxvt solves this with configuration: You can tell it to skip to the end of a flood (omitting lines) or print it screen by screen (i.e. no scrolling, it just wipes the whole screen). The former is called "skipScroll" and is enabled by default:

skipScroll: boolean
       True: (the default) specify that skip scrolling should be used. When receiving lots of lines, urxvt will only scroll once in a while
       (around 60 times per second), resulting in far fewer updates. This can result in urxvt not ever displaying some of the lines it
       receives; option -ss.

With this option disabled, cat takes 3.5 seconds.

[avih]

As I said earlier, mintty already behaves like skipScroll in urxvt. Feel free to record the screen and see for yourself.

[mintty]

There is an update delay which is set at two places in wintext.c as a parameter of function win_set_timer. Output speed of ~10MB depends on this delay about as follows:
16ms: 13s
32ms: 8s
64ms: 7s
128ms: 6s

Of course, also input/echo/output interactive response delay becomes more noticeable when increasing this delay, for me getting unpleasant at 128ms. So I could live with an increase to 32ms, but others maybe not.
A good thing (as suggested above) would be to adapt it dynamically to the current output volume.

[digulla]

I still don't understand how the update frequency affects the output speed. In my mind, mintty reads as much stdout as possible and then renders just those lines which would be visible on screen.

So there should be two threads: One which processes the stdout and updates the scroll buffer and another thread which renders the current visible portion of the scroll buffer at certain intervals.

[avih]

The main problem is that the render is relatively slow. Until this this becomes faster, it's compromises all the way down.

Personally I think 60fps should be the goal, but mintty already cannot meet this goal on complex output.

OTOH, other than making comparisons between terminals in this specific "benchmark", IMHO it's not a practical enough use case to make the code more complex just to handle it with another/configurable compromise.

[digulla]

I'm guessing that code isn't easy to change. My preferred solution would be threads (as mentioned above).

The read thread reads the output and preprocesses it into the scroll buffer (i.e. transforms the escape codes). When it's done, it will send a ping to the update thread.

The update thread will lock the buffer, determine which part is visible on the screen, create a copy of that part (just a few KB) and unlock the buffer. It will then render the copy and go to sleep again unless there are pings pending.

If there are pings pending, it will flush them all (just one update, even when the read thread has sent hundreds of update requests).

[mintty]

This is about what mintty actually does already, just that the update part is not a separate Posix thread but a Windows callback, set up by a timer. I doubt a thread redesign would make a significant change. It's the timer handling to apply adjustment to.

[mintty]

A worked-out tweak is now available, based on skipping refresh intervals (rather than extending them).
Please test.

[mintty]

Released 2.9.6.