TUI cursor motion, SGR, and scrolling optimizations

[jdebp]

The TUI sends a lot of extraneous stuff to the terminal. It repeats SGR sequences unnecessarily. It uses inefficient cursor motions. It does not shortcut repainting the cleared area in scrolls for terminals announced as BCE. It does not take advantage of terminal scrolling margin capabilities.

This builds upon the work in #6792 (née #6685) to address these, and reduce the amount of data that is sent to update and to repaint the screen. Yes, this is still important to people using SSH and the like.

Some of these optimizations I remember from the days of stevie.

[jdebp]

Running the test suite on FreeBSD here does not produce the failure that is shown by the automatic tests. It was showing some other errors, caused by error message spelling differences that the test harnesses were not expecting. I have fixed them in #6791.

[justinmk]

The QB failure may be a timing issue or may just require a test adjustment, I'll take a closer look.

22:23:14,464 INFO  - # test/functional/terminal/tui_spec.lua @ 282: tui focus event handling can handle focus events in terminal mode
22:23:14,464 INFO  - not ok 2156 - tui focus event handling can handle focus events in terminal mode
22:23:14,464 INFO  - # test/functional/terminal/tui_spec.lua @ 282
22:23:14,464 INFO  - # Failure message: ./test/functional/helpers.lua:252:
22:23:14,464 INFO  - # retry() attempts: 2
22:23:14,464 INFO  - # ./test/functional/ui/screen.lua:301: Row 1 did not match.
22:23:14,464 INFO  - # Expected:
22:23:14,464 INFO  - # |*ready $ |
22:23:14,464 INFO  - # |[Process exited 0]{1: } |
22:23:14,464 INFO  - # | |
22:23:14,464 INFO  - # | |
22:23:14,464 INFO  - # | |
22:23:14,464 INFO  - # |gained |
22:23:14,464 INFO  - # |{3:-- TERMINAL --} |
22:23:14,464 INFO  - # Actual:
22:23:14,464 INFO  - # |*nal |
22:23:14,464 INFO  - # |{1: } |
22:23:14,464 INFO  - # |{4:~ }|
22:23:14,464 INFO  - # |{4:~ }|
22:23:14,464 INFO  - # |{4:~ }|
22:23:14,464 INFO  - # |gained |
22:23:14,464 INFO  - # |{3:-- TERMINAL --} |
22:23:14,464 INFO  - #
22:23:14,464 INFO  - # To print the expect() call that would assert the current screen state, use
22:23:14,465 INFO  - # screen:snaphot_util(). In case of non-deterministic failures, use
22:23:14,465 INFO  - # screen:redraw_debug() to show all intermediate screen states.
22:23:14,465 INFO  - # stack traceback:
22:23:14,465 INFO  - #     ./test/functional/helpers.lua:252: in function 'retry'
22:23:14,465 INFO  - #     test/functional/terminal/tui_spec.lua:286: in function <test/functional/terminal/tui_spec.lua:282>

Regarding the changes in general, I'll take your word for it, you clearly have deep experience with terminal programming. Maybe @leonerd could take a look :)

[ZyX-I]

By the way, I remember Vim having a hack (workaround for terminal emulators bugs) that sends cursor goto even if it should not be necessary after some characters with “ambiguous” width (specifically those for which terminal emulator may have other idea regarding width then Vim has (newest emoji?)). Does Neovim has something like this?

[jdebp]

That's ambiwidth. It's dealt with at a level above the TUI, in the UI and screen layer. I suspect that there was a pre-existing problem here; because the UI layer hides this from lower layers.