feat(ui): ext_cmdline/messages for the TUI

[luukvbaal]

Problem: We have an unmaintained Vimscript parser and cmdline
highlighting mechanism, with which it is hard to leverage the
treesitter highlighter. Long messages result in a hit-enter-prompt.

Solution: Implement a vim.ui_attach() UI, that replaces the message
grid (orphaning some 3000+ LOC core C code). Introduce an experimental
vim._extui module, because removing the message grid at the same time is
too risky. The new UI leverages the bundled treesitter highlighter and
parser for Vimscript, as well as the matchparen plugin, to highlight the
cmdline. Messages are truncated in the cmdline area, or placed in a
floating message box in the bottom right corner. Special ("list_cmd")
messages and the message history are shown in a, "more prompt" (now a
fully interactive regular window). Various default UI elements ('showcmd',
'ruler') are also placed in the cmdline area, as virtual text.

require('vim._extui').enable({}) enables the experimental UI.
{ msg.pos = 'box' } or :set cmdheight=0 enables the message
box variant.

Followup:

• Come to a consensus for how best to represent messages (by default).
• Start removing message grid when this is deemed a successful replacement.
  When that is finished, make this new UI the default and update a lot of tests.

---

Given the following highlighted cmdline:

Pressing enter with cfg.msg.pos = 'cmd' yields a message in the cmdline with duplicate and overflow indicators. User can press g< to open the more-prompt, now in a (fully interactive) regular window.

cfg.msg.pos = 'box' yields:

[clason]

Come to a consensus for how best to represent messages, alternative:
Only move messages that do not fit the 'cmdheight' area to a dedicated
floating message box.

related: #22478

TL;DR: we need to distinguish (at least) three kinds of messages

1. single line "status messages" (-> route to vim.notify; print on cmdline as default)
2. error/deprecation messages/stacktraces etc. that should not be ignored (-> ???, floating window sounds good but probably needs to go through a vim.notify hook for plugins)
3. long-form output like :map (-> full buffer for search, yank, etc.)

(Of course, the line between 2 and 3 is blurry -- for example, :Inspect output would fit better in a floating window, so expected length of output is pobably a better distinguisher, and a separate "priority" option should be applied to must-read error messages. Prompt messages could be handled by a similar flag.)

[luukvbaal]

My starting point (which I mentioned in #27811 (comment), and is now present in the PR) is that all messages go in a (disappearing) floating window. Only when they exceed some length are they moved to a persistent floating window (mimicking a more prompt).

If @neovim/core (or people who try out this feature) feel strongly about still placing messages that fit the 'cmdheight' area where we are used for them to be, we can change that. The logic for determining based on text height is present in this PR. My hope was that people can get used to even small messages appearing in the bottom right corner.

[clason]

That's fine; I just wanted to link the context. The main points are that a) we need "message metadata" to allow different handling of messages (length is a good stand-in for an MVP, but maybe not enough in the long run) and b) we need to expose that to plugins like noice that want to shiny up the message UX. Again, not a blocker, just something to keep in mind.

[przepompownia]

vim.schedule(function()
  vim.notify('w', vim.log.levels.WARN)
end)
vim.schedule(function()
  vim.notify('e', vim.log.levels.ERROR)
end)

with set ch=0 displays only e for me. Both messages are visible by :mes.

[luukvbaal]

For whatever reason msg_show->replace_last == true for the second message.

The only place I know of where replace_last is useful for us is where Nvim sends two message events for an incomplete message to show busy state, later to be appended to (e.g. when writing a file).

But maybe we should just identify those two messages by it's content because I don't think we want to honor replace_last anywhere else. It's only useful to exactly mimic the current message grid implementation when it replaces a message that will not be scrolled.

[przepompownia]

I didn't look how messages with search_count, bufwrite, undo and completion kinds are handled here but found replace_last flag useful for those kinds (and nothing else) when started writing my custom plugin.

[przepompownia]

The recent commit makes the above case working as expected, thanks!

[przepompownia]

Do you consider adding (optional?) LSP progress handler here or (if it could be out of scope of this PR) at least some user documentation how to implement it?

[luukvbaal]

Definitely seems out of scope to me, I guess you want to re-use the message box for that?

[przepompownia]

Yes, it would be a pity to waste this opportunity. At the first glance M.show_msg allows replacing the recent message, but there is no guarantee that it would be a progress message. I cannot find here a way how to access a message marked before as a special and update it (and probably display it at some fixed position, .e.g. lowest).

[clason]

Definitely out of scope. Recall the stated problem:

We have an unmaintained Vimscript parser and cmdline
highlighting mechanism, with which it is hard to leverage the
treesitter highlighter. Long messages result in a hit-enter-prompt.

Let's get this solved; then we can go from there. This PR has been open for more than a year; please help get it done, not delay it further.

[justinmk]

I'm trying this out locally. Since it's opt-in currently, there are of course no real blockers to merging, but we definitely want the next step to be to get rid of old code and make this the default.

some drive-by comments:

• ctrl-c or something should probably exit the g> window? currently requires ctrl-w c
• the cmdline border is a bit much, and doesn't seem to respect 'winborder' ?

[luukvbaal]

The cmdline shouldn't have a border (or respect 'winborder' for now, I think). If it does that's a bug, one that I haven't encountered.

I'll add a mapping for ctrl-c.

[bfredl]

Should be low-risk to merge this now, and iterate and fix things on master.

We can decide when 0.12 is drawing closer whether it is ready to be advertised as a feature or remain a WARNING: EXPERIMENTAL thing for that release.

[garydwatson]

this is probably the wrong place to say this seeing as how this is already merged... however... when enabling this feature by...
require('vim._extui').enable({}) enables the experimental UI.
{ msg.pos = 'box' } or :set cmdheight=0 enables the message

it completely breaks the :ls / :buffers command... the list of buffers has a paged view that can be scrolled up and down either line by line or a page at a time, and futher, you can hit ":" at any point while looking through the list to enter command mode so you could do a ":bd 1 7 8 9 42" while looking at the list...

if you replace that very fully functional view with this, then buffer navigation and management will effectively be broken.

[justinmk]

the :ls / :buffers command

Use g< to enter the pager. There will also be ways to change the behavior. And we're still working out the default behavior.