feat(defaults): enable 'termguicolors' by default when supported by terminal

[gpanders]

Enable 'termguicolors' automatically when Nvim can detect that truecolor is supported by the host terminal.

If $COLORTERM is set to "truecolor" or "24bit", or the terminal's terminfo entry contains capabilities for Tc, RGB, or setrgbf and setrgbb, then we assume that the terminal supports truecolor. Otherwise, the terminal is queried (using both XTGETTCAP and SGR + DECRQSS). If the terminal's response to these queries (if any) indicates that it supports truecolor, then 'termguicolors' is enabled.

---

This follows the process documented in https://github.com/termstandard/colors#checking-for-colorterm.

Experimental/test results:

'termguicolors' is correctly set on:

• iTerm2
• Ghostty
• Wezterm
• Kitty
• Alacritty
• tmux

All of the terminal emulators listed above set $COLORTERM, so 'termguicolors' support is assumed from that. If we explicitly unset $COLORTERM and force TERM=xterm-256color (mimicking an SSH session lacking proper terminfo, for instance), the results are all still correct except for Alacritty, which does not include Tc, RGB, setrgbf, or setrgbb in its terminfo and does not support XTGETTCAP or DECRQSS.

'termguicolors' is correctly not set on:

• Terminal.app

I was not able to directly test foot or Xterm. I'm confident foot will work. Xterm should work since it parses DECRQSS (and I even bent over backwards to ensure we parse Xterm's DECRQSS response, since it differs from everyone else's).

[echasnovski]

I can confirm that stock st does set termguicolors by default and it does indeed support it.

[justinmk]

Would it reduce potential "flicker" if we actually make this the hardcoded default?

neovim/src/nvim/options.lua

Line 8762 in 1fd29a2

defaults = { if_true = false },

So the detection logic is used to disable the default, if necessary. But the common case is that 'tgc' is enabled and don't need to redo its side-effects.

[justinmk]

Probably a good time to introduce a _tui.lua or _tty.lua file, since we will be adding more and more TUI stuff here.

[gpanders]

Would it reduce potential "flicker" if we actually make this the hardcoded default?

neovim/src/nvim/options.lua

Line 8762 in 1fd29a2

defaults = { if_true = false },

So the detection logic is used to disable the default, if necessary. But the common case is that 'tgc' is enabled and don't need to redo its side-effects.

Either way it has no effect on the flicker. But I agree that 'termguicolors' being enabled will be the common case, so better to optimize for that. I've changed the hardcoded default to enabled now.

Probably a good time to introduce a _tui.lua or _tty.lua file, since we will be adding more and more TUI stuff here.

Maybe. For now all of this stuff is still just part of "defaults" though.

[gpanders]

While updating the tests in tui_spec.lua I did a little bit of refactoring to remove some of the messy and difficult to understand/modify string munging and eliminated a few old TODOs along the way. Hopefully the tui_spec tests are slightly less scary now.

[gpanders]

Would it reduce potential "flicker" if we actually make this the hardcoded default?

neovim/src/nvim/options.lua

Line 8762 in 1fd29a2

defaults = { if_true = false },

So the detection logic is used to disable the default, if necessary. But the common case is that 'tgc' is enabled and don't need to redo its side-effects.

Either way it has no effect on the flicker. But I agree that 'termguicolors' being enabled will be the common case, so better to optimize for that. I've changed the hardcoded default to enabled now.

Actually, I need to think about this more. This gets tricky in the case that $COLORTERM and terminfo aren't available:

1. In that case, we disable 'termguicolors'
2. We then query the terminal to see if it supports 'termguicolors', and enable it if it does, but only if the user hasn't already set 'termguicolors'
3. But because we ourselves set 'termguicolors' in (1), we can no longer tell if the user has set it!

[justinmk]

3. But because we ourselves set 'termguicolors' in (1), we can no longer tell if the user has set it!

But if it's already enabled, why do we need to set it again? And conversely, if the user disabled it, we can skip/ignore all of our detection stuff.

[gpanders]

3. But because we ourselves set 'termguicolors' in (1), we can no longer tell if the user has set it!

But if it's already enabled, why do we need to set it again?

We disable it if we can't tell that the terminal emulator supports truecolor (i.e. when $COLORTERM is undefined or not set to truecolor/24bit and when terminfo doesn't contain Tc/RGB). We then re-enable it only if querying the terminal shows that the terminal actually does support truecolor. We cannot wait until the query is finished to disable 'tgc' because that would result in a visible (and annoying) flicker effect on terminals which don't support truecolor.

Keeping the hardcoded default false and only enabling if we detect support avoids this problem (this is what I had implemented before this comment). I may revert back to that if I can't think of a solution here.

EDIT: We can check against last_set_sid. A little hacky but it will work.

EDIT 2: No, that won't work, because all Lua scripts have the same SID (-8).

And conversely, if the user disabled it, we can skip/ignore all of our detection stuff.

The detection stuff runs before any user config files, so it's not possible for the user to set it before that happens. We have to check if the user has disabled it when the responses to our terminal queries arrive.

[gpanders]

Reverting the hardcoded default back to false and enabling in _defaults.lua (instead of vice versa). This works around the issues described above (and the visible effect for the user is identical).

[gpanders]

Last failing test:

FAILED   1 test, listed below:
FAILED   test/functional/terminal/tui_spec.lua @ 1851: TUI argv[0] can be overridden #23953
test/functional/terminal/tui_spec.lua:1865: Row 1 did not match.
Expected:
  |*{1: }                                                 |
  |*{4:~                                                 }|
  |*{4:~                                                 }|
  |*{4:~                                                 }|
  |*{5:[No Name]                       0,0-1          All}|
  |                                                  |
  |{3:-- TERMINAL --}                                    |
Actual:
  |*                                                  |
  |*[Process exited 0]{1: }                               |
  |*                                                  |
  |*                                                  |
  |*                                                  |
  |                                                  |
  |{3:-- TERMINAL --}                                    |

But this fails for me on master as well.

EDIT: To clarify, this fails for me on master on macOS. I can't reproduce the failure on Linux on this branch, with any build options.

[lewis6991]

Great work on the turnaround time for this!

[justinmk]

This test is failing about 40% of the time now:

FAILED   test/functional/terminal/tui_spec.lua @ 1449: TUI paste: streamed paste with isolated "stop paste" code
test/functional/terminal/tui_spec.lua:1493: Expected objects to be the same.
Passed in:
(table: 0x56350fb6d640) {
  [1] = 1
  [2] = 2
 *[3] = 2
  [4] = 3 }
Expected:
(table: 0x56350fb587a0) {
  [1] = 1
  [2] = 2
 *[3] = 3 }

stack traceback:
	(tail call): ?
	test/functional/terminal/tui_spec.lua:1493: in function <test/functional/terminal/tui_spec.lua:1449>

[gpanders]

@justinmk should be fixed by #26437