The new default colorscheme breaks legacy vim colorscheme: workarounds and future steps?

[wookayin]

First of all, I appreciate the huge efforts and work done in the new default colorscheme #26334, it's a great feature and nice enhancement of neovim.

I hope this can be a follow-up thread as a part of the tracking issue #26369, about the awesome new default colorscheme introduced in #26334. Initiated from my reddit comment:

Description of the problem

One issue I have is that the new default colorscheme to break many existing legacy colorschemes that are based on the plain "vim" colorscheme. Those legacy colorschemes --- including those are "shipped with neovim" and community-maintained collections or curations --- are affected and now look quite different because they did not exhaustively define all the highlight group being used, i.e. because of their implicit dependency/assumptions on the default (vim) colorscheme.

Colorschemes usually have hi clear at the beginning of the script file, but this reverts all the highlight configs back to the new neovim default; so having some lines like :colorscheme vim or :source colors/vim.vim early in init.vim will have no effect. These lines must be put after hi clear in each of the colorscheme script.

For example, if a colorscheme does not explicitly define "String" as :hi link String Constant (which used to default to "Constant" -> "Red"), this will no longer have a highlight color same as "Constant" but "NvimLightGreen". This is a problem of a legacy, outdated colorscheme, so shouldn't be a major problem or blocker, but we might have some decisions or suggested solutions for those legacy colorschemes.

Also, the built-in colorschemes https://github.com/neovim/neovim/tree/master/runtime/colors would need an corresponding update or a removal.

Workaround or solutions

On an end user's side:

• (1) Users should avoid using legacy vim colorschemes, and move to the modern, neovim-aware plugins that have a continued neovim support.

• (2) An user whose config is unfortunately based on an "old" vim colorscheme (say base_legacy) can copy (or "fork") colors/base_legacy.vim into their neovim config and add the source colors/vim.vim line after hi clear). Sorry, you can't use a plugin any longer unless it is still maintained.

• (3) Update your existing colorscheme or config to explicitly configure all the built-in highlight groups, that behave differently e.g. WinSeparator, String, Function, MoreMsg, WarningMsg, etc.

Community/plugin:

• (4) The maintainer of colorschemes should update their plugin to make it "aware of neovim 0.10" -- but I don't think is always possible; Many "legacy" colorschemes are no longer updated and maintained (https://github.com/flazz/vim-colorschemes). I don't expect all the (legacy) colorschemes in such a "collection" (even including a fork) can be updated.

Neovim side:

• (5) Add a backward-compatibility option for hi clear to allow an user to explicitly opt out of the new default colorscheme; e.g. if some switch is turned on, hi clear will revert to (or source) the vim colorscheme. Or, if :colorscheme vim is ever called once in user's init.vim or init.lua, hi clear will revert to the vim colorscheme instead of the new neovim default.

  Since this is only to support an old vim legacy, I'm not sure if it is really worth doing to add a new flag or configuration point. But I don't see any other solutions that can make legacy vim colorschemes look exactly the same without changing the behavior of hi clear.

[clason]

We should fix the bundled colorschemes, as these are useful and maintained (e.g., by adding a colorscheme vim after the hi clear). For external "legacy" colorschemes -- especially unmaintained ones -- I don't think there's much we can and should do, they are missing too many modern Neovim highlight groups (UI, treesitter, LSP) to be really useful.

[clason]

I'm fine with "breaking changes" because neovim's decision is to force users to actively move away from legacy colorschemes.

This is false; nobody is moving anybody away from anything. We just don't consider spending effort on unmaintained external projects worthwhile.

One possible proposal is to make :colorscheme vim (more than :source colors/vim.vim) set some global variable (like g:colors_name set by :colorscheme), and hi clear be aware of this so that :hi clear work as if :colorscheme vim were done.

No, we should strive for the minimally invasive fix for the bundled(!) colorschemes. We already need to modify the boilerplate at the top due to different t_co handling, so this is not a big step.

Also, can you please show some before/after screenshots showing bundled (habamax, retrobox, quiet, etc.) colorschemes? I haven't noticed anything from my (admittedly brief) testing.

[echasnovski]

Thanks for this detailed issue!

I think that the core issue of the new default color scheme being breaking is not really fixable. There are these main culprits:

• Vim's default has group linking to the other group, while Neovim's new default has it linking to the other group or not link at all. Examples are indeed:
  • WinSeparator was linking to VertSplit, but I changed to Normal as VertSplit is deprecated in Neovim.
  • String and Function are now not linking but define their own foreground, as those are two main targets for "green" and "cyan" colors.
  • @variable and @lsp.type.variable are cleared instead of linking to Identifier to reduce visual noise.
  • DiagnosticFloating* now don't link to Diagnostic*, but this will probably be reverted.
  • There might be more...
• Vim's default defines colors explicitly, while Neovim's links them. I don't think there was such change, but I might misremember something.
• Vim's default doesn't define certain attribute (guibg, for example), while Neovim's default does. This will probably be brought to a bare minimum in the follow up PR in the (hopefully) near future.

I don't really agree on MoreMsg and WarningMsg, though. Both in Vim's and Neovim's default they only define guifg and ctermfg, son only colors changed.

My initial suggestion for the users of "old" non-built-in color schemes is to patch the highlight groups which differ from Vim's default behavior. There won't be many. Mostly because this is already the case with Neovim-specific highlight groups (NormalFloat, Diagnostic*, etc.).

For built-in color schemes (retrobox, habax, etc.) Neovim maintainers should update them. But as far as I can tell by the their code, they all define every highlight group explicitly (as color scheme plugins should, in my opinion, if they want stability). Would you mind pointing at exact difference?

[clason]

WinSeparator was linking to VertSplit, but I changed to Normal as VertSplit is deprecated in Neovim.

This was deliberate for backward compatibility, though. (Not saying we need to keep these forever, mind you, and now may be a good time for dropping it.)

For built-in color schemes (retrobox, habax, etc.) Neovim maintainers should update them.

No, I don't think so (beyond adding standard boilerplate to account for technical differences). They are maintained by vim/colorschemes (and very well so, as you mention) and we don't want to fork them.

[justinmk]

Resolution

• To close this issue:

  We should fix the bundled colorschemes ... (e.g., by adding a colorscheme vim after the hi clear).

• For externally-maintained plugins, there's nothing actionable (we already have documented the change).

[echasnovski]

The compatibility issue with most probable cause and suggested solutions are now mentioned in the breaking changes and difference to Vim.

[fionn]

(3) Update your existing colorscheme or config to explicitly configure all the built-in highlight groups, that behave differently e.g. WinSeparator, String, Function, MoreMsg, WarningMsg, etc.

Is there an exhaustive list somewhere?

[clason]

• :h group-name
• :h highlight-groups
• :h treesitter-highlight-groups
• :h diagnostic-highlights
• :h lsp-highlight
• :h lsp-semantic-highlight

or check src/nvim/highlight_group.c directly.

[echasnovski]

Plus relevant part of 'news-0.10.txt'.

[shmerl]

Is there an exhaustive list somewhere?

More in the same context, is there some example that uses all the values that you can actually open and check how it looks? I.e. I'd like to troubleshoot vim-gotham theme by comparing old and new look side by side. If someone made a comprehensive example with all the definitions used, that would expose all the regressions.

[clason]

The help pages I linked actually use the corresponding highlight groups to color the group name, for that exact purpose.

[shmerl]

Perfect, I'll use that to compare. It works as a comparison at least for those ones which aren't new.

[shmerl]

Slightly off-topic, but where is the definition of the new default color scheme? I.e. if I wanted to convert one from vimscript to lua, what can I look at as the stock example that defines all groups?

[clason]

highlight_groups.c

[shmerl]

Well, I want a lua example to make a theme. .c doesn't sound like it. So default theme isn't even defined in lua?

Is the fallback one a sufficient / good example? I.e. https://github.com/neovim/neovim/blob/master/runtime/colors/vim.lua

[clason]

Yes, the default theme is not defined in Lua but in C. (That's how it is the default.)

Please ask usage questions in discussions or on Matrix.