feat(highlights)!: redesign of highlighting approach

[xiyaowong]

This PR rewrites the approach for handling highlights. It now renders cells based on columns. It aims to maintain consistency with nvim in most daily use cases. Only a few rare edge cases in daily work may still have issues.

• rendering
  Fixes bug: hop.nvim doesn't work correctly when the text contains Korean, Japanese, and Chinese #1440

• syntax
  Resolve: Set syntax on #519 by feat(highlight): rework 'syntax' xiyaowong/vscode-neovim#1
  Close: bug: The plugin vim-matchup doesn't work properly #1468
  Close: bug: Incorrect indentation after o #799

Some demonstration images.

• hop & flash

• tabs

• emojis

• simple benchmarks

Regular document

Unusual document 👀

Although there is noticeable lag in the second image, I still think it is sufficient.

[5h5ong]

normal.mov

writing.mov

code.1.clojure.mov

code.2.javascript.mov

I'm using flash.nvim. It works smoothly in real usage environments. It seems that the problems i used to face have been wll resolved.

[xiyaowong]

Marked as “ready for review” to get attention and assistance with testing from others.

[theol0403]

I'm extremely busy these days so might take me a bit to review this, just because it's a sensitive area and there are a lot of changes. Thanks for your work on this!
Fwiw eol highlighting is important for fake visual cursor.

[theol0403]

Also, I haven't really looked at the code, but I feel like the highlight provider has the potential to reduce a lot of lines of code/simplify, if that's at all something you can work towards, as it was pretty messy before.

[xiyaowong]

This PR took up almost all of my time last week 😷. For the most part, it’s stable in most scenarios now, and I’m currently using the latest compiled extensions in my daily work. The remaining issue lies with multi-byte characters due to differences in handling character encoding length between positions in vim, the editor, and CSS.

But these are mostly edge cases that most people wouldn’t encounter in their daily work. For example, if you try selecting, moving, or manipulating this string: “🎉🔧🥇🎉 🔧 🥇 🤝 🎉 🔧 🥇 ❤️”, you’ll see where the issue lies. Through debugging, I’ve confirmed that the current position calculations are problematic for multi-byte characters. While I attempted to fix it, I found that it would impact other modules and introduce new issues. Therefore, I decided not to continue with the fix as I don’t want to invest too much effort in solving a rarely encountered problem. Personally, I prefer addressing such rare cases in the future.

[theol0403]

Hahah it's easy to get sucked into spending a lot of time on this project 😉

[theol0403]

A quick test, everything seems to be working. I'll have to review the code later.

[theol0403]

Sometimes, when using leap.nvim, I get some labels that float & get stuck well past the end of the line:

[justinmk]

Sometimes ... some labels float & get stuck

I see that with the current release (CTRL-L clears it up), so I wouldn't say it's a blocker for this PR.

[xiyaowong]

@theol0403 @justinmk Solution for syntax. xiyaowong#1

[xiyaowong]

@theol0403 I think this pr is stable enough now. It can be reviewed and considered for merging. Due to the numerous changes in this PR, it has become challenging to proceed with other refactoring works(#1471) and new features. Dealing with conflicts can be a headache.

[theol0403]

Sorry for the wait, it looks great. Just clean up a few of the commented code that isn't being used.

I don't have time to extensively test, but we can fix things as they come up.

[theol0403]

Feel free to merge after you've done the final touches.

[stevenmqnguyen]

Given this redesign, is flash.nvim able to do away with manually setting colors and instead use the colors of current vscode theme?

I'm not sure if I completely understand how the interaction between vscode-neovim and flash.nvim work but I tested removing the custom vscode logic in flash.nvim and it seems like in vscode the highlights aren't showing up.

[xiyaowong]

@stevenmqnguyen

Given this redesign, is flash.nvim able to do away with manually setting

No, plugins should not link to the builtin groups. xiyaowong#1 (comment)

There are two ways to customize colors:

1. set colors in nvim
2. set colors in vscode.
   References:
   • https://github.com/vscode-neovim/vscode-neovim/blob/4d7a9be9be95ea5e8da793fd810345171a865cd2/package.json#L190C1-L200C26
   • ThemeColor