Expose the index of syntax element locations to vimscript

[yomotherboard]

Is your feature request about something that is currently impossible or hard to do? Please describe the problem.

It is currently hard to find information about where syntax items are located in a file in vim. It is possible to use synID() and synIDattr() to find information about a specific location, but this becomes very convoluted and inefficient to use when syntax information around a general area or across the whole file is needed. This has led to many reimplementations of elements of the syntax engine in even highly popular plugins (e.g. commentary.vim, NERD_commenter).

I am beginning to write a plugin which extends vim's "in" and "around" movement patterns to allow for more flexible and universally "jumping to the next occurrence" when the cursor isn't currently within the delimiters. For example in native vim you can use ci" at the beginning of a line which later contains a quoted string, and vim will jump to those quotes before performing the normal ci" operation.

Having a knowledge of what syntax elements (for example vimStrings in a .vim file) are at which locations would allow me to solve this problem in a far less hacky and redundant way. Given that the syntax highlighting engine must already know these locations to perform the highlighting, I think this would only be a matter of making this information available in vimScript.

Describe the solution you'd like

Create a function or set of functions that allow broad information on the location of various syntax elements in a file. I think a minimal solution that would provide all the necessary information would be a function that returns a list of entries [synID, lineStart, lineEnd, colStart, colEnd] for either the whole file or some range of lines. To go further a function could be provided that takes a syntax element name/ID and returns a list of entries [lineStart, lineEnd, colStart, colEnd] for all syntax elements with that name/ID.

Describe alternatives you've considered

I have considered using the output of :syntax list {group} somehow to use the regex definitions to find matches. But this seems not only very difficult, but also highly redundant. Not only would it be reimplementing the vim's syntax engine, but it would be searching through a file for items that have clearly already been indexed by the syntax engine.

Additional context

The syntax of a file is useful for far more than syntax highlighting. I believe that making this available would lead to the development of many novel solutions to text-editing inefficiencies. I also believe it would improve the performance of popular vim plugins and eliminate redundancy.

[brammool]

Quote: "the syntax highlighting engine must already know these locations". That is only true when displaying text. Once a character has been drawn the information can be dropped. There is some caching going on to improve the performance, but Vim does not remember the start and end of each match. It's a very complicated mechanism, since a new match can appear at just about any position. Thus the only way to get all information is to go over the text from start to end.

There are ideas to parse the text once and keep all the state. Now that most PCs have plenty of memory this has become much more viable. Still, at every change of the text the state must be updated in a clever way. Imagine inserting "#if 0" in a C file, all the rest of the file is suddenly a comment. Neovim has included Treesitter, which is an implementation of this. Once Vim9 is done I'll have a look at whether it is a good choice to include with Vim.