RFC: Adopting full syntax support into vimtex

[lervag]

As many may know, vimtex does not provide its own syntax plugin. Instead, it builds on top of the one provided by Dr. Chip which is shipped with both Vim and neovim.

I've started to think about changing this. As a first step, I would adopt the latest version by Dr. Chip and rewrite it in terms of vimtex code style. Through some iterations of improvements, I think this could lead to a reduced workload with respect to maintenance and further development. Pros and cons of the top of my head.

Pros:

1. Having full control means we can merge and optimize the implementation.
2. Having full control means we can fix issues raised here that are not really vimtex issues.
3. Can add more tests for syntax related features to assist in controlled development.
4. I may learn more about syntax scripts.
5. May be possible to fix some sync related bugs and similar. With time.
6. It would mean that vimtex becomes a full filetype plugin in every sense, i.e. no clash with internal plugins.

Cons:

1. More work for me.
2. More area where there can be failures.
3. Would need to consider some cases like whether g:tex_flavor should be respected/overwritten and similar.
4. Unsure about the license of tex.vim and whether it is OK to "fork" it.

My thoughts for how to do it is something like this:

• Copy latest version.
• Clean up code (non breaking), adapt option names into vimtex schemes (e.g. g:tex_conceal into something like g:vimtex_syntax_config = {'conceal': ...} and so on).
• Remove some legacy stuff (e.g., I think perhaps things like g:tex_fast is unnecessary, similar with g:tex_nospell and perhaps also g:tex_conceal).

In any case, I would be happy to hear other peoples thoughts and opinions on this.

[clason]

I would welcome this (not that Dr Chip isn't doing a great job!) and couldn't think of a person I'd trust more. Con 1 is of course a big con and would understandably be sufficient in itself.

The possibility of faster updates would indeed be appealing -- there was a recent nasty bug in the syntax file that took quite a long time to percolate down to neovim. (Of course, it's trivial to put a fixed file into .config/nvim/syntax, but still.)

Another advantage is that you can be more opinionated about defaults than a default vim syntax -- for example, finally making latex the default TeX flavor. Now that you target vim8, you'd also be free to rely on more recent vim features should those improve either performance or maintainability (although I'm not sure how much room for either there is in a syntax file).

[adigitoleo]

Hi,

Firstly, let me just say that I appreciate this plugin very much. I was able to write my honours thesis in neovim and it was awesome.

Now, back on topic, I have noticed one issue in the syntax highlighting for a while, but it was not a big deal so I didn't worry. I came here to post an issue, but maybe I will piggyback in this thread first, since I am not sure now if it is a problem in vimtex or in the syntax file.

When I place the cursor on a begin{<env>} (or end{<env>}), the the current line as well as the matching \end (or \begin) are highlighted (I think using hi Search?). Now, when I switch to the next/previous buffer the highlights remain visible on those lines. In other words, these environment highlights are not local to the tex buffer or are not cleared when leaving. It seems to be the same regardless of set hidden.

Does vimtex draw these highlights? Or should I go to the syntax maintainer (Dr Chip?). It should be trivial to reproduce, so I could post a proper issue if you prefer.

Since I am here, I will give a quick vote on full syntax support: Yes. I agree with @clason that such a framework (even as an opt-in) could be useful. I am very green with LaTeX, but it seems there are many versions/variations and people could hopefully help to add support for their favourite setup. For example, I would welcome support for TikZ and friends.

[Rmano]

+1; notice that just looking around I found this: https://github.com/gi1242/vim-tex-syntax (not tested).

Another really nice thing would be to add LaTeX3 syntax highlighting; it would be activated between \ExplSyntaxOn and \ExplSyntaxOff --- right now it's a bit a pain in the back:

(I'll copy here the file if somebody is interested in giving it a try; notice that the syntax highlight on GitHub is also off, this is quite new stuff)

There is a "dirty hack" here, that I use manually: https://tex.stackexchange.com/questions/44624/vim-configuration-for-expl3

% see https://tex.stackexchange.com/questions/557511/how-to-query-iffilehaschanged
\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
    \str_new:N \g_myfile_name_str
    \str_gset:Nn \g_myfile_name_str {myfile.txt}
    \str_new:N \g_myfile_old_mdfive_str
    \str_new:N \l_myfile_mdfive_str
    \cs_new_protected:Npn \getmdfive % get hash is not expandable, so protect
    {
        \file_get_mdfive_hash:nN {\str_use:N \g_myfile_name_str} \l_myfile_mdfive_str
        \str_use:N \l_myfile_mdfive_str
    }
    \cs_new:Npn \getoldmdfive
    {
        \str_use:N \g_myfile_old_mdfive_str
    }
    \cs_new:Npn \IfMyfileChanged #1 #2
    {
        \str_if_eq:NNTF \g_myfile_old_mdfive_str \l_myfile_mdfive_str {#2} {#1}
    }
    \AtEndDocument
    {
        \iow_now:cx { @auxout }
        {
            \token_to_str:N \ExplSyntaxOn
            \iow_newline:
            \str_gset:Nn \token_to_str:N \g_myfile_old_mdfive_str {\str_use:N \l_myfile_mdfive_str}
            \iow_newline:
            \token_to_str:N \ExplSyntaxOff
        }
    }
\ExplSyntaxOff
\begin{document}

MD5 was \getoldmdfive

MD5 is now \getmdfive

Changed? \IfMyfileChanged{Yes, it has changed}{No, it's the same as before}

\end{document}

[clason]

Another thing that this would allow is cleaning up and consolidating (or extending, if necessary) the highlight groups. I've been thinking of finally biting the bullet and writing my own tex-specific color scheme and ideally would like to color math commands differently from text commands (and differently from environments and labels etc.), and this would be a good opportunity. (Let me know if discussion of tex color schemes would make for a good separate issue.)

If you do go for this, I'd second @Rmano's suggestion of looking at https://github.com/gi1242/vim-tex-syntax as a basis, which looks quite clean.

Also, @Rmano: did you see https://github.com/wtsnjp/vim-expl3?

[Rmano]

Also, @Rmano: did you see https://github.com/wtsnjp/vim-expl3?

No, I was not aware... is it compatible with vimtex? I really do not know how to use it alognside the normal latex. It should be a sublanguage, triggered by \ExplSyntaxOn and ..Off or the expl3 sprecific include file (I do not remember how to check them, but for sure not by the extension).

[clason]

I would assume it won't conflict so you can just load it, but I don't know the best way to inject it into a tex syntax; I'd look into how vimtex extends the builtin syntax for the different packages in https://github.com/lervag/vimtex/tree/master/autoload/vimtex/syntax.

In any case, Expl3 support should probably be a separate issue, independent of whether the base tex syntax file is replaced.

[lervag]

@clason @Rmano @adigitoleo Thanks for the support/votes! I think I want to try this, but as said and as I think you understand, it will not be something I'll be doing first thing. Perhaps I'll start working on it and finishing a version before Christmas, but I won't make promises.

@clason

... not that Dr Chip isn't doing a great job!

Hear hear!

Another advantage is that you can be more opinionated about defaults than a default vim syntax -- for example, finally making latex the default TeX flavor.

Yes, this is one of the reasons I want to do this. I think and hope that it is possible to both simplify and "standardise" more on what I think are more sensible and modern defaults.

(Let me know if discussion of tex color schemes would make for a good separate issue.)

It might be. I'm not 100% sure what you mean, and I don't mind discussing things. I would prefer to keep the topics separated, so let's do it in a separate issue.

@adigitoleo

Firstly, let me just say that I appreciate this plugin very much. I was able to write my honours thesis in neovim and it was awesome.

Thanks! :D

I came here to post an issue, but maybe I will piggyback in this thread first, since I am not sure now if it is a problem in vimtex or in the syntax file.
...

The feature you describe sounds like the matchparen feature, see :help g:vimtex_matchparen_enabledand alsovimtex-nf-enhanced-matchparen`. If you want to further discuss this, please open a new issue for it.

@Rmano

Another really nice thing would be to add LaTeX3 syntax highlighting; it would be activated between \ExplSyntaxOn and \ExplSyntaxOff --- right now it's a bit a pain in the back:

Interesting! But for my own sanity and for keeping topics separated, I would prefer if we discuss this in a new/other thread. It seems @clason found a relevant plugin. If it does not immediately work with vimtex, feel free to open an issue and we can discuss it further there.

@Rmano and @clason

... I found this: https://github.com/gi1242/vim-tex-syntax (not tested).

Thanks for the heads up!

If you do go for this, I'd second @Rmano's suggestion of looking at https://github.com/gi1242/vim-tex-syntax as a basis, which looks quite clean.

I agree that it at looks clean. I'm worried, though, that it would require a more full cleanup of all the syntax stuff. Has anyone tried to use vim-tex-syntax with vimtex?

---

When I start work on this I will definitely work in a separate branch that will be merged as a PR. The change will be significant, and so I think it might require a major version bump when the PR is merged. Thus I also don't want to merge until I'm confident that users will

1. not really notice any major changes, or
2. recognize the change as an improvement.

I will be very happy if I get help with testing when things are progressing. I would also like feedback on which of the old/legacy features are important to keep and which can be "safely" removed.

One relatively important aspect is efficiency. Thus, it may be very useful to be able to test the speed of the syntax before and after. I want to simplify things and remove e.g. g:tex_fast, but this means I must be sure that the script is actually fast enough! If anyone has a good idea for how to test this that would be much appreciated.

[clason]

It might be. I'm not 100% sure what you mean, and I don't mind discussing things. I would prefer to keep the topics separated, so let's do it in a separate issue.

👍 (it does have some bearing on this, since it involves the highlight groups that are defined in the syntax file and how they are linked, but it makes sense to first discuss what can be done with the current file).

I agree that it at looks clean. I'm worried, though, that it would require a more full cleanup of all the syntax stuff. Has anyone tried to use vim-tex-syntax with vimtex?

I've briefly put it in my config/nvim/syntax/ directory, and nothing broke -- some things looked better (with my color scheme), some worse, and I noticed about a 60% reduction in startuptime (for the syntax file only!)

When I start work on this I will definitely work in a separate branch that will be merged as a PR. The change will be significant, and so I think it might require a major version bump when the PR is merged. Thus I also don't want to merge until I'm confident that users will
...
I will be very happy if I get help with testing when things are progressing. I would also like feedback on which of the old/legacy features are important to keep and which can be "safely" removed.

Might I suggest then instead to develop on the main branch but put the new syntax behind a (by default off) option g:vimtex_new_syntax? Then if you're happy, you can just flip this to default on (and bump the version). This way you get the development version under more eyes and make it easier for us to do A/B testing (or turning it off temporarily if it breaks for a while).

(Assuming this is even feasible.)

One relatively important aspect is efficiency. Thus, it may be very useful to be able to test the speed of the syntax before and after. I want to simplify things and remove e.g. g:tex_fast, but this means I must be sure that the script is actually fast enough! If anyone has a good idea for how to test this that would be much appreciated.

Definitely (you know me ;))! Another aspect would be to allow using "high-level" syntax groups to be used in ex commands -- for example, replace only in math zones, or outside math zones (but not commands) etc.

[Rmano]

Interesting! But for my own sanity and for keeping topics separated, I would prefer if we discuss this in a new/other thread. It seems @clason found a relevant plugin. If it does not immediately work with vimtex, feel free to open an issue and we can discuss it further there.

Done!

[Rmano]

Might I suggest then instead to develop on the main branch but put the new syntax behind a (by default off) option g:vimtex_new_syntax? Then if you're happy, you can just flip this to default on (and bump the version). This way you get the development version under more eyes and make it easier for us to do A/B testing (or turning it off temporarily if it breaks for a while).

(Assuming this is even feasible.)

If it's feasible, I agree. Another possibility is to define a syntax like tex+ and then switch manually during the development, with set ft, and rename when ready...

[adigitoleo]

@lervag Thanks for the pointers, you are correct my issue is related to matchparen and it's not specific to vimtex.

I will keep an eye on the syntax side of things and help if/when I can, although I can't say I'm very experienced in vim-fu (or development in general 😄 )

[lervag]

I've briefly put it in my config/nvim/syntax/ directory, and nothing broke -- some things looked better (with my color scheme), some worse, and I noticed about a 60% reduction in startuptime (for the syntax file only!)

Ok, that does sound good! I'll look into if it is possible to use this as a starting point, then.

Might I suggest then instead to develop on the main branch but put the new syntax behind a (by default off) option g:vimtex_new_syntax? Then if you're happy, you can just flip this to default on (and bump the version). This way you get the development version under more eyes and make it easier for us to do A/B testing (or turning it off temporarily if it breaks for a while).

Good idea! I'll do this. But note, the changes will require changing a lot of the current syntax code as well, and when that starts to happen I can't simply "flip a switch". So I might take dev to a branch at that point. But as long as it is possible to keep the current behaviour intact while working on the new syntax scripts I can do this with a temporary option g:vimtex_use_new_syntax. I'll not document this option, because it will be discarded when the work is completed. Does this sound reasonable?

Definitely (you know me ;))! Another aspect would be to allow using "high-level" syntax groups to be used in ex commands -- for example, replace only in math zones, or outside math zones (but not commands) etc.

Great! :D I'm not sure I fully understand what you mean by high-level syntax groups, perhaps you could give an example? Or we could discuss it later, unless it is something worth discussing early to avoid redundant work?

[lervag]

I've outlined the work in #1799. Feel free to comment on the plan. If you think I should do things differently I'll be glad to hear it and discuss.

This thread is now deprecated by #1799 and I'll close it.

[clason]

But as long as it is possible to keep the current behaviour intact while working on the new syntax scripts I can do this with a temporary option g:vimtex_use_new_syntax. I'll not document this option, because it will be discarded when the work is completed. Does this sound reasonable?

Yes, perfect!

Great! :D I'm not sure I fully understand what you mean by high-level syntax groups, perhaps you could give an example? Or we could discuss it later, unless it is something worth discussing early to avoid redundant work?

It's probably not quite relevant at this point, but maybe it helps keeping in mind?

What I'm hoping for is something like a syntax group isMath (or isText or isCommand) that aggregates the relevant more specific syntax groups so I can do something like

:%s/foo/\=synIDattr(synID(line('.'),col('.'),1),'name')==#'isMath'?bar:foo/g

to only replace foo by bar in math (or text, or command) zones; see also https://www.vim.org/scripts/script.php?script_id=848. (Naturally, I'd write a macro for this -- or maybe vimtex could provide them? 😄 Separate issue, though.)

[lervag]

What I'm hoping for is something like a syntax group isMath (or isText or isCommand) that aggregates the relevant more specific syntax groups so I can do something like ...

Ah, I see. I mean, this is sort of already possible? At least in the sense that all math is captured by groups like texMathZoneA. If I understand you correctly, you are thinking that this should all be more consistent? In this sense, I think it would help with a good discussion of how to interpret latex code itself. E.g., it seems to me that we can separate things into mostly "text" and "math". However, one can also distinguish e.g. TikZ and pgfplots content as a sort of separate thing.

[clason]

Ah, I see. I mean, this is sort of already possible? At least in the sense that all math is captured by groups like texMathZoneA. If I understand you correctly, you are thinking that this should all be more consistent?

Consistent, and documented, yes -- I think that is the core of what I'm asking about here (and in the color scheme issue).

In this sense, I think it would help with a good discussion of how to interpret latex code itself. E.g., it seems to me that we can separate things into mostly "text" and "math". However, one can also distinguish e.g. TikZ and pgfplots content as a sort of separate thing.

That's what I was trying to get at in #1800. Besides text and math, I'd distinguish "tex commands" -- TikZ and pgfplots would fall under that. (If I want to do something TikZ specific, I'll probably either do vae first or have them in separate files anyway.)

By the way, after writing that comment, I realized that there's already

:s/A/\=vimtex#util#in_mathzone()?'B':'A'/g

Not documented, though ;)

[adigitoleo]

@clason Hi, you wrote:

If I want to do something TikZ specific, I'll probably either do vae first or have them in separate files

FWIW I agree, that is probably smarter than adding tikz specifics, generic "tex commands" patterns are probably easier to maintain. I only recently started using multiple files -- could you elaborate a bit on what vae does? Do you still use tex filetype for the tikz stuff?

[Rmano]

I do not know if it's possible (I am not an expert at all here --- I know what a RE is, but I suspect that syntax highlight is more than this), the best way is to have sublanguages. I think there should be a kind of "sub-mode" for example for LUA code in LuaLaTeX (this is often inline...), and TikZ would be probably treated like this.

[clason]

@adigitoleo vae visually selects all (as opposed to inside) the current environment. Try it in a tikzpicture (or any LaTeX) environment :) And yes, I have an autocommand that sets ft=tex for .tikz files.

@Rmano That is how TikZ is currently treated. To actually inject another, full, language syntax (e.g., in Markdown code blocks) in vim is currently not possible, and one of the problems that tree-sitter is meant to solve. (Not supported yet there, either, but it's on the docket.)

[lervag]

By the way, after writing that comment, I realized that there's already

:s/A/\=vimtex#util#in_mathzone()?'B':'A'/g

Not documented, though ;)

Yes, sorry. I've not really documented the "developer API". Pragmatic choice, really; everything takes time, and I've barely got enough time as it is. I try to write the code in a manner that makes it relatively easy to read, though.

[clason]

Oh, that wasn't a complaint, and no need to apologize! (Though you do have to know what's there in order to look at the code for it.)

Take it simply as a (single) vote for how useful that function (and similar) might be and for promoting it to a "user API".

[lervag]

Thanks, and I agree that there are probably quite a lot of more or less useful functions such as vimtex#util#in_mathzone that (skilled) users can leverage for advanced customization. Perhaps this is something that could be added to the github wiki? I'll admit I don't maintain that much myself, but I'd be happy to help review and describe such a user API section if anyone were to attempt to write one. :)

[clason]

Yes, I think the wiki would be a good place for this (e.g., as a subsection "Advanced usage" under "Usage"). It wouldn't be quite as discoverable as in the documentation, but should still be visible to Google and could be linked from other places.

The main issue remains knowing what is there in the first place -- I only know about vimtex#util#in_mathzone because you mentioned it a long time ago in a comment on a similar question of mine, and I for once had the presence of mind to write it down in a cheat sheet :)

Maybe we can open a new issue for other people to share their favorite "pro tips" (especially you ;)) to collect them for the wiki?

[lervag]

Maybe we can open a new issue for other people to share their favorite "pro tips" (especially you ;)) to collect them for the wiki?

Yes, that's OK. I don't really have that many "pro tips", though; if I "want" something it's typically just in the "defaults" or provided as an option :)

But I'll definitely help in creating this as it seems it could be a useful resource.