improvement in LaTeX3 highlighting

[xunam]

Hello,

I see that vimtex includes some support for LaTeX3 notations (since about a year, according to #1834) but I would like to suggest two small improvements to it, based on my experience using this language :

• highlight function signatures differently from function names (I find that it improves readability)
• support @@ in function and variable names, since that is used when writing packages with l3docstrip

The code I have been using for some years, putting it in .vim/after/syntax/tex.vim, is as follows:

syn match texL3Statement "\\\([a-zA-Z_]\|@@_\)\+:\@="
syn match texL3Type "\(\\\([a-zA-Z_]\|@@_\)\+\)\@<=:[a-zA-Z]*"
syn match texL3Variable "\\[gl]_\([a-zA-Z_]\|@@_\@=\)*_[a-zA-Z]\+:\@!"
syn match texL3Constant "\\c_\([a-zA-Z_]\|@@_\@=\)*_[a-zA-Z]\+:\@!"
hi link texL3Statement Statement
hi link texL3Type Type
hi link texL3Variable Identifier
hi link texL3Constant Constant
syn cluster texCmdGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texEnvGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texFoldGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texBoldGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texItalGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texMatchGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texMatchNMGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texStyleGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texPreambleMatchGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texMathMatchGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant
syn cluster texMathZoneGroup add=texL3Statement,texL3Type,texL3Variable,texL3Constant

As a side note, I named these L3 instead of E3 because after more than ten years being stable and in practical use, the LaTeX3 programming conventions from expl3 can hardly be called experimental nowadays...

[lervag]

I see that vimtex includes some support for LaTeX3 notations (since about a year, according to #1834) but I would like to suggest two small improvements to it, based on my experience using this language:

Appreciated! And sorry for the slow response!

• highlight function signatures differently from function names (I find that it improves readability)

Seems like a good idea.

• support @@ in function and variable names, since that is used when writing packages with l3docstrip

I'll take your word for this. I don't really use this stuff myself.

The code I have been using for some years, putting it in .vim/after/syntax/tex.vim, is as follows: ...

Thanks! Could I also ask you to provide a short example file that shows each of these? Perhaps also with a screenshot of how it looks for you with your custom file and how it currently looks with pure VimTeX?

As a side note, I named these L3 instead of E3 because after more than ten years being stable and in practical use, the LaTeX3 programming conventions from expl3 can hardly be called experimental nowadays...

I agree this is sensible. But does it warrant making a breaking change? I don't mind that too much - I often push breaking changes if I think it improves something.

[lervag]

I was curious about the difference between the statement, type, variable and constant. Could you explain what these are in LaTeX3, more specifically?

texL3Statement

\\\([a-zA-Z_]\|@@_\)\+:\@=

This will match things like \cmd: and \cmd@@_something:, but excluding the final colon.

texL3Type

\(\\\([a-zA-Z_]\|@@_\)\+\)\@<=:[a-zA-Z]*

This matches the :word part of e.g. \cmd:word.

texL3Variable

\\[gl]_\([a-zA-Z_]\|@@_\@=\)*_[a-zA-Z]\+:\@!

This matches things like \g_..._word: and \l_..._word:. Not sure why you use the \@= on the _ in the match group?

texL3Constant

\\c_\([a-zA-Z_]\|@@_\@=\)*_[a-zA-Z]\+:\@!

Same as above.

[xunam]

Thanks! Could I also ask you to provide a short example file that shows each of these? Perhaps also with a screenshot of how it looks for you with your custom file and how it currently looks with pure VimTeX?

It's hard to provide a precise example because most LaTeX3 code I write is in package source in dtx files, and for some reason VimTeX does not activate expl3 syntax matching in those files. Anyway, here is a piece of code using the @@ convention, taken from a package I am currently working on :

\cs_new:Nn \@@_shift_item:nn {
  \str_set:Nx \l_tmpa_str { \str_head:n { #1 } }
  \str_case:VnF \l_tmpa_str {
    { x } {
      \prop_put:Nnx \l_@@_b { #1 }
        { \dim_eval:n { #2 + \l_@@_shiftx_dim } } }
    { y } {
      \prop_put:Nnx \l_@@_b { #1 }
        { \dim_eval:n { #2 + \l_@@_shifty_dim } } }
  } {
    \prop_put:Nnn \l_@@_b { #1 } { #2 }
  }
}

This is how my Vim renders it using a solarized dark color scheme:

and the same with the @@ expanded:

and finally the same with pure VimTeX:

I agree this is sensible. But does it warrant making a breaking change? I don't mind that too much - I often push breaking changes if I think it improves something.

I don't know how much it would break. But renaming things for only cosmetic reasons in the source code does not usually qualify as good practice... Better leave the names as they are.

I was curious about the difference between the statement, type, variable and constant. Could you explain what these are in LaTeX3, more specifically?

What I call texL3Statement in this syntax file is actually what LaTeX3 calls function, i.e. a macro that is intended to be called with appropriate arguments (I think I used the word statement because it matches the meaning of Vim's Statement class). Similarly, texL3Type matches what LaTeX3 calls argument specifications, which is a concept similar to a type, since it dictates the calling convention for the function. As for variables and constants, they are exactly what their names suggest: although they are TeX control sequences, they are meant to be used as storage for information and not called as functions. For clarity and robustness, they are distinguished by precise naming conventions, described in the expl3 documentation.

\\[gl]_\([a-zA-Z_]\|@@_\@=\)*_[a-zA-Z]\+:\@!

This matches things like \g_..._word: and \l_..._word:.

It actually matches when there is no colon right after, because variables do not have a argument specifications like functions (this is part of the conventions).

Not sure why you use the \@= on the _ in the match group?

The point of the _\@= is to make sure that each @@ is followed by an underscore (which is part of the @@ convention) while avoiding a conflict with the _ after the group, which matches the required underscore before the final word in the variable name. Otherwise, something like \l_@@_int would not match (although it does not perfectly follow the expl3 guidelines since it lacks a word).

[lervag]

It's hard to provide a precise example because most LaTeX3 code I write is in package source in dtx files, and for some reason VimTeX does not activate expl3 syntax matching in those files.

Interesting. This might be worth opening an issue over. Right now, the expl3 syntax is only activated inside e.g. \ExplSyntaxOn … \ExplSyntaxOff. There may be more conditions that should activate it. But I don't know these things, so if you open an issue, please also be willing to offer help.

Anyway, here is a piece of code using the @@ convention, taken from a package I am currently working on : …

Thanks!

and the same with the @@ expanded:

I assume this is manually expanded, right? IF not, then you must explain what you mean with "@@ expanded".

Better leave the names as they are.

Great - agreed then.

What I call texL3Statement in this syntax file is actually what LaTeX3 calls function, i.e. a macro that is intended to be called with appropriate arguments

Great - I'll adopt the "correct" conventions, then.

Similarly, texL3Type matches what LaTeX3 calls argument specifications, which is a concept similar to a type, since it dictates the calling convention for the function.

Then texL3Type seems a good name, thanks!

As for variables and constants, they are exactly what their names suggest: …

Thanks for the other clarifications as well!

I'll make some updates now. I hope it will improve things.

[lervag]

The point of the @= is to make sure that each @@ is followed by an underscore (which is part of the @@ convention) while avoiding a conflict with the _ after the group, which matches the required underscore before the final word in the variable name. Otherwise, something like \l@@_int would not match (although it does not perfectly follow the expl3 guidelines since it lacks a word).

Could you provide some examples - preferably both "positives" and "negatives"? Would be nice for testing and checking if things work as expected...

[lervag]

I believe the current version should be good now. But it may not work directly in .dtx files yet, so let's follow up that part either here or in a new issue.

[lervag]

The syntax groups are now texE3Function, texE3Variable, texE3Constant, texE3Type. The defaults are specified here:

vimtex/autoload/vimtex/syntax/core.vim

Lines 754 to 761 in 700c40d

	highlight def link texE3Cmd texCmd
	highlight def link texE3Delim texDelim
	highlight def link texE3Function texCmdType
	highlight def link texE3Opt texOpt
	highlight def link texE3Parm texParm
	highlight def link texE3Type texParm
	highlight def link texE3Variable texCmd
	highlight def link texE3Constant texE3Variable

[xunam]

and the same with the @@ expanded:

I assume this is manually expanded, right? IF not, then you must explain what you mean with "@@ expanded".

No, it is expanded by docstrip (see section 6 of the documentation). Essentially, each occurrence of @@ gets replaced by a module name with two underscores before, as per the convention for internal names in LaTeX3.

Could you provide some examples - preferably both "positives" and "negatives"? Would be nice for testing and checking if things work as expected...

The cases that strictly follow the convention, and where it must work, are like

\@@_function:nn
\l_@@_variable_tl

Accepting any control sequence name where any occurrence of @@ is surrounded by underscores is a natural extension of the above that my code tries to capture.

I can't think of any obvious negative case, where one would not want to match a similar-looking word. The @ character is not used in LaTeX3 (except for this @@ notation in source code) but is still categorized as letter as far as I can tell, so it should be matched as such in control sequences.

[lervag]

Thanks for the explanations!