Syntax highlighting for Vim script comments is broken

[chdiza]

Steps to reproduce

Make a buffer with &ft=vim and syntax highlighting on. Make a comment line that also has an additional double-quote in it later, e.g.,:

" This is what we call " blah

Expected behaviour

That entire line should be colored as a comment. But only the part from the second quote onward is colored that way. It looks like "This is what we call has the highlight group "vimIsCommand".

This is a bad regression; Vim didn't formerly do this. It's common to have lines that have a trailing comment, and then to comment out those very lines later. In my colorschemes, this makes the "This is what we call part colored the same as the "Normal" group; it defeats the purpose of syntax highlighting if one can't tell comments from non-comments by color.

Bisection reveals that the first bad commit is the following recent runtime file update:

f269eabc6c4f5bdcef989cd5b4b95ba8ccaa4d8a is the first bad commit
commit f269eabc6c4f5bdcef989cd5b4b95ba8ccaa4d8a
Author: Bram Moolenaar <Bram@vim.org>
Date:   Mon Oct 3 18:04:35 2022 +0100

    Update runtime files

Version of Vim

9.0.688

Environment

Ubuntu 20.04, though it also happens on macOS.
Terminal is kitty, or uxterm, doesn't matter
Also happens in gVim.

Logs and stack traces

No response

[brammool]

### Steps to reproduce Make a buffer with &ft=vim and syntax highlighting on. Make a comment line that also has an additional double-quote in it later, e.g.,: ```vim " This is what we call " blah ``` ### Expected behaviour That entire line should be colored as a comment. But only the part from the second quote onward is colored that way. It looks like `"This is what we call ` has the highlight group "vimIsCommand". This is a bad regression; Vim didn't formerly do this. It's common to have lines that have a trailing comment, and then to comment out those very lines later. In my colorschemes, this makes the `"This is what we call ` part colored the same as the "Normal" group; it defeats the purpose of syntax highlighting if one can't tell comments from non-comments by color.

The problem is that in Vim9 script a string in double quotes is looking just like a comment in legacy script. It's very hard to tell them apart. For the time being, an unmatched quote in a legacy comment is considered uncommon, therefore the highlighting as it is now. A proper solution would know whether the line is in legacy script or Vim9 script, which currently isn't done.

…

-- CONCORDE: Message for you, sir. He falls forward revealing the arrow with the note. "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[chdiza]

A proper solution would know whether the line is in legacy script or Vim9 script,

I don't understand. Surely vim is already aware that it's highlighting a legacy comment---otherwise it wouldn't make any of my example line colored like a comment, because there are no '#' characters in it.

Plus, in the commit mentioned above, the line of vim.vim that got changed and broke things has got nothing to do with vim9script, AFAICT.

Paging Dr. @cecamp.

[Shane-XB-Qian]

i also thought for now such highlight for comment (for such case) is bad, which caused hard to find it was a comment.

[Shane-XB-Qian]

vim9 with # looks ok vs but legacy with " behavior different now at least.
// it thought whole line is a comment when vim9 with #.

[arp242]

A proper solution would know whether the line is in legacy script or Vim9 script,

I don't understand. Surely vim is already aware that it's highlighting a legacy comment---otherwise it wouldn't make any of my example line colored like a comment, because there are no '#' characters in it.

Plus, in the commit mentioned above, the line of vim.vim that got changed and broke things has got nothing to do with vim9script, AFAICT.

The issue is that this:

vim9script
var x = "hello"

is valid in Vim9Script, it assigns a string to the variable x, but in VimScript the " is interpreted as a comment.

Disambiguating between the two uses of " is pretty hard.

That patch changed vimLineComment from +^[ \t:]*".*$+ to +^[ \t:]*"\("[^"]*"\|[^"]\)*$+. I'm not sure on the context of that change (i.e. what it fixed), but it's probably something that got highlighted as a comment when it shouldn't have.

[chdiza]

In that example, Vim knows what to do because vim9script is present in that buffer. I don't use vim9script, so no such line is in any of my buffers, so Vim should know that it's dealing with legacy script.

[arp242]

In that example, Vim knows what to do because vim9script is present in that buffer. I don't use vim9script, so no such line is in any of my buffers, so Vim should know that it's dealing with legacy script.

You can use fun in vim9script and def in VimScript. You can mix and match the two.

Clearly things could be improved, but it's not that simple.

[brammool]

> In that example, Vim knows what to do because `vim9script` is > present in that buffer. I don't use vim9script, so no such line is > in any of my buffers, so Vim should know that it's dealing with > legacy script. You can use `fun` in vim9script and `def` in VimScript. You can mix and match the two. Clearly things could be improved, but it's not that simple.

Indeed. I believe @lacygoill has an alternate syntax for Vim, you can give it a try. The specific change is a quick fix to tackle: var foo = "text" This was highlighting "text" as a comment, which also caused trouble for indenting. It's not perfect, but should work in most common situations.

…

-- ARTHUR: Well, I can't just call you `Man'. DENNIS: Well, you could say `Dennis'. ARTHUR: Well, I didn't know you were called `Dennis.' DENNIS: Well, you didn't bother to find out, did you? The Quest for the Holy Grail (Monty Python) /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[chdiza]

You can use fun in vim9script and def in VimScript. You can mix and match the two.

But you can't mix and match using the double-quote for comments. So what's wrong with making the vim syntax file work like this: if there is already a line that says "vim9script", then handle the quotes one way; otherwise, handle them the normal legacy way?

[dkearns]

Legacy comments are required in legacy functions in a Vim9 script file.

Ignoring the wider fix, there's already a shared FuncBody syntax region for function blocks. It probably just needs to be split into two and each limited to including the appropriate comment type.

[dkearns]

Here's a very quick attempt at fixing this. The general approach is that only the correct comments for the script version are allowed at TOP level and the rest are contained. This means that some inappropriate comments will still be highlighted but they would generally be invalid syntax.

This certainly needs more work and the comment highlighting could be simplified generally. E.g., vim9Comment appears to be unnecessarily contained in several syntax groups where vimComment was only included to prevent strings from being erroneously highlighted.

diff --git a/runtime/syntax/vim.vim b/runtime/syntax/vim.vim
index 839ae163f..3f4a351b6 100644
--- a/runtime/syntax/vim.vim
+++ b/runtime/syntax/vim.vim
@@ -243,14 +243,16 @@ endif
 " Functions : Tag is provided for those who wish to highlight tagged functions {{{2
 " =========
 syn cluster    vimFuncList     contains=vimCommand,vimFunctionError,vimFuncKey,Tag,vimFuncSID
-syn cluster    vimFuncBodyList contains=vimAbb,vimAddress,vimAugroupKey,vimAutoCmd,vimCmplxRepeat,vimComment,vim9Comment,vimContinue,vimCtrlChar,vimEcho,vimEchoHL,vimEnvvar,vimExecute,vimIsCommand,vimFBVar,vimFunc,vimFunction,vimFuncVar,vimGlobal,vimHighlight,vimIsCommand,vimLet,vimLetHereDoc,vimLineComment,vimMap,vimMark,vimNorm,vimNotation,vimNotFunc,vimNumber,vimOper,vimOperParen,vimRegion,vimRegister,vimSearch,vimSet,vimSpecFile,vimString,vimSubst,vimSynLine,vimUnmap,vimUserCommand
-syn match      vimFunction     "\<\(fu\%[nction]\)!\=\s\+\%(<[sS][iI][dD]>\|[sSgGbBwWtTlL]:\)\=\%(\i\|[#.]\|{.\{-1,}}\)*\ze\s*("       contains=@vimFuncList nextgroup=vimFuncBody
-syn match      vimFunction     "\<def!\=\ze\s*(" contains=@vimFuncList nextgroup=vimFuncBody
+syn cluster    vimFuncBodyList contains=vimAbb,vimAddress,vimAugroupKey,vimAutoCmd,vimCmplxRepeat,vimContinue,vimCtrlChar,vimEcho,vimEchoHL,vimEnvvar,vimExecute,vimIsCommand,vimFBVar,vimFunc,vimFunction,vimFuncVar,vimGlobal,vimHighlight,vimIsCommand,vimLet,vimLetHereDoc,vimMap,vimMark,vimNorm,vimNotation,vimNotFunc,vimNumber,vimOper,vimOperParen,vimRegion,vimRegister,vimSearch,vimSet,vimSpecFile,vimString,vimSubst,vimSynLine,vimUnmap,vimUserCommand
+syn match      vimFunction     "\<\(fu\%[nction]\)!\=\s\+\%(<[sS][iI][dD]>\|[sg]:\)\=\%(\i\|[#.]\|{.\{-1,}}\)\+\ze\s*("        contains=@vimFuncList nextgroup=vimFuncBody
+syn match      vimFunction                  "\<def!\=\s\+\%(<[sS][iI][dD]>\|[sg]:\)\=\%(\i\|[#.]\|{.\{-1,}}\)\+\ze\s*("        contains=@vimFuncList nextgroup=vim9FuncBody

 if exists("g:vimsyn_folding") && g:vimsyn_folding =~# 'f'
- syn region    vimFuncBody  contained  fold start="\ze\s*("    matchgroup=vimCommand end="\<\(endf\>\|endfu\%[nction]\>\|enddef\>\)"           contains=@vimFuncBodyList
+ syn region    vimFuncBody  contained  fold start="\ze\s*("    matchgroup=vimCommand end="\<endfu\%[nction]\>" contains=@vimFuncBodyList,vimComment,vimLineComment
+ syn region    vim9FuncBody contained  fold start="\ze\s*("    matchgroup=vimCommand end="\<enddef\>"  contains=@vimFuncBodyList,vim9Comment,vim9LineComment
 else
- syn region    vimFuncBody  contained  start="\ze\s*("         matchgroup=vimCommand end="\<\(endf\>\|endfu\%[nction]\>\|enddef\>\)"           contains=@vimFuncBodyList
+ syn region    vimFuncBody  contained  start="\ze\s*("         matchgroup=vimCommand end="\<endfu\%[nction]\>" contains=@vimFuncBodyList,vimComment,vimLineComment
+ syn region    vim9FuncBody contained  start="\ze\s*("         matchgroup=vimCommand end="\<enddef\>"  contains=@vimFuncBodyList,vim9Comment,vim9LineComment
 endif
 syn match      vimFuncVar   contained  "a:\(\K\k*\|\d\+\)"
 syn match      vimFuncSID   contained  "\c<sid>\|\<s:"
@@ -301,18 +303,34 @@ syn match vimUserAttrbCmplt contained     "custom,\u\w*"

 " Lower Priority Comments: after some vim commands... {{{2
 " =======================
-syn match      vimComment      excludenl +\s"[^\-:.%#=*].*$+lc=1       contains=@vimCommentGroup,vimCommentString
-syn match      vimComment      +\<endif\s\+".*$+lc=5   contains=@vimCommentGroup,vimCommentString
-syn match      vimComment      +\<else\s\+".*$+lc=4    contains=@vimCommentGroup,vimCommentString
-syn region     vimCommentString        contained oneline start='\S\s\+"'ms=e   end='"'
-" Vim9 comments - TODO: might be highlighted while they don't work
-syn match      vim9Comment     excludenl +\s#[^{].*$+lc=1      contains=@vimCommentGroup,vimCommentString
-syn match      vim9Comment     +\<endif\s\+#[^{].*$+lc=5       contains=@vimCommentGroup,vimCommentString
-syn match      vim9Comment     +\<else\s\+#[^{].*$+lc=4        contains=@vimCommentGroup,vimCommentString
-" Vim9 comment inside expression
-syn match      vim9Comment     +\s\zs#[^{].*$+ms=s+1   contains=@vimCommentGroup,vimCommentString
-syn match      vim9Comment     +^\s*#[^{].*$+  contains=@vimCommentGroup,vimCommentString
-syn match      vim9Comment     +^\s*#$+        contains=@vimCommentGroup,vimCommentString
+let s:isVim9Script = "\n" .. getline(1, 10)->join("\n") =~# '\n\s*vim9\%[script]\>'
+if s:isVim9Script
+  syn match    vimComment      excludenl +\s"[^\-:.%#=*].*$+lc=1       contains=@vimCommentGroup,vimCommentString contained
+  syn match    vimComment      +\<endif\s\+".*$+lc=5   contains=@vimCommentGroup,vimCommentString contained
+  syn match    vimComment      +\<else\s\+".*$+lc=4    contains=@vimCommentGroup,vimCommentString contained
+  syn region   vimCommentString        contained oneline start='\S\s\+"'ms=e   end='"' contained
+  " Vim9 comments - TODO: might be highlighted while they don't work
+  syn match    vim9Comment     excludenl +\s#[^{].*$+lc=1      contains=@vimCommentGroup,vimCommentString
+  syn match    vim9Comment     +\<endif\s\+#[^{].*$+lc=5       contains=@vimCommentGroup,vimCommentString
+  syn match    vim9Comment     +\<else\s\+#[^{].*$+lc=4        contains=@vimCommentGroup,vimCommentString
+  " Vim9 comment inside expression
+  syn match    vim9Comment     +\s\zs#[^{].*$+ms=s+1   contains=@vimCommentGroup,vimCommentString
+  syn match    vim9Comment     +^\s*#[^{].*$+          contains=@vimCommentGroup,vimCommentString
+  syn match    vim9Comment     +^\s*#$+                contains=@vimCommentGroup,vimCommentString
+else
+  syn match    vimComment      excludenl +\s"[^\-:.%#=*].*$+lc=1       contains=@vimCommentGroup,vimCommentString
+  syn match    vimComment      +\<endif\s\+".*$+lc=5   contains=@vimCommentGroup,vimCommentString
+  syn match    vimComment      +\<else\s\+".*$+lc=4    contains=@vimCommentGroup,vimCommentString
+  syn region   vimCommentString        contained oneline start='\S\s\+"'ms=e   end='"'
+  " Vim9 comments - TODO: might be highlighted while they don't work
+  syn match    vim9Comment     excludenl +\s#[^{].*$+lc=1      contains=@vimCommentGroup,vimCommentString contained
+  syn match    vim9Comment     +\<endif\s\+#[^{].*$+lc=5       contains=@vimCommentGroup,vimCommentString contained
+  syn match    vim9Comment     +\<else\s\+#[^{].*$+lc=4        contains=@vimCommentGroup,vimCommentString contained
+  " Vim9 comment inside expression
+  syn match    vim9Comment     +\s\zs#[^{].*$+ms=s+1   contains=@vimCommentGroup,vimCommentString contained
+  syn match    vim9Comment     +^\s*#[^{].*$+          contains=@vimCommentGroup,vimCommentString contained
+  syn match    vim9Comment     +^\s*#$+                contains=@vimCommentGroup,vimCommentString contained
+endif

 " Environment Variables: {{{2
 " =====================
@@ -649,8 +667,13 @@ syn match  vimCtrlChar     "[^A-^H^K^O-^_]"

 " Beginners - Patterns that involve ^ {{{2
 " =========
-syn match      vimLineComment  +^[ \t:]*"\("[^"]*"\|[^"]\)*$+  contains=@vimCommentGroup,vimCommentString,vimCommentTitle
-syn match      vim9LineComment +^[ \t:]\+#.*$+ contains=@vimCommentGroup,vimCommentString,vimCommentTitle
+if s:isVim9Script
+  syn match    vimLineComment  +^[ \t:]*".*$+  contains=@vimCommentGroup,vimCommentString,vimCommentTitle contained
+  syn match    vim9LineComment +^[ \t:]\+#.*$+ contains=@vimCommentGroup,vimCommentString,vimCommentTitle
+else
+  syn match    vimLineComment  +^[ \t:]*".*$+  contains=@vimCommentGroup,vimCommentString,vimCommentTitle
+  syn match    vim9LineComment +^[ \t:]\+#.*$+ contains=@vimCommentGroup,vimCommentString,vimCommentTitle contained
+endif
 syn match      vimCommentTitle '"\s*\%([sS]:\|\h\w*#\)\=\u\w*\(\s\+\u\w*\)*:'hs=s+1    contained contains=vimCommentTitleLeader,vimTodo,@vimCommentGroup
 syn match      vimContinue     "^\s*\\"
 syn region     vimString       start="^\s*\\\z(['"]\)" skip='\\\\\|\\\z1' end="\z1" oneline keepend contains=@vimStringGroup,vimContinue

[Aster89]

I wonder if it is related to #11458.

[lukelbd]

For anyone coming across this thread looking for a quick drop-in .vimrc fix (as I was), and who hasn't transitioned to vim9 script yet, the following seems to restore original syntax highlighting behavior:

function! s:syntax_overrides() abort
  syntax match customComment /^[ \t:]*".*$/  contains=vimCommentTitle,vimCommentString,@vimCommentGroup containedin=.*\(Body\|\)
  highlight link customComment vimComment
endfunction

augroup syntax_overrides
  au!
  au BufEnter .vimrc,*.vim call s:syntax_overrides()
augroup END

[dkearns]

Closing as I believe everything discussed in this issue is now highlighted properly.