Alternate file in vimscript occasionally breaks syntax highlighting

[acc987]

Steps to reproduce

1. Open $VIMRUNTIME/**/defaults.vim and navigate to where the DiffOrig command is defined.
2. In r ++edit #, the alternate file # and the trailing text is highlighted as a comment, see attached screenshot below.

Expected behaviour

I expect the use of # not to break syntax highlighting in this case.

I'm assuming that highlighting breaks because comments=sO:# -,mO:# ,eO:##,:#,sO:" -,mO:" ,eO:"",:" is set by the vim.vim ftplugin, most likely due to Vim9Script, but that does not necessarily have to be the root cause as syntax highlighting still works correctly in, for example, nnoremap <silent> ,b :buffer! #<CR>. So I guess this is more of a syntax parsing issue than an issue with the way 'comments' is set. I have not been able to make any progress beyond that and would appreciate it, if someone familiar with the code base could look into this.

Version of Vim

9.0.1632

Environment

Operating system: openSUSE Leap

Logs and stack traces

No response

[chrisbra]

I suppose it comes from here:

vim/runtime/syntax/vim.vim

Line 314 in 11d2aee

syn match vim9Comment +\s\zs#[^{].*$+ms=s+1 contains=@vimCommentGroup,vimCommentString

But it looks ambiguous, not sure there is a way to detect that # is not meant as a vim comment here.

[lacygoill]

If # is followed by a bar, I would argue it's more likely to be the alternate file rather than the start of an inline Vim9 comment:

diff --git a/runtime/syntax/vim.vim b/runtime/syntax/vim.vim
index d1ce01ec8..0172d7c16 100644
--- a/runtime/syntax/vim.vim
+++ b/runtime/syntax/vim.vim
@@ -307,11 +307,11 @@ syn match	vimComment	+\<endif\s\+".*$+lc=5	contains=@vimCommentGroup,vimCommentS
 syn match	vimComment	+\<else\s\+".*$+lc=4	contains=@vimCommentGroup,vimCommentString
 syn region	vimCommentString	contained oneline start='\S\s\+"'ms=e	end='"'
 " Vim9 comments - TODO: might be highlighted while they don't work
-syn match	vim9Comment	excludenl +\s#[^{].*$+lc=1	contains=@vimCommentGroup,vimCommentString
+syn match	vim9Comment	excludenl +\s#\%(\s*|\)\@![^{].*$+lc=1	contains=@vimCommentGroup,vimCommentString
 syn match	vim9Comment	+\<endif\s\+#[^{].*$+lc=5	contains=@vimCommentGroup,vimCommentString
 syn match	vim9Comment	+\<else\s\+#[^{].*$+lc=4	contains=@vimCommentGroup,vimCommentString
 " Vim9 comment inside expression
-syn match	vim9Comment	+\s\zs#[^{].*$+ms=s+1	contains=@vimCommentGroup,vimCommentString
+syn match	vim9Comment	+\s\zs#\%(\s*|\)\@![^{].*$+ms=s+1	contains=@vimCommentGroup,vimCommentString
 syn match	vim9Comment	+^\s*#[^{].*$+	contains=@vimCommentGroup,vimCommentString
 syn match	vim9Comment	+^\s*#$+	contains=@vimCommentGroup,vimCommentString
 

[ychin]

I think this is a deeper issue than that. Vim's syntax highlighting for .vim files does not know how to distinguish normal Vimscript vs vim9script. This has caused misc issues already regarding how it tries to highlight comments because one language uses " and the other one uses # (e.g. see #11307 and #11560). I think we should try to think more carefully about how to move forward instead of hacking in solutions for this (e.g. in vim9script you have to use %% for alt file, which tbh I find it weird, but that's the syntax, and so you would not have this problem).

Some additional links for discussions around legacy Python2 vs Python3 syntax files (#12066 and #8033). The situation is a little different because Python 2 is pretty much deprecated at this point, but there are still lots of Vimscript going around.

There are different options we could go for (e.g. using a separate .vim9 file extension, using more complicated parsing to detect vim9script, using… tree-sitter :P, etc), but I just think we shouldn't just hack in a solution because it may break in other places.

[chrisbra]

Well, vim9script was specifically designed to allow it to live in the same file as legacy vimscript. I guess the issue here is therefore somehow expected and can only worked around with just a mentioned hack as we will have to live with a certain ambiguity.

Perhaps another solution would be to have a configuration variable (off by default), that allows to disable vim9 specific syntax elements in legacy vimscript. But that would have to be configured by the user himself beforehand. That would of course be only useful, if one does not intend to use vim9script together with legacy vimscript. So this may also qualify as a crude hack.

[ychin]

Yeah having a manually toggled option does seem kind of annoying esp with mixed scripts versions (which I have seen a fair amount). I guess I'm not really opposed to getting this in first to fix the issue but I just feel like it will be whack-a-mole in terms of chasing down weird syntax highlighting oddities.

I wonder if the syntax rules detect if it's within a "def" block, or detect "vim9script" and only do Vim 9 stuff there? Those are the only two conditions for entering vim9script mode right?

[dkearns]

I did some work, more generally, in handling Vim9 vs legacy script in the Vim filetype support files.

Most of it was aborted because Bram was planning to do some work before the next release on generating a proper in memory parse tree which would render much it moot. I'll look at resurrecting it some time in the near future.

However, this particular problem would appear to be trivial. vim9Comment shouldn't be defined at TOP level in a legacy script only in compiled functions and autoload/command blocks.

I've fixed this before, somewhere, but it needs a bit of buy-in from the maintainer. I'd prefer not to be randomly patching that file without approval.

I'll try and output something by next week. Ping me if I don't.

@chrisbra you can assign me if we're doing that sort of thing now, in the brave new world. :)

[chrisbra]

Thanks, but I don't think I need to assign anybody to anything :) But I agree, would be good to hear back from Charles @cecamp (hopefully)