Using \zs and \ze invalidates syntax match in some cases

[hakbra]

Stemming from this thread on reddit

---

Issue

the intention is to highlight 'foo' in the context below

self.foo

The entire expression can be matched using

syntax match Test /\vself\.\w+/    " (1)

And only the desired part can be highlighted using

syntax match Test /\vself\.\zs\w+\ze/  " (2)

However, in my case the markdown syntax was already loaded and (2) did no longer work despite that the very similar (1) was still working. I have narrowed it down to an expression similar to this:

syntax match LineStart "^" (3)

I would expect (1) and (2) to match the same every time and only differ in their highlighting, but (2) does not highlight anything when (3) is present

Reproduction

Data in file:

self.foo

Running

syntax clear
syntax match LineStart "^"
syntax match Test /\vself\.\w+/

hi link Test Folded

Result: self.foo
Expected: self.foo

---

Running

syntax clear
syntax match LineStart "^"
syntax match Test /\vself\.\zs\w+\ze/

hi link Test Folded

Result: self.foo
Expected: self.foo

---

Running

syntax clear
syntax match Test /\vself\.\zs\w+\ze/

hi link Test Folded

Result: self.foo
Expected: self.foo

Environment

All tests ran with

vim -N -u NONE

Output of vim --version

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Oct 21 2015 13:04:09)
Included patches: 1-712
Modified by pkg-vim-maintainers@lists.alioth.debian.org
Compiled by buildd@
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +syntax
+arabic          +file_in_path    +mouse_sgr       +tag_binary
+autocmd         +find_in_path    -mouse_sysmouse  +tag_old_static
-balloon_eval    +float           +mouse_urxvt     -tag_any_white
-browse          +folding         +mouse_xterm     -tcl
++builtin_terms  -footer          +multi_byte      +terminfo
+byte_offset     +fork()          +multi_lang      +termresponse
+cindent         +gettext         -mzscheme        +textobjects
-clientserver    -hangul_input    +netbeans_intg   +title
-clipboard       +iconv           +path_extra      -toolbar
+cmdline_compl   +insert_expand   -perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con      -lua             +rightleft       +windows
+diff            +menu            -ruby            +writebackup
+digraphs        +mksession       +scrollbind      -X11
-dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     -xim
+emacs_tags      -mouseshape      -sniff           -xsmp
+eval            +mouse_dec       +startuptime     -xterm_clipboard
+ex_extra        +mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    -xpm
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H     -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl    -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions  

[ZyX-I]

LineStart consumes s from self, so it can no longer be matched when at the start of the line. You can have zero-width items in pattern, but syntax rules can’t be zero-width.

Try using positive look-behinds instead: \v%(self\.)@5<=\w+: here match starts at f after dot and not s, so it is not affected by the fact that s was consumed by LineStart.

Note the difference between \zs/\ze and look-aheads/-behinds: \z* matches the whole pattern regardless of \z* and then part of the match is excluded from highlighting thus making pattern affected by a number of things including other matches in “non-matched” part. Look-aheads/-behinds match only the needed part from the very beginning, but are slower.

5 in the example is part of the optimization and makes engine examine only at most 5 preceding bytes to test whether \w+ is preceded by self\.. I hope some day this number will be determined automatically for such simple patterns.

[hakbra]

Thanks, that worked. I would have found using \zs and \ze to be be easier to understand though.