regexpengine=2 with whole pattern (\@>) and multi-line (\_.) atom fails to match when searching

[inkarkat]

Example

I've tried to match a BEGIN...END block only if there's no MIDDLE inside; like the second block in the following example:

BEGIN
    some
    MIDDLE
    stuff
END
BEGIN
    other
    stuff
END

I came up with the following regular expression, using the whole pattern multi (\@>) to avoid backtracking and a non-greedy multi-line atom (\_.\{-}) to match just a single block.

/\%(BEGIN\_.\{-}\%(MIDDLE\|END\)\)\@>\%(END\)\@<=/

In Vim version 8.1.553 (huge version with GTK2 GUI, on Ubuntu 16.04.5 LTS x64), this only matches with the old engine ('regexpengine' set to 1); it does not match with the new, NFA-based one. I think this is a bug; the behavior should not differ.

Investigations

The problem is not triggered when the entire match is on a single line. The following joins the text into one line, and this gives 1 as expected:

vim --clean \
    -c 'let @x = "BEGIN some MIDDLE stuff END BEGIN other stuff END "' \
    -c 'silent put! x' \
    -c 'echomsg search("\\%#=2\\%(BEGIN\\_.\\{-}\\%(MIDDLE\\|END\\)\\)\\@>\\%(END\\)\\@<=")'

The problem is only triggered when doing text searches (with /, or :echo search(), but not with =~ or matchstr(). The following gives old:1 new:1:

vim --clean \
    -c 'let @x = "BEGIN\nsome\nMIDDLE\nstuff\nEND\nBEGIN\nother\nstuff\nEND\n"' \
    -c 'echon " old:" @x =~ "\\%#=1\\%(BEGIN\\_.\\{-}\\%(MIDDLE\\|END\\)\\)\\@>\\%(END\\)\\@<="' \
    -c 'echon " new:" @x =~ "\\%#=2\\%(BEGIN\\_.\\{-}\\%(MIDDLE\\|END\\)\\)\\@>\\%(END\\)\\@<="'

How to reproduce

The following two invocations use the different engines; the first correctly gives 6, the second wrongly 0:

vim --clean \
    -c 'let @x = "BEGIN\nsome\nMIDDLE\nstuff\nEND\nBEGIN\nother\nstuff\nEND\n"' \
    -c 'silent put! x' \
    -c 'echomsg search("\\%#=1\\%(BEGIN\\_.\\{-}\\%(MIDDLE\\|END\\)\\)\\@>\\%(END\\)\\@<=")'
vim --clean \
    -c 'let @x = "BEGIN\nsome\nMIDDLE\nstuff\nEND\nBEGIN\nother\nstuff\nEND\n"' \
    -c 'silent put! x' \
    -c 'echomsg search("\\%#=2\\%(BEGIN\\_.\\{-}\\%(MIDDLE\\|END\\)\\)\\@>\\%(END\\)\\@<=")'

I tried to write an automated test for it, but I only found coverage of old-vs-new regexp engines in (old-style) test44.in / test99.in, and this multi-line example here doesn't seem to fit in with that very well. I guess you'll be able to quickly build a new-style test from the code above.