Add a match-iterator API

[NWilson]

The largest chunk of detailed code inside pcre2demo.c involves explaining how to iterate over matches. This is non-trivial. The caller needs to try again at the end of the previous match location (in case \G does the bumpalong), and also try again (with "not-empty-at-start"), and if that doesn't match, then do a manual bump-along to search starting from the next character (but also skip \r\n as one sequence, if appropriate).

This logic is complex, and it's repeated in pcre2_substitute.

It may even be wrong, because the \G handling differs from Perl. In Perl, \G always means "end of previous match", but when there's a manual bump-along with repeated matches, PCRE2 gives it the behaviour of "start of current match process". We should at least offer an API to set a field on the match context, so that callers can specify a location for \G which is distinct from the start-offset used to begin the search for a match.

Let's offer a new API, "pcre2_match_iterator_create" which returns an object that handles this state. Then users can just call "pcre2_match_iterator_next" to go through all the matches. We'd use it ourselves in pcre2_substitute, but we'd also be able to make the demo code simpler and shorter.

[NWilson]

I've struggled to understand the motivation behind the current code.

Here's how it works (although actually differently in pcre2demo and pcre2_substitute):

• I will ignore the handling of\K in lookaround. That's another story!
• After a nonempty match, the next match attempt is simply at ovector[0].
• After empty, the code tries again for a different match at the same point, no additional flags (detail: this is actually skipped when the match occurred at startchar). This could yield an exact repeat of the match, in which case it is discarded and we move on to the next strategy
• Then, a match attempt is made again at the same point, but this time with noempty-at-start and anchored flags
• Then, if that fails, the search finally does a manual bumpalong.

It does all work (except the extremely minor incompatibility of \G with Perl after the bumpalong).

However, it seems greatly overcomplicated.

Here's what it should be:

• Search for the next match starting from ovector[1]
• If it was an empty match, use the noempty-at-start flag (only)

I have implemented this, and every pcre2_substitute test passes without change.

I have even convinced myself it's correct by logic, but I am suspicious of myself.

@PhilipHazel Would you be able to recall or comment on why my suggested logic wouldn't work?

[PhilipHazel]

I can only remember that this was tricky to get right when first encountered, and that was in pcretest (now pcre2test) with the /g option. The code in pcre2demo and in pcre2_substitute() ought to implement the same logic. There's a whole pile of comment in pcre2test.c that tries to explain it. It isn't (to my recollection) exactly what you have described above. When just doing a /g set of matches (i.e. not substituting), if any match is the null string, try again at the same point with non-empty-at-start + anchored and do a bumpalong if that fails. If it doesn't fail, then by definition there is new endpoint.

[NWilson]

Thank you @PhilipHazel. I have worked through the pcre2test code as well, and that also seems to work identically with the simplification that I am proposing.

My best guess is that the code just evolved, and as it was added-to over time, people didn't notice that the final set of logic could be squashed down into a simpler thing. It's always easier to extend logic by adding additional cases!

I'll think it over a little more, then open a PR.