Strange behavior with lookarounds at the begin or end of a regexp

[Ferry612]

I’ve observed a – in my eyes – strange behavior of PCRE when using lookarounds at the begin or end of a regular expression.

See this example:

/(?<=\d\d).+/ =~ ’42 ist the answer.‘

This yields as match result: „ ist the answer.“ (with a leading blank).

Until now, I thought that lookarounds do not capture anything, but in this case it does – ‘42’ is captured by the lookbehind and does not appear as match result although it is part of the match of „.+“.

The same also occurs with a lookahead at the end:

/.+(?=\d\d)/ =~ „The answer is 42.“

yields „The answer is „ (with a trailing blank).

The digits and the dot are captured by the lookahead and are not returned in the match of „.+“.

Is this a bug? Why PCRE behaves like this ?

Kind greetings from Vienna,

Ferry

[NWilson]

You were correct that lookbehinds do not capture anything.

In your first example: /(?<=\d\d).+/ =~ '42 ist the answer.' we have 42 which is matched by the lookbehind (but not "captured"). It does not appear as a match result, because it is not part of the match of .+. The regex says, .+ captures as much as possible of the input string; however, the start of the match must satisfy (?<=\d\d), that is, the start of the match must be anchored at the first position just after two digits. So PCRE2 attempts a match at '4' (offset 0 in the string); the assertion fails, so it tries at offset 1; assertion fails again; so it tries at offset 2 (just between '2' and the space). The assertion matches, so .+ then consumes the rest of the string.

[Ferry612]

Hi NWilson, many thanks for your answer. But things are still not clear to me.

Until now, I thought that if the condition of a lookaround construct can be satisfied, the regex engine processes as if the lookaround construct didn't exist at all. In other words, after processing the lookaround successfully, it continues to process in the search string at the same position as it did when starting process the lookaround. See this example:

/(?=\w{3})(x|y)+/ =~ /xyx/

This yield "xyx". The regex engine starts before the first "x". The lookahead requests 3 alphanumeric characters - this condition is satified because the search string is "xyx". Next, the regex engine looks at the same position (the beginning of the search string) for any number of "x" or "y". Because there is a string "xyx", this string is returned.

So in my mind, after processing the lookbehind construct (?<=\d\d) successfully (which was applied at the start of the "42 is the answer." string, the regex still should continue at the same position (the start of the search string), and because ".+" matches any number of any characters, the entire string should be returned.

Can you explain the different behavior with these regexes?

Greetings, Ferry

[NWilson]

Until now, I thought that if the condition of a lookaround construct can be satisfied, the regex engine processes as if the lookaround construct didn't exist at all. In other words, after processing the lookaround successfully, it continues to process in the search string at the same position as it did when starting process the lookaround. See this example:

That is correct.

So in my mind, after processing the lookbehind construct (?<=\d\d) successfully (which was applied at the start of the "42 is the answer." string, the regex still should continue at the same position (the start of the search string), and because ".+" matches any number of any characters, the entire string should be returned.

Ah - but the lookbehind doesn't match at the start. As I explained, it will attempt a match at the start, but at the start, there are not two digits behind the current position (there's nothing at all behind). So matching moves on to the next position, and the next, until you are just after the '42'. That's where the lookbehind succeeds, so .+ consumes from then on.

[Ferry612]

Sorry, I can't understand. I wrote "If the condition of a lookaround construct can be satisfied, the engine processes as if the lookaround construct didn't exist." and you confirmed this.

So, in the regex

/(?<=\d\d).+/

after processing the lookbehind successfully, the remaining rexeg is:

/.+/

and because the search in the search string starts at the same position and /.+/ matches every charachter, the result must be the entire search string, including the leading digits!

When starting the lookbehind, this occurs at the start of the search string because no character was processed until now. So the engine starts at the position before the first digit and I think it is THIS position which is must be stored for later processing.

The regex now has to advance two characters to satify the lookbehind. Well, at this moment, the position in the search string is before the blank. All right. But now, the lookbehind was satisfied and the engine has to restore the position in the search string to the one before the lookbehind started - the position before the first digit.

It doesn't matter that the engine had to advance two characters to satisfy the lookbehind because this was only made to satisfy it. Processing a lookaround only changes temporary the position in the search string - until the lookaround was satisfied (or fails).
Because no capturing of characters occured until now, all characters in the search string are still available for processing and therefore a /.+/ must include the leading digits as well.

A lookaround construct cannot change the resulting match - it can affect it because it doesn't match - but if it matches, the matching result must be the same. Otherwise, this isn't a real "lookaround" because characters were captured.

[PhilipHazel]

I'm afraid that's not the way it works. The pattern is not satisfied at position 0. It is only at position 2 that the lookbehind can be satisfied, so that's the position from which it satisfies the rest of the pattern. Note that Perl behaves in exactly the same way as, I'm sure, do other engines that support lookbehind. Your pattern could be described as "Find a position that has two digits immediately before it. Then (at that position) match as many characters as you can." The lookahead equivalent /(?=\d\d).+/ is "Find a position that has two digits after it, then match as many characters as possible", which, as you have noted, DOES include the digits.

[Ferry612]

And why with

/.+(?=\d\d)/ =~ /The answer is 42./

the trailing digits (and the dot) aren't returned? Now we're talking about lookahead, not lookbehind!

In any case, the statement "If a lookaround can be satisfied, the regex engine behaves in the same way as if the lookaround hadn't existed" is incorrect because the match results with and without lookaround are not always the same!

[PhilipHazel]

That quoted statement is not true. The whole point of a lookaround is to affect how the pattern is matched. There would be no point in having lookarounds if they didn't affect the match!

In the lookahead that you have just quoted, the engine at first swallows all the characters as part of the match. Then it finds it cannot satisfy the lookahead, so it backtracks by one character (removing it from the match), but it still can't satisfy the lookahead, so it backtracks by one more character. Still a lookahead failure, so a third backtrack. Then the lookahead is satisfied. There's no more pattern to match, so there it stops, with all of the subject except the final three characters as the matched string.

[Ferry612]

Well, I understand. In the last example, the lookahead will capture three characters, but this contradicts the statement that lookarounds don't capture any characters. Although they match, in this example, the last three characters aren't part of the result.

[PhilipHazel]

Aha! I see what your misunderstanding is. It's the word "capture". This is the word we use when we are talking about parentheses that capture some part of the subject and return its value to the caller. A lookahead does not "capture" in this sense. In the last example, the lookahead MATCHES the last three characters, but it does not capture them and return them to the caller as a substring (in Perl terms, $1, $2, etc). I hope this clarifies things.

[Ferry612]

Hm... I've tried this short program in Perl:

use warnings;
"The answer is 42." =~ /.+(?=\d\d)/;
print "$& = '$&'\n";
print "$1 = '$1'\n";

I got:

$& = 'The answer is '
Use of uninitialized value $1 in concatenation (.) or string at x.pl line 4.
$1 = ''

So it doesn't seem like the last tree characters are returned in $1, as you wrote...

[PhilipHazel]

I don't think I wrote that. I didn't mean to. I wrote "the lookahead MATCHES the last three characters, but it does not capture them and return them to the caller" -- note "does not capture them". Maybe I should have written NOT in caps. Perl is doing exactly what I would expect.

[PhilipHazel]

Try

"The answer is 42." =~ /.+(?=(\d\d))/;

Note the extra parentheses round \d\d. They specify capturing parentheses, and indeed they capture.

[Ferry612]

Sorry for the misunderstanding. Of course, if you add an additional pair of parentheses around \d\d, then its value will be available in $1.

What I still can't understand is what will happen with the digits. In an expression like

/.+(?=\d\d)/

I understand that /.+/ is greedy and will consume all characters in the search string. Next, because the regex isn't finished yet, the engine must backtrack to satisfy both digits of the lookaround. But what happenend with them thereafter? They aren't returned in the match result although /,+/ would match them. So it looks like the lookaround has "consumed" them which I didn't expect.

I think my problem is that I'm not sure what the "starting position" is in the search string after a lookaround has processed. I always thought that this is still the same position where the lookaround was applied. But I understand now that isn't always the case - at least not with a lookbehind at the start of a regex or a lookahead at its end.

[carenas]

What I still can't understand is what will happen with the digits. In an expression like

Lookarounds are a little confusing for sure, but I think they are clearer once you understand that the state machine moves forward in the string but the lookaround position doesn't change, because they are "assertions" and therefore don't describe anything in the match per se.

See the Regex Tutorial website for a non PCRE specific explanation of them.