\b not working with Unicode characters

[jonashaag]

Saw this using pyre2:

>>> import re, re2
>>> re.search(r"\bä", "aä")
>>> re2.search(r"\bä", "aä")
<re2.Match object; span=(1, 2), match='ä'>

Test case for re2 that fails:

diff --git a/re2/testing/re2_test.cc b/re2/testing/re2_test.cc
index b1f7d73..6818254 100644
--- a/re2/testing/re2_test.cc
+++ b/re2/testing/re2_test.cc
@@ -1387,6 +1387,12 @@ TEST(RE2, UnicodeClasses) {
   EXPECT_FALSE(RE2::FullMatch("A", "\\P{Lu}"));
   EXPECT_TRUE(RE2::FullMatch("A", "\\P{Ll}"));
 
+  EXPECT_FALSE(RE2::PartialMatch("xä", "(x\\b)", &a));
+  EXPECT_EQ("x", a);
+
+  EXPECT_FALSE(RE2::PartialMatch("xä", "(\\bä)", &a));
+  EXPECT_EQ("ä", a);
+
   EXPECT_TRUE(RE2::FullMatch("譚", "\\p{L}"));
   EXPECT_FALSE(RE2::FullMatch("譚", "\\p{Lu}"));
   EXPECT_FALSE(RE2::FullMatch("譚", "\\p{Ll}"));

Update: Added test case other way round (Unicode in needle vs haystack).

[junyer]

Sorry, RE2 supports \b, \d, \s, \w and their counterparts for ASCII only.

It's possible to simulate Unicode word boundaries to some extent. For example, if you can approximate \b as [\w\pL\pN] on one side and (?:[\W\PL\PN]|\A|\z) on the other, then your test can be rewritten as re2.search(r"(?:[\W\PL\PN]|\A|\z)ä", "aä"). That pollutes the match ($0), of course, but a capturing group can obtain the equivalent result in the submatch ($1).

[mbasmanova]

We ran into this problem in Velox when matching German strings:

SELECT REGEXP_LIKE('Insidern auch als Grenzenüberschreiter bekannt', '(?i)(\b)Grenzen(\b)')

This query returns 'true' while we (and our users) expect 'false'.

Is there any workaround?

CC: @zacw7

[junyer]

In this case, you might consider using the Rust regex crate, which does support Unicode-aware \b. (There's a C wrapper.)

[mbasmanova]

In this case, you might consider using the Rust regex crate, which does support Unicode-aware \b. (There's a C wrapper.)

@junyer Paul, thank you for the tip. Where can I learn more about "Rust regex crate"?

[BurntSushi]

https://github.com/rust-lang/regex