[Show & Tell] Native Regex Unicode Escape Support in Gluten (Velox & Bolt)

[KushShriv]

[Show & Tell] Native Regex Unicode Escape Support in Gluten (Velox & Bolt)

Hey everyone!

I just opened a pull request to solve a long-standing pain point with regular expressions in Gluten: handling Java-style Unicode escapes (\uXXXX and surrogate pairs) on RE2-backed execution engines like Velox and Bolt.

Check out the PR here: #13278

1. The Problem

Previously, when a Spark query included regex patterns containing Java-style Unicode escapes (such as \u4e00-\u9fa5 for CJK characters or supplementary characters written as surrogate pairs like \uD83D\uDE00), RE2 couldn't parse them. This caused compilation failures on the native side, forcing the entire operator to fall back to Spark (tracked in FallbackSuite).

2. The Solution: Two-Layer Defense

To fix this cleanly without breaking ClickHouse or introducing runtime crashes, we implemented a two-pronged approach:

1. Scala-Side Translation (ExpressionConverter):
   We added an escape-aware scanner (Re2PatternUtils.translateJavaPatternToRe2) that rewrites literal patterns into RE2-compatible syntax right before they are written into the Substrait plan. It covers RLike, RegExpReplace, RegExpExtract, RegExpExtractAll, RegExpInstr, and StringSplit.

   • Lookarounds, backreferences (\1–\9), and unpaired surrogates safely fall back to Spark (GlutenNotSupportException).
   • Backend gating ensures this only runs when needsRe2PatternTranslation = true (overridden for Velox and Bolt; ClickHouse is untouched, and Bolt relies purely on the Scala rewrite).

2. C++ Validator as a Strict Safeguard (Velox):
   validatePattern() compiles patterns with RE2 exactly as given. If a raw \u ever slips past (e.g., from a plan not built by Gluten's converter or a new expression added without Scala wiring), it is rejected at validation time rather than failing mid-execution with a cryptic VeloxUserError. The operator simply falls back to Spark gracefully.

3. Test Results & Validation

We ran an end-to-end comparison on a 6,000-row table against vanilla Spark and an independent Java regex oracle across 10 different query shapes (rlike, regexp_replace, split, surrogate pairs, etc.):

• Main branch: Translated shapes fell back to Spark.
• This branch: All query shapes executed natively in Velox with 15/15 checks passing matching Spark results!

Suite	Result
common_test (C++)	21/21
Re2PatternUtilsSuite	8/8
VeloxStringFunctionsSuite	50/50
FallbackSuite	16 passed, 0 failed, 1 ignored (fallback final aggregate of collect_set, unrelated)

Style checks (scalastyle, checkstyle, and spotless) are fully passing for both backends-velox and gluten-substrait.

4. What do you think?

Would love to hear your thoughts or feedback on this architecture, especially if you've run into similar regex hurdles with RE2 translation in other connectors! Feel free to drop comments or questions below.

[zhouyuan]

Cc: @philo-he @baibaichen

[philo-he]

@KushShriv, thanks for bringing up this discussion.

The proposed translation helps preserve Spark’s regex semantics during native execution. This raises a broader question: should we also let users explicitly choose native-engine semantics where they are willing to accept differences from Spark?

For example, we could consider per-feature settings, or a broader “native semantics mode,” that allows RE2 semantics and skips the corresponding compatibility translation and fallback checks.

This could extend beyond regex to other areas whose behavior depends on underlying libraries, such as JSON parsing. Some users may prefer a documented native behavior contract over strict Spark compatibility, particularly where compatibility adds substantial complexity or limits native execution.

[KushShriv]

Hey @philo-he, thanks for the thoughtful suggestion, I think it's a good direction.

For this PR. I'd lean towards keeping it as is. The \uXXXX -> \x{XXXX} rewrite means the same thing in both engines, so it only removes a spurious fallback and doesn't seem to need a switch. The checks for lookaround, backreferences, and unpaired surrogates also can't be skipped by a mode, as far as I can tell, because RE2 can't compile those constructs. Skipping them would turn a fallback into a runtime failure.

A native-semantics option feels like it would matter where both engines accept a pattern but behave differently, and I haven't audited those cases yet. One I ran into, if I understand it correctly, is [\12], which Java rejects but RE2 reads as an octal escape.

I'd be glad to help if you think it's worth pursuing, would you prefer to discuss the design in an issue first, or do you have a different idea of how to scope it? I'm happy to start by writing up the Java/RE2 regex differences if that would be useful.

[philo-he]

@KushShriv, thanks for the discussion. Agreed. Keeping this PR as-is is the right call. Compatibility is what many users care about, and the \uXXXX -> \x{XXXX} rewrite only removes a fallback without changing semantics, so it doesn't need a switch.

The native-semantics mode is worth keeping in mind for separate, future planning. Before fixing semantic issues, we'd want to understand whether each behavior is a deliberate Spark design choice or just an arbitrary one, so we can decide which mismatches are actually worth addressing, especially since some would be hard to fix, requiring intrusive changes to third-party libraries.

The Java/RE2 differences write-up would be very useful. Please go ahead when you get a chance.