Add unicode handling to keyword match #52
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the keyword matching functionality to properly handle Unicode characters by replacing ASCII-only word boundaries with Unicode-aware regex patterns.
- Replaces
\bword boundaries with(?<!\w)and(?!\w)lookaround assertions for Unicode-aware matching - Adds explicit
re.UNICODEflag to the regex compilation - Includes comprehensive test coverage for various Unicode scenarios including Chinese characters, numeric tokens, and underscore handling
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/guardrails/checks/text/keywords.py | Updated regex pattern to use Unicode-aware word boundaries instead of ASCII \b, added re.UNICODE flag, and improved variable naming |
| tests/unit/checks/test_keywords.py | Added 8 new test cases covering partial word matching, numeric tokens, underscore handling, Chinese characters, and multi-token patterns with Unicode boundaries |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@codex pls review |
|
Codex Review: Didn't find any major issues. Swish! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…o dev/steven/unicode_keyword
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Updated
keyword_filterfor better unicode handlingThank you to @yehorkardash for identifying this in our JS version