buffs: add Turkish locale case mapping buffs - #1997
Open
fevziegeyurtsevenler wants to merge 1 commit into
Open
Conversation
Turkish keeps the dot on i as a distinguishing feature across case changes, so it has two i pairs (I/ı and İ/i) where the default Unicode case mapping has one. str.lower() and str.casefold() implement the default mapping, so a guardrail that normalises with them before matching a keyword list can be stepped around: "IGNORE" lowered under tr rules is "ıgnore", which no longer contains "ignore", while the text stays readable to a model. Going the other way, İ lowercases to i + U+0307, so the dot survives a later normalisation pass and still fails an equality test. Adds TurkishLowercase and TurkishUppercase alongside buffs.lowercase, so the two mappings can be compared against the same target. No new dependencies. Signed-off-by: Fevzi Ege Yurtsevenler <egeyurtsevenler@gmail.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
buffs.turkish_casefoldwithTurkishLowercaseandTurkishUppercase.Why
Turkish keeps the dot on
ias a distinguishing feature across case changes, so it has two i-pairs —I/ıandİ/i— where the default Unicode case mapping has one.str.lower()andstr.casefold()implement the default mapping, so anything that normalises with them before matching a keyword list can be stepped around:The other direction leaves a residue that survives normalisation, because
İlowercases toi+ U+0307 rather than toi:This is distinct from the homoglyph substitution in
probes.smuggling— no characters are swapped for lookalikes from another script. These are the codepoints atrlocale legitimately produces, so the text survives round-tripping through anything that is locale-aware.What
Two buffs, deliberately placed next to
buffs.lowercaseso the default andtrmappings can be run against the same target and compared.lang = "*", since the transformation applies to any source language.Notes
str.translateplus a two-entry map.tests/buffs/test_buffs.pypicks the new classes up automatically via plugin enumeration.index_buffs.rst;tests/test_docs.pypasses.Local run:
pytest tests/buffs/ tests/test_docs.py tests/test_configurable.py— all green.