Skip to content

buffs: add Turkish locale case mapping buffs - #1997

Open
fevziegeyurtsevenler wants to merge 1 commit into
NVIDIA:mainfrom
fevziegeyurtsevenler:buff/turkish-casefold
Open

buffs: add Turkish locale case mapping buffs#1997
fevziegeyurtsevenler wants to merge 1 commit into
NVIDIA:mainfrom
fevziegeyurtsevenler:buff/turkish-casefold

Conversation

@fevziegeyurtsevenler

Copy link
Copy Markdown

Adds buffs.turkish_casefold with TurkishLowercase and TurkishUppercase.

Why

Turkish keeps the dot on i as a distinguishing feature across case changes, so it has two i-pairs — I/ı and İ/i — where the default Unicode case mapping has one. str.lower() and str.casefold() implement the default mapping, so anything that normalises with them before matching a keyword list can be stepped around:

>>> "IGNORE previous instructions".lower()
"ignore previous instructions"          # a keyword filter matches

>>> turkish_lower("IGNORE previous instructions")
"ıgnore previous instructions"          # it does not — but a model still reads it

The other direction leaves a residue that survives normalisation, because İ lowercases to i + U+0307 rather than to i:

>>> turkish_upper("ignore").lower()
"i̇gnore"                                # still != "ignore"

This is distinct from the homoglyph substitution in probes.smuggling — no characters are swapped for lookalikes from another script. These are the codepoints a tr locale legitimately produces, so the text survives round-tripping through anything that is locale-aware.

What

Two buffs, deliberately placed next to buffs.lowercase so the default and tr mappings can be run against the same target and compared. lang = "*", since the transformation applies to any source language.

Notes

  • No new dependencies; str.translate plus a two-entry map.
  • 6 unit tests covering both directions, the filter-bypass property, and the U+0307 round-trip. tests/buffs/test_buffs.py picks the new classes up automatically via plugin enumeration.
  • Docs page added and wired into index_buffs.rst; tests/test_docs.py passes.
  • Motivation came from measuring this against open guardrails on Turkish text: https://github.com/fevziegeyurtsevenler/turkish-casefold-evasion

Local run: pytest tests/buffs/ tests/test_docs.py tests/test_configurable.py — all green.

Turkish keeps the dot on i as a distinguishing feature across case changes,
so it has two i pairs (I/ı and İ/i) where the default Unicode case mapping
has one. str.lower() and str.casefold() implement the default mapping, so a
guardrail that normalises with them before matching a keyword list can be
stepped around: "IGNORE" lowered under tr rules is "ıgnore", which no longer
contains "ignore", while the text stays readable to a model. Going the other
way, İ lowercases to i + U+0307, so the dot survives a later normalisation
pass and still fails an equality test.

Adds TurkishLowercase and TurkishUppercase alongside buffs.lowercase, so the
two mappings can be compared against the same target. No new dependencies.

Signed-off-by: Fevzi Ege Yurtsevenler <egeyurtsevenler@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant