Hidden-Unicode text integrity: scanner, CI, editor, guard, normalize - #111
Merged
Conversation
Add a general-purpose hidden/suspicious Unicode audit (fuckmark-hidden-scan-v1) as the defensive inverse of the mix. fuckmark --scan reports hidden characters without changing the text; fuckmark --clean strips them while keeping the visible text. Coverage is general, not FuckMark-only: bidirectional controls (Trojan Source, CVE-2021-42574), zero-width and invisible spacing, Unicode tag characters (hidden-text / prompt-injection smuggling), variation selectors, enclosing marks, deprecated interlinear controls, other Cf format controls, C0/C1 controls, private-use codepoints, noncharacters, and surrogates. Tab, newline, carriage return, and space are never flagged; ordinary combining accents are left alone. Wire --scan/--clean into the CLI (mutually exclusive with --detect; neither combines with --visible), add POST /api/scan to fuckmark web, and export the Python API (scan_hidden_characters, clean_hidden_characters, classify_hidden_codepoint, ScanResult, HiddenFinding). Version stays 0.4.1 and frozen confirmation artifacts are untouched. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
byte271
marked this pull request as ready for review
September 1, 2026 19:54
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3702a3c70e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| ) | ||
|
|
||
| _VARIATION_SELECTOR = frozenset({*range(0xFE00, 0xFE10), *range(0xE0100, 0xE01F0)}) |
There was a problem hiding this comment.
The advertised variation-selector scan misses U+180B–U+180D and U+180F. Those Mongolian free variation selectors are Mn, so they do not fall into any later general-category check: for example, fuckmark --scan --text $'a\u180bb' reports clean and --clean preserves the hidden selector. Add this range to the variation-selector classification so the defensive scanner and cleaner cover the named category.
Useful? React with 👍 / 👎.
Ship the adoption surface for the scanner: a 'fuckmark lint' subcommand that walks files and directories, reports hidden or malicious Unicode per file, and exits non-zero on findings (0 clean, 1 findings or files changed by --fix, 2 usage). Options: --json, --fix, --select (default security set, or 'all'), --exclude, --max-bytes, -q. Binary, non-UTF-8, oversized, symlinked, and vendored/VCS paths are skipped. Dispatched like 'fuckmark web'; the transform CLI is unchanged. Add a drop-in GitHub Action (action.yml) and pre-commit hooks (.pre-commit-hooks.yaml: fuckmark, fuckmark-fix) so any repository can block Trojan Source bidi overrides (CVE-2021-42574), zero-width characters, and tag smuggling in CI or on commit. Docs in docs/lint.md, README, and CHANGELOG. Version stays 0.4.1; frozen artifacts untouched. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
cursor
Bot
changed the title
Ship the editor surface: a zero-dependency, no-build VS Code/Cursor extension under editors/vscode that reveals hidden or malicious Unicode inline (red box plus a visible U+XXXX badge even on zero-width codepoints), lists it in the Problems panel (high-risk classes as warnings), shows a live status-bar count, and cleans files or selections on command. The scanner (editors/vscode/scan.js) is a faithful JavaScript port of fuckmark-hidden-scan-v1, pinned to the Python engine across all 1,114,112 codepoints by tests/test_vscode_scanner_parity.py (skips when node is absent). Offsets are UTF-16 correct, including astral tag and variation-selector characters. Docs in editors/vscode/README.md, README, and CHANGELOG. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
Ship fuckmark-guard-v1, the drop-in sanitizer for prompt-injection smuggling via hidden Unicode. protect() / inspect() / Guard / @guard().wrap walk strings, lists, and dicts (OpenAI-style messages included) and strip the security category set by default. on_findings=refuse raises HiddenTextRefused without calling the inner function; report scans only. Unicode tag characters (U+E0020-U+E007E) are decoded to ASCII as tag_payload so operators can log the smuggled instruction. extract_tag_payload is exported from the Python API and from editors/vscode/guard.js. CLI: fuckmark guard (--json, --refuse, --report, --receipt, --select). HTTP: POST /api/guard on fuckmark web. Shared SECURITY_SCAN_CATEGORIES with fuckmark lint. Honest limit: this is not a semantic jailbreak detector. Docs in docs/guard.md, README, docs/cli.md, CHANGELOG. Version stays 0.4.1; frozen artifacts untouched. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
cursor
Bot
changed the title
CI / Python 3.11 failed because unicodedata 14 classifies U+13439-U+1343F (and other Unicode 15 format controls) as unassigned Cn, while the JS port uses explicit Unicode 15 Cf ranges and labels them format. Classify those ranges as format in the Python engine before consulting unicodedata, matching editors/vscode/scan.js. Keep a parity-test allowance for leftover Cn vs format drift, and add a regression that Egyptian format controls stay format even on older Unicode databases. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
Co-authored-by: Byte271 <byte271@users.noreply.github.com>
Context-aware severity and explanations on fuckmark-hidden-scan-v1 findings, a normalize pipeline with NFC/lookalike/strip receipts, and hash-bound conformance vectors under specs/. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
cursor
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns FuckMark's engine into a defensive, broadly adoptable text-integrity surface. One engine (
fuckmark-hidden-scan-v1), six surfaces.1. Scanner / cleaner (
--scan/--clean)fuckmark --scan— audit any text for hidden or suspicious Unicode without changing it.fuckmark --clean— strip it while keeping the visible text (also reverses a FuckMark mix).General classification (not FuckMark-only):
bidi_control(Trojan Source, CVE-2021-42574),zero_width,tag(hidden-text / LLM prompt-injection smuggling),variation_selector,enclosing_mark,line_separator,deprecated,format,control,private_use,noncharacter,surrogate. Tab/newline/CR/space are never flagged; ordinary accents (Mn) are left alone.Findings now carry context (
identifier/emoji/string/prose), severity (critical/high/medium/info), and a one-line why/remedy. A bidi override inside an identifier is critical; a ZWJ inside an emoji cluster is info.Frozen spec:
specs/fuckmark-hidden-scan-v1.protocol.md+vectors.json+freeze.json. Conformance vectors are codepoint arrays only;tests/test_hidden_scan_spec.pyreplays them on Python and oneditors/vscode/scan.js.2. Normalize (
fuckmark normalize)Pipeline default: NFC, optional identifier lookalike fold (UTS #39-inspired subset, not a full confusable map), then strip of the security category set, plus a JSON receipt of input/output hashes and steps.
--cleanremains the blunt strip with no receipt.fuckmark normalize --receiptnormalize_text,skeleton_foldPOST /api/normalizedocs/normalize.md3. CI / pre-commit (
fuckmark lint+ Action + hook)fuckmark lint PATHS— walks files/dirs, reports per file, exits non-zero on findings.--jsonlocations now include context/severity/why/remedy.--fix,--select,--exclude,--max-bytes,-q.action.yml):uses: byte271/FuckMark@main..pre-commit-hooks.yaml): idsfuckmark,fuckmark-fix.4. Editor (VS Code / Cursor,
editors/vscode)Inline red box + a visible
‹U+XXXX›badge on every hidden character, hover text with why it matters, Problems-panel diagnostics (criticalas Error), a live status-bar count, clean-file / clean-selection commands, and optionalfuckmark.cleanOnSave. Zero dependencies, no build.scan.jsis pinned to the Python engine across all 1,114,112 codepoints.5. LLM input guard (
protect/fuckmark guard)Drop-in sanitizer for prompt-injection smuggling via hidden Unicode (especially Unicode tag characters that encode a second ASCII string a human will not see).
Guard(on_findings="refuse")raisesHiddenTextRefusedwithout calling a wrapped complete function.@Guard().wrapsanitizes every string argument (OpenAI-stylemessagesincluded).fuckmark guard --json < messages.jsonPOST /api/guardeditors/vscode/guard.jsHonest limit: this is not a semantic jailbreak detector. It only removes characters that hide a second payload in the byte stream.
Notes
SECURITY_SCAN_CATEGORIES) across lint, guard, and normalize strip so they cannot drift.--cleanalso strips emoji ZWJ / variation selectors;clean_hidden_characters(text, categories=...)accepts a subset. Guard/lint/normalize default to the security set, so emoji variation selectors are kept.Tests
tests/test_scan.py,tests/test_lint.py,tests/test_vscode_scanner_parity.py,tests/test_guard.py,tests/test_normalize.py,tests/test_hidden_scan_spec.py.