Hidden-Unicode text integrity: scanner, CI, editor, guard, normalize

.pre-commit-hooks.yaml

@@ -0,0 +1,14 @@
+- id: fuckmark
+  name: FuckMark hidden-Unicode scan
+  description: "Fail on hidden or malicious Unicode: Trojan Source bidi overrides, zero-width characters, and tag smuggling."
+  entry: fuckmark lint
+  language: python
+  types: [text]
+  pass_filenames: true
+- id: fuckmark-fix
+  name: FuckMark hidden-Unicode fix
+  description: "Strip hidden or malicious Unicode in place, keeping the visible text."
+  entry: fuckmark lint --fix
+  language: python
+  types: [text]
+  pass_filenames: true

CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## Unreleased
+
+- Defensive hidden-Unicode scanner and cleaner (`fuckmark-hidden-scan-v1`). `fuckmark --scan` audits any text for hidden or suspicious Unicode without changing it; `fuckmark --clean` strips it while keeping the visible text. Coverage is general, not FuckMark-only: bidirectional controls (Trojan Source, CVE-2021-42574), zero-width and invisible spacing, Unicode tag characters (hidden-text / prompt-injection smuggling), variation selectors, enclosing marks, deprecated interlinear controls, other `Cf` format controls, C0/C1 controls, private-use codepoints, and noncharacters. Tab, newline, carriage return, and space are never flagged, and ordinary combining accents (`Mn`) are left alone.
+- `--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. `--clean` removes every flagged category (so it reverses a FuckMark mix back to the visible text) and reports the count removed. `--detect`, `--scan`, and `--clean` are mutually exclusive; neither `--scan` nor `--clean` combines with `--visible`.
+- `fuckmark web` serves the same engine at `POST /api/scan`. New Python API: `scan_hidden_characters`, `clean_hidden_characters`, `classify_hidden_codepoint`, `scan_dict`, `scan_human_report`, `scan_machine_line`, `ScanResult`, and `HiddenFinding` (exported from `fuckmark` and `fuckmark.product`).
+- `fuckmark lint PATHS` walks files and directories, reports hidden Unicode per file, and exits non-zero on findings (`0` clean, `1` findings or files changed by `--fix`, `2` usage). Options: `--json`, `--fix`, `--select CATS` (default security set `bidi_control,zero_width,tag,control,noncharacter,surrogate`, or `all`), `--exclude GLOB`, `--max-bytes`, `-q`. Binary, non-UTF-8, oversized, symlinked, and vendored/VCS paths are skipped. Dispatched like `fuckmark web`; the main transform CLI is unchanged.
+- Ship a GitHub Action (`action.yml`) and pre-commit hooks (`.pre-commit-hooks.yaml`, ids `fuckmark` and `fuckmark-fix`) that run `fuckmark lint`, so any repository can gate hidden Unicode in CI or on commit. Reference: `docs/lint.md`.
+- VS Code / Cursor extension under `editors/vscode`: inline reveal of hidden characters (red box plus a visible `U+XXXX` badge even on zero-width codepoints), Problems-panel diagnostics (high-risk classes as warnings), a status-bar count, and clean-file / clean-selection / toggle-reveal / rescan commands. Zero runtime dependencies and no build step. Its scanner (`editors/vscode/scan.js`) is a faithful port of `fuckmark-hidden-scan-v1`, pinned to the Python engine across all 1,114,112 codepoints by `tests/test_vscode_scanner_parity.py` (which skips when `node` is absent, e.g. in CI).
+- LLM input guard (`fuckmark-guard-v1`). `protect()` / `inspect()` / `Guard` / `@Guard().wrap` sanitize strings, lists, and dicts (OpenAI-style `messages` included) before a model call. Default action is strip; `on_findings="refuse"` raises `HiddenTextRefused` without calling the inner function; `report` scans only. Unicode tag characters are decoded to ASCII as `tag_payload` on the receipt (`extract_tag_payload`). CLI: `fuckmark guard` (`--json`, `--refuse`, `--report`, `--receipt`, `--select`). HTTP: `POST /api/guard`. JS: `editors/vscode/guard.js`. This does not detect semantic prompt injection. Default categories are the shared `SECURITY_SCAN_CATEGORIES` also used by `fuckmark lint`. Reference: `docs/guard.md`.
+- Hidden-scan `format` classification uses the same explicit Cf ranges as `editors/vscode/scan.js` (including U+13430-U+1343F and U+0890-U+0891) instead of only `unicodedata.category`, so Python 3.11 (Unicode 14) agrees with the JS port and with Python 3.12+ (Unicode 15).
+- Hidden-scan findings now carry **context**, **severity**, and a one-line **why/remedy**. A bidi override inside an identifier is `critical` (Trojan Source); a ZWJ inside an emoji cluster is `info`. Lint JSON, `--scan` reports, and the VS Code hover/Problems panel all show those fields. Diagnostic rank: `critical` is Error, `high` is Warning.
+- Frozen classification spec `fuckmark-hidden-scan-v1` under `specs/` (`protocol.md`, `vectors.json`, `freeze.json`). Conformance vectors are codepoint arrays only; `tests/test_hidden_scan_spec.py` replays them on the Python engine and on `editors/vscode/scan.js`.
+- `fuckmark normalize` (`fuckmark-normalize-v1`): NFC, optional identifier lookalike fold (UTS #39-inspired subset, not a full confusable map), then strip of the security category set, plus a JSON receipt (`input_sha256` / `output_sha256` / `steps` / `report_hash`). `--clean` remains the blunt strip with no receipt. HTTP: `POST /api/normalize`. Python: `normalize_text`, `skeleton_fold`. Reference: `docs/normalize.md`.
+- VS Code / Cursor: `fuckmark.cleanOnSave` (default off) strips hidden Unicode on save.
+- Frozen confirmation files, hashes, mix mechanisms, and the published wheel are unchanged. Package version stays 0.4.1.
+
 ## v0.4.1
 
 Audit findings D01-D16, E01-E04, L01-L06, P01, and G01 against `d7dc98b7ee99fc46b767c93444a9f051fdbea2ba`. Do not retag v0.4.0. Frozen confirmation files and their SHA-256 sums are unchanged.

README.md

@@ -122,6 +122,81 @@ printf 'I do not agree.\n' | .venv/bin/fuckmark --inspect >/tmp/fm.out
 
 Pipes and files write the payload to stdout. `--visible` prints the original visible text. `--status` writes a machine-readable outcome to stderr. `--inspect` writes a character-level map to stderr. `fuckmark --help` is enough to start.
 
+## Scan and clean hidden Unicode (defensive)
+
+FuckMark also works the other way. `--scan` audits any text for hidden or suspicious Unicode and reports it without changing the text. `--clean` strips those characters while keeping the visible text. This is a general audit, not a FuckMark-only scan: it covers bidirectional controls (the [Trojan Source](https://trojansource.codes/) class, CVE-2021-42574), zero-width and invisible spacing, Unicode tag characters used to smuggle hidden text into LLM prompts, variation selectors, enclosing marks, deprecated interlinear controls, other `Cf` format controls, C0/C1 controls, private-use codepoints, and noncharacters.
+
+```text
+printf 'if (x != \u202eadmin\u202c) {\n' | .venv/bin/fuckmark --scan
+.venv/bin/fuckmark --scan --file suspect.txt
+.venv/bin/fuckmark --clean --file suspect.txt -o clean.txt
+```
+
+`--scan` prints a human report by default, a machine line with `-q`, and a `fuckmark-scan ...` status line to stderr with `--status`. Findings include context (`identifier` / `emoji` / `string` / `prose`) and severity (`critical` / `high` / `medium` / `info`): a bidi override inside an identifier is critical; a ZWJ inside an emoji cluster is info. `--clean` removes every flagged category (including FuckMark's own carriers, so `--clean` reverses a mix back to the visible text) and reports how many characters it removed. The same engine is available in the browser tool via `POST /api/scan`, and in Python through `scan_hidden_characters`, `clean_hidden_characters`, and `classify_hidden_codepoint`. Frozen spec: [`specs/fuckmark-hidden-scan-v1.protocol.md`](specs/fuckmark-hidden-scan-v1.protocol.md).
+
+Whitespace tab, newline, carriage return, and space are never flagged, and ordinary combining accents (`Mn`) are left alone. `--clean` strips emoji zero-width joiners and variation selectors as well, so pass a category subset to `clean_hidden_characters(...)` in Python if you need to keep emoji sequences intact.
+
+For pipelines that should canonicalize rather than only strip, `fuckmark normalize` NFC-folds, optionally skeleton-folds a small identifier lookalike subset, strips the security category set, and writes a JSON receipt of what changed. That lookalike table is not a full UTS #39 map. Full reference: [`docs/normalize.md`](docs/normalize.md).
+
+```text
+printf 'notes\n' | .venv/bin/fuckmark normalize --receipt
+```
+
+## Guard a repository (CI, pre-commit, editors)
+
+`fuckmark lint` scans files and directories and exits non-zero on findings, so you can block hidden Unicode before it lands. It catches [Trojan Source](https://trojansource.codes/) bidi overrides (CVE-2021-42574), zero-width characters, and Unicode tag smuggling by default.
+
+```text
+fuckmark lint .            # scan the tree; exit 1 on findings
+fuckmark lint --json .     # machine-readable report
+fuckmark lint --fix .      # strip findings in place
+```
+
+Drop-in **GitHub Action**:
+
+```yaml
+- uses: actions/checkout@v4
+- uses: actions/setup-python@v5
+  with:
+    python-version: "3.12"
+- uses: byte271/FuckMark@main
+```
+
+Drop-in **pre-commit** hook:
+
+```yaml
+repos:
+  - repo: https://github.com/byte271/FuckMark
+    rev: v0.4.1
+    hooks:
+      - id: fuckmark
+```
+
+Binary, non-UTF-8, oversized, and vendored/VCS paths are skipped; `--exclude GLOB`, `--select`, and `--max-bytes` tune the run. Full reference: [`docs/lint.md`](docs/lint.md).
+
+## See it in your editor (VS Code / Cursor)
+
+The [`editors/vscode`](editors/vscode) extension reveals hidden Unicode inline as you read code: a red box and a visible `‹U+202E›` badge on every hidden character, hover text with why it matters, diagnostics in the Problems panel (`critical` as Error), a status-bar count, one-click clean, and optional clean-on-save (`fuckmark.cleanOnSave`). It is zero-dependency plain JavaScript with no build step, and its scanner is a faithful port of `fuckmark-hidden-scan-v1` — pinned to the Python engine across every Unicode codepoint by [`tests/test_vscode_scanner_parity.py`](tests/test_vscode_scanner_parity.py), so the editor and the CLI agree exactly. Load it with `code --extensionDevelopmentPath=editors/vscode` or open the folder and press `F5`.
+
+## Guard model input (LLM prompt-injection smuggling)
+
+Hidden Unicode — especially Unicode **tag** characters — can smuggle a second instruction into a prompt that a human reviewer will not see. `protect()` strips that payload before the text reaches a model, and can decode the smuggled ASCII so you can log it. It is not a semantic jailbreak detector.
+
+```python
+from fuckmark import protect, inspect
+
+safe = protect(user_text)
+cleaned, receipt = inspect(user_text)
+# receipt.tag_payload is the hidden ASCII, when tags were used
+```
+
+```text
+printf 'user text\n' | .venv/bin/fuckmark guard
+.venv/bin/fuckmark guard --json < messages.json
+```
+
+Wrap an existing complete function with `@Guard().wrap`, or refuse hidden input with `Guard(on_findings="refuse")`. Same engine on `POST /api/guard` and in `editors/vscode/guard.js`. Full reference: [`docs/guard.md`](docs/guard.md).
+
 ## What it guarantees
 
 `VISIBLE(original) == VISIBLE(transformed)` under approved-carrier projection. FuckMark inserts U+034F or U+FE00, a C0/C1 control, U+20DD, a cycling U+13430-U+13438 format control, and a cycling U+FFF9-U+FFFB annotation control after eligible letter and emoji grapheme clusters. It does not contract, paraphrase, homoglyph, or add spaces. Transformation selection does not use detectors or watermark keys.

action.yml

@@ -0,0 +1,54 @@
+name: "FuckMark hidden-Unicode scan"
+description: "Fail the build on hidden or malicious Unicode: Trojan Source bidi overrides, zero-width characters, and Unicode tag smuggling."
+author: "Byte271"
+branding:
+  icon: "eye"
+  color: "purple"
+
+inputs:
+  paths:
+    description: "Space-separated files or directories to scan."
+    required: false
+    default: "."
+  select:
+    description: "Comma-separated categories to fail on, or 'all' for every category."
+    required: false
+    default: "bidi_control,zero_width,tag,control,noncharacter,surrogate"
+  fix:
+    description: "Strip the selected hidden characters in place instead of only reporting."
+    required: false
+    default: "false"
+  json:
+    description: "Emit a machine-readable JSON report to the step output."
+    required: false
+    default: "false"
+  args:
+    description: "Extra raw arguments passed through to 'fuckmark lint'."
+    required: false
+    default: ""
+
+runs:
+  using: "composite"
+  steps:
+    - name: Install FuckMark
+      shell: bash
+      run: python3 -m pip install --disable-pip-version-check --quiet "${{ github.action_path }}"
+    - name: Scan for hidden Unicode
+      shell: bash
+      run: |
+        set -euo pipefail
+        cmd=(fuckmark lint --select "${{ inputs.select }}")
+        if [ "${{ inputs.fix }}" = "true" ]; then
+          cmd+=(--fix)
+        fi
+        if [ "${{ inputs.json }}" = "true" ]; then
+          cmd+=(--json)
+        fi
+        if [ -n "${{ inputs.args }}" ]; then
+          # shellcheck disable=SC2206
+          cmd+=(${{ inputs.args }})
+        fi
+        # shellcheck disable=SC2206
+        cmd+=(${{ inputs.paths }})
+        printf 'Running:'; printf ' %q' "${cmd[@]}"; printf '\n'
+        "${cmd[@]}"

docs/cli.md

@@ -63,6 +63,11 @@ printf 'I do not agree.\n' | fuckmark --status >/tmp/fm.out
 printf 'I do not agree.\n' | fuckmark --inspect >/tmp/fm.out
 fuckmark --detect --text "I do not agree."
 printf 'paste\n' | fuckmark --detect
+fuckmark --scan --file suspect.txt
+fuckmark --clean --file suspect.txt -o clean.txt
+fuckmark lint src/
+fuckmark guard --json < messages.json
+fuckmark normalize --receipt < notes.txt
 fuckmark web
 fuckmark --text "I don’t agree." --status
 ```
@@ -101,11 +106,44 @@ Existing files are read as UTF-8 bytes with no newline conversion. LF, CRLF, CR,
 | `--status` | Write one `fuckmark-status` line to stderr (`result`, `processed`, `insertions`, `sites`, `last_index`, `source_length`, `capped`, `first_unsupported`). |
 | `--inspect` | Write a character-level coverage map to stderr. Stdout stays the payload. |
 | `--detect` | Scan for FuckMark insertions without transforming. Stdout is the detect report. If none are found, the report includes `Fhelp@q1z.org`. |
+| `--scan` | Audit any text for hidden or suspicious Unicode without transforming. Stdout is the scan report (human by default, `fuckmark-scan ...` machine line with `-q`). |
+| `--clean` | Strip hidden or suspicious Unicode while keeping the visible text. Stdout is the cleaned payload. Reports the count removed on stderr. |
 | `--no-color` | Disable color on stderr. `NO_COLOR` does the same. |
 
+`--detect`, `--scan`, and `--clean` are mutually exclusive. Neither `--scan` nor `--clean` combines with `--visible`.
+
+### Hidden-Unicode scan and clean
+
+`--scan` and `--clean` are the defensive inverse of the mix. The scan is general, not FuckMark-only. Flagged categories: `bidi_control` (Trojan Source, CVE-2021-42574), `zero_width`, `variation_selector`, `tag` (hidden-text / prompt-injection smuggling), `enclosing_mark`, `line_separator`, `deprecated` (interlinear annotation and deprecated format controls), `format` (other `Cf`), `control` (C0/C1), `private_use`, `noncharacter`, and `surrogate`. Tab, newline, carriage return, and space are never flagged; ordinary combining accents (`Mn`) are left alone.
+
+Findings include context (`identifier`, `emoji`, `string`, `prose`) and severity (`critical`, `high`, `medium`, `info`). A bidi override next to an identifier is `critical`; a ZWJ inside an emoji cluster is `info`. The frozen table is [`specs/fuckmark-hidden-scan-v1.protocol.md`](../specs/fuckmark-hidden-scan-v1.protocol.md).
+
+`--clean` removes every flagged category, so it reverses a FuckMark mix back to the visible text. It also removes emoji zero-width joiners and variation selectors; the Python `clean_hidden_characters(text, categories=...)` call accepts a category subset when emoji sequences must be preserved. The browser tool exposes the same engine at `POST /api/scan`.
+
+### `fuckmark normalize`
+
+NFC-fold, optionally skeleton-fold a small identifier lookalike subset, then strip the security hidden-character set. Emits a JSON receipt of hashes and steps. This is the pipeline default; `--clean` is strip-only and has no receipt. The lookalike table is UTS #39-inspired, not a full confusable map. Reference: [`normalize.md`](normalize.md).
+
+```text
+fuckmark normalize --receipt < notes.txt
+fuckmark normalize --confusable --receipt notes.txt
+```
+
+### `fuckmark guard`
+
+Sanitize text or JSON before it reaches a model. Strips the security category set by default (same as `fuckmark lint`) and can recover Unicode-tag smuggling as `tag_payload` on the receipt. Does not detect semantic prompt injection.
+
+```text
+printf 'user text\n' | fuckmark guard
+fuckmark guard --json < messages.json
+fuckmark guard --refuse --receipt --json < messages.json
+```
+
+`--json` walks every string. `--refuse` exits 1 and writes nothing when hidden Unicode is present. `--report` scans without changing the payload. `--receipt` writes the JSON receipt to stderr. Python: `protect()`, `inspect()`, `Guard`, `HiddenTextRefused`. Reference: [`guard.md`](guard.md).
+
 ### `fuckmark web`
 
-Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health` and `POST /api/remove-marks`. Detect and strip on that page use `detect_fuckmark_insertions` and `project_visible_v1` when the API is up.
+Open the local browser tool (same UI as `docs/mark.html`). Aimed at beginners who prefer a page over pipes and flags. The server also exposes a Python API: `GET /api/health`, `POST /api/remove-marks`, `POST /api/scan`, `POST /api/guard` (sanitize text or chat messages before a model call), and `POST /api/normalize`.
 
 ```text
 fuckmark web