Add a public sanitizer-restore robustness bench - #117
Conversation
Ship fuckmark-robustness-bench-v1 as `fuckmark robustness`: 180 hashed cells of live mix versus Unicode sanitizers, with no GPT-2 or SynthID rerun. Detector numbers stay on the sealed Gate v2 scorecard. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 30f923b6fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| SPEC_DIR = Path(__file__).resolve().parents[1] / "specs" | ||
| PROTOCOL_PATH = SPEC_DIR / "fuckmark-robustness-bench-v1.protocol.md" | ||
| VECTORS_PATH = SPEC_DIR / "fuckmark-robustness-bench-v1.vectors.json" | ||
| FREEZE_PATH = SPEC_DIR / "fuckmark-robustness-bench-v1.freeze.json" |
There was a problem hiding this comment.
When FuckMark is installed from the wheel, this path resolves to site-packages/specs, but the wheel only packages files under fuckmark and contains none of the referenced vectors, protocol, freeze, or scorecard files. Consequently every installed invocation of fuckmark robustness raises FileNotFoundError while loading the vectors instead of running the advertised public command; these artifacts need to be installed as package data and resolved from that packaged location.
Useful? React with 👍 / 👎.
| vectors = load_vectors() | ||
| mismatches = compare_to_vectors(cells, vectors) | ||
| sealed = sealed_detector_track() | ||
| sealed_ok = sealed["scorecard_hash"] == sealed["expected_scorecard_hash"] |
There was a problem hiding this comment.
When the scorecard contents are changed while its embedded scorecard_hash field is left unchanged, sealed_detector_ok remains true even though file_sha256 no longer matches the frozen value; similarly, the loaded vectors are trusted without checking their frozen file or canonical hashes. Because run_robustness_bench never reads or compares the freeze record, coordinated sanitizer/vector drift or modified detector counts can still produce exit 0, defeating the advertised hashed integrity check.
Useful? React with 👍 / 👎.
Load protocol, vectors, freeze, and the sealed scorecard from packaged module data so `fuckmark robustness` works from a wheel. Compare those file hashes to the freeze record before exit 0, including scorecard bytes when the embedded hash is left unchanged. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
Windows Package E2E failed because Git checkout CRLF changed protocol, vectors, and scorecard file SHA-256. Hash UTF-8 after LF folding so the freeze matches, and pin those artifacts to LF in .gitattributes. Co-authored-by: Byte271 <byte271@users.noreply.github.com>
2 participants
Public sanitizer-restore bench for the live five-layer mix (Phase 4 /
WATERMARK-ROBUSTNESS-BENCHv1).This is a hashed, model-free restore grid. It does not rerun GPT-2, SynthID, or any neural detector. Detector numbers stay on the sealed Gate v2 confirmation scorecard (identity 188/192, mix 0/192 after required sanitizers, visible 192/192). Frozen confirmation artifacts and the published wheel are unchanged. Package version stays 0.4.1.
What landed
fuckmark robustness(--json,-q,--fixture,--attack)fuckmark-robustness-bench-v1restores_source, mix/output visible projection, closed-set carrier detect,fuckmark-hidden-scan-v1residual categories, SHA-256 of mix and attack outputdigitshas no eligible site so mix is a no-op and all 18 attacks restore;ascii_prose/mn_me_us_cfrestore false, carrier false, projection false (UnicodeSanitizer turns annotation controls into spaces)specs/fuckmark-robustness-bench-v1.{protocol.md,vectors.json,freeze.json}fuckmark.cli(no cycle throughcycle8.benchmark/gate_v2)fuckmark.robustnessare pinned to the cycle8 reference implementations by testdocs/robustness.md, plus CLI / README / CHANGELOG / research / limitstests/test_robustness.pybinds freeze hashes and replays every cellExit
0match,1mix or sanitizer drift,2usage.Codex follow-up (
ea6aa95)fuckmark robustnessloads protocol, vectors, freeze, and the sealed scorecard fromfuckmark/robustness_data/(must matchspecs/after LF folding), so a wheel install no longer looks forsite-packages/specsscorecard_hashfield alone is still a mismatchtools/verify_release_install.py) runsfuckmark robustness --json --fixture digits --attack identityand requiresfreeze_okWindows Package E2E (
cee921d)Git checkout CRLF changed raw file SHA-256 of protocol/vectors/scorecard on
windows-latest. Artifact hashes now fold CRLF to LF (same digest as the Linux freeze)..gitattributespins those files to LF.A later v2 may add opt-in detector adapters. This freeze is local and deterministic. Short public fixtures are not the GPT-2 Gate v2 corpus. Sanitizer restore is not a detector miss, and a GPT-2 / SynthID miss is not a general AI-detector rate.