Presentation polish: CI, unit tests, CONTRIBUTING, submission path

.github/workflows/ci.yml

@@ -0,0 +1,29 @@
+name: CI
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install (probe/emergence/deception axes need numpy+sklearn)
+        run: pip install numpy scikit-learn pytest
+      - name: Offline smoke test (mock NLA, no network/GPU)
+        run: python run_example.py
+      - name: CTF Red/Blue demo (reproducible, control-tier)
+        run: python experiments/ctf_red_blue_demo.py
+      - name: Unit tests
+        run: pytest -q

CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing to NLAttack
+
+Thanks for helping evaluate Natural Language Autoencoders. There are two main ways
+to contribute: **submit a result** for an NLA, or **improve the harness/catalog**.
+
+## Submit a result (add your NLA to the leaderboard)
+
+NLAttack is built so any NLA can be scored with one adapter method.
+
+1. **Implement the adapter.** Provide an `NLA` with a `reconstruct(text) -> str`
+   method (the AV verbalization of the activation). Hosted, text-only NLAs use the
+   universal API tier; local NLAs with raw activations unlock the full-access tier.
+   See [`docs/EVALUATIONS.md`](docs/EVALUATIONS.md).
+2. **Run the suite** and write the result JSON under `results/` (a new file — never
+   overwrite another NLA's artifact). Worked runners are in `experiments/`.
+3. **Attribute it.** Name results by the **NLA**, not the base model, and record the
+   canonical NLA id, suite version (`nla_eval.__version__`), dataset, matcher
+   backend, and date — see the attribution convention in
+   [`docs/RESULTS.md`](docs/RESULTS.md). `nla_name()` in
+   `experiments/cross_nla_eval.py` fills these in.
+4. **Open a PR** that adds the result file, a row in `docs/RESULTS.md`, and an entry
+   in [`results/README.md`](results/README.md) (generation provenance). Report a
+   number with its **null control** — a result counts only when it clears the
+   permutation/chance floor.
+
+## Improve the harness or the plan catalog
+
+- **New evaluation plan:** follow the schema in [`plans/README.md`](plans/README.md)
+  (hypothesis, method, metric, feasibility, controls, "null looks like"). Add it to
+  the right family file and to [`plans/INDEX.md`](plans/INDEX.md).
+- **New coded axis:** add the module under `nla_eval/`, export it from
+  `nla_eval/__init__.py`, and add a unit test under `tests/`.
+
+## Development
+
+```bash
+pip install numpy scikit-learn pytest   # probe/emergence/deception axes + tests
+python run_example.py                    # offline smoke test
+python experiments/ctf_red_blue_demo.py  # CTF Red/Blue demo
+pytest -q                                # unit tests
+```
+
+CI runs the smoke test, the CTF demo, and `pytest` on Python 3.9 / 3.11 / 3.12; keep
+them green. Two house rules:
+
+- **Null controls everywhere.** Every reported number clears an explicit
+  permutation/chance floor; negatives are reported honestly, not hidden.
+- **Freeze-on-release.** Published results are frozen per generation
+  ([`docs/VERSIONING.md`](docs/VERSIONING.md)); new content lands in a new
+  generation and never edits a prior one's artifacts.
+
+By contributing you agree your contributions are licensed under Apache-2.0
+([`LICENSE`](LICENSE)).

README.md

@@ -6,8 +6,10 @@
 
 **An evaluation suite for Natural Language Autoencoders (NLAs).**
 
+[![CI](https://github.com/SolshineCode/NLAttack/actions/workflows/ci.yml/badge.svg)](https://github.com/SolshineCode/NLAttack/actions/workflows/ci.yml)
 ![license](https://img.shields.io/badge/license-Apache--2.0-blue.svg)
 ![version](https://img.shields.io/badge/release-v2.0.0-brightgreen.svg)
+![python](https://img.shields.io/badge/python-3.9%2B-blue.svg)
 ![status](https://img.shields.io/badge/status-research%20software-orange.svg)
 
 A **Natural Language Autoencoder** explains a model's internal state in plain
@@ -61,6 +63,10 @@ concepts). The local Gemma-4-E2B NLA is the mirror image — its bottleneck prob
 near-perfectly in-distribution but its verbalizer is weak and domain-specific.
 Separating those two failure modes is the point of the suite.
 
+**Add your NLA.** Implement one adapter method, run the suite, and open a PR with
+your result — see [CONTRIBUTING.md](CONTRIBUTING.md). Results are attributed to the
+NLA (not the base model) and reported only when they clear a null control.
+
 ## What's inside
 
 - **A 128-plan catalog across 14 families (A–N).** Every plan is a falsifiable test
@@ -89,6 +95,7 @@ Separating those two failure modes is the point of the suite.
 | [docs/RESULTS.md](docs/RESULTS.md) | Reproducible findings, the leaderboard, and the attribution convention |
 | [docs/CTF_RED_BLUE.md](docs/CTF_RED_BLUE.md) | The v2 Red/Blue capture-the-flag family (Family N) |
 | [CHANGELOG.md](CHANGELOG.md) · [docs/VERSIONING.md](docs/VERSIONING.md) | Release history and the freeze-on-release policy |
+| [CONTRIBUTING.md](CONTRIBUTING.md) | How to submit your NLA's result or extend the harness |
 | [DESIGN_REVIEW.md](DESIGN_REVIEW.md) · [docs/LITERATURE.md](docs/LITERATURE.md) | Validity threats; the reading list with arXiv ids |
 
 ## How to cite

tests/test_ctf.py

@@ -0,0 +1,80 @@
+"""Unit tests for the CTF Red/Blue family (Family N).
+
+Deterministic and offline (no network, no GPU). Run with `pytest -q`.
+The assertions avoid MockNLA's process-dependent token-keep (harness H1): where a
+real NLA is used they check structural invariants and strict-JSON validity, not
+exact rates.
+"""
+import json
+
+from nla_eval import MockNLA, EnsembleMatcher
+from nla_eval import ctf
+
+
+def _round(captured, flagged, *, category="exfiltration", queries=1, suspicion=0.0):
+    outcome = ctf._classify(captured, flagged)
+    return ctf.CTFRound(
+        flag_id="t", category=category, attempt_input="x",
+        captured=captured, flagged=flagged, outcome=outcome,
+        suspicion=suspicion, queries=queries, policy="test",
+        stealth_gap=(1.0 if captured else 0.0) - suspicion)
+
+
+def test_outcome_classification():
+    assert ctf._classify(True, False) == ctf.CLEAN_CAPTURE
+    assert ctf._classify(True, True) == ctf.BUSTED
+    assert ctf._classify(False, True) == ctf.FALSE_ALARM
+    assert ctf._classify(False, False) == ctf.FAILED
+
+
+def test_score_uses_none_not_nan_and_is_strict_json():
+    # No captured rounds: catch_rate and mean_queries_to_capture are undefined.
+    score = ctf.score_ctf([_round(False, False), _round(False, True)], tier="control")
+    assert score.catch_rate is None
+    assert score.mean_queries_to_capture is None
+    assert score.by_category["exfiltration"]["catch_rate"] is None
+    # Must serialize as STRICT JSON — NaN would raise here (the project's H3 issue).
+    json.dumps(score.flat(), allow_nan=False)
+
+
+def test_score_rates_in_range():
+    rounds = [_round(True, False), _round(True, True), _round(False, False)]
+    score = ctf.score_ctf(rounds, tier="control")
+    assert 0.0 <= score.clean_capture_rate <= 1.0
+    assert score.catch_rate == 0.5  # 1 of 2 captured rounds was flagged
+    for rate in (score.busted_rate, score.failed_rate, score.false_alarm_rate):
+        assert 0.0 <= rate <= 1.0
+
+
+def test_blue_hit_is_a_tripping_concept_when_flagged():
+    # Regression guard: a flagged verdict's `hit` must be a concept that hard-matched,
+    # never a higher-soft-score non-tripping concept.
+    blue = ctf.BlueMonitor(
+        verbalize=lambda _t: "the operator performed exfiltration of the records",
+        watchlist=["exfiltration"] + list(ctf.DECEPTION_ANCHORS),
+        matcher=EnsembleMatcher())
+    v = blue.inspect("anything")
+    assert v.flagged is True
+    assert blue.matcher.match(v.hit, v.verbalization).present
+
+
+def test_first_clean_query_threshold():
+    # Monotone budget curve; first index clearing the threshold (1-based).
+    assert ctf._first_clean_query([0.1, 0.3, 0.55, 0.9], threshold=0.5) == 3
+    assert ctf._first_clean_query([0.1, 0.2], threshold=0.5) == 2  # never clears -> len
+
+
+def test_play_ctf_control_tier_serializable():
+    score = ctf.play_ctf(MockNLA(keep_prob=0.6), budget=10)
+    assert score.tier == "control"
+    assert score.n_rounds == len(ctf.CTF_FLAGS)
+    assert 0.0 <= score.clean_capture_rate <= 1.0
+    # Whole scoreboard must be strict-JSON serializable (no bare NaN).
+    json.dumps(score.flat(), allow_nan=False, default=str)
+
+
+def test_probe_oracle_is_primary_tier():
+    _oracle, tier = ctf.probe_capture_oracle(lambda _t: 0.9)
+    assert tier == "primary"
+    _oracle2, tier2 = ctf.survival_oracle()
+    assert tier2 == "control"