attacklm-dataset v0.9.0 — first real defensive audit

Highlights

• First real defensive audit on AttackLM/uncensored (Qwen2.5-Coder-14B-Instruct-uncensored, non-finetuned). 51 records (17 per source × 3 sources).
• MIA Track 1 calibration fix validated: threshold = 5th percentile of membership scores (default --mia-threshold-mode percentile --mia-percentile 5) → 3/51 flagged (5.9%) instead of the 50%-by-construction median threshold of the 2026-07-07 pilot.
• 0 exact matches across all 3 sources. Per-source NLL/pt: atomic-red-team 1.63, metasploit-framework 2.62, sigma-hq 2.53.
• Per-record evidence chain preserved (prompt_text, best_reconstruction, suffix_text, per_token NLL, membership score) per the v0.4.1 audit bug fix.

Install

The attacklm-dataset.tar.gz attached below is the canonical tarball used by attacklm init (via the AttackLM package). The versioned attacklm-dataset-v0.9.0.tar.gz is identical content.

pip install attacklm
attacklm --init
attacklm init  # downloads the tarball
attacklm audit --help

Methodology

• Carlini 2021 prefix-completion extraction probe, K=5, max_new_tokens=128
• MUSE 2023 per-token MIA scoring (the --mia-method per_token default)
• Threshold calibration: 5th percentile of probed scores (per docs/MIA_THRESHOLD_CALIBRATION.md Track 1)
• Full audit results: data/audit/2026-07-16-defensive-v1/ (chmod 0700 parent + 0600 on raw records)

Data layout

24,652 records across 18 source directories in per-source layout (data/datasets/buckets/sources/<source>/<bucket>/<tactic>/data.jsonl). No restricted sources (RTA, infection_monkey, BPL) included in the public tarball. No new sources since v0.8.0.

Per-record provenance

Every record carries base provenance (source, source_uri, license, license_uri, rights_contact) plus license-specific fields for BSD-3-Clause (Metasploit) and DRL 1.1 (Sigma).

History

• v0.8.0 — held-out NLL evaluation suite (Sprint 4, MAI-Thinking-1 §2.3)
• v0.7.0 — memorization-aware epoch capping (Sprint 3, MAI-Thinking-1 §2.5.4)
• v0.6.0 — 20-gram fuzzy decontamination (Sprint 2, MAI-Thinking-1 §2.3.1 + §2.4.3)
• v0.5.0 — closed-loop audit + LiRA (Sprint 1, MAI-Thinking-1 §5.2)
• v0.4.3 — scripts/score_shadow.py + --mia-method offline (LiRA workflow closure)
• v0.4.2 — 5 audit-harness bug fixes (3 MUST-FIX + 2 quality)
• v0.4.1 — per-source fix for --max-records
• v0.3.1 — per-source dataset layout, MIA threshold Track 1

attacklm-dataset v0.8.0 — held-out NLL evaluation

Sprint 4: NLL-based held-out eval suite. Implements MAI-Thinking-1 §2.3 weighted Eq-3 aggregate. New: split_held_out.py, held_out_nll.py, docs/HELD_OUT_NLL.md. 30 new tests.

attacklm-dataset v0.7.0 — memorization-aware epoch capping

Implements MAI-Thinking-1 §2.5.4 NLL<0.01 fraction memorization proxy. New scripts/memorization_report.py. Default behavior: report-only (does NOT auto-apply caps). Caps are a recommendation for review before use. 588 tests pass. 475 baseline + 113 new across v0.5.0-v0.7.0.

attacklm-dataset v0.6.0 — 20-gram fuzzy decontamination

Implements MAI-Thinking-1 §2.3.1 + §2.4.3 20-gram fuzzy dedup methodology. New scripts/decontam.py with MinHash LSH. Ships 3 eval-set fixtures under data/eval_sets/. Default behavior: report-only (no deletions); --quarantine-output writes matched records to a separate file for review. 551 tests pass.

attacklm-dataset v0.5.0 — closed-loop audit (--audit-iter)

Implements MAI-Thinking-1 §5.2 TAP-style closed-loop adversarial audit pattern. New --audit-iter flag on inversion_audit.py. Backward compatible: --audit-iter 1 is identical to v0.4.3 behavior. 511 tests pass. 475 existing + 36 new.

attacklm-dataset v0.4.3 — LiRA workflow closure

LiRA workflow closure

What's new

• scripts/score_shadow.py (NEW): Closes the LiRA workflow Step 2. The
  docstring of shadow_train.py referenced a non-existent script that users
  had to write themselves. Now it ships in-repo: load a HF model, score a
  JSONL of records, write shadow_{K}.json. Atomic write, --dry-run,
  duplicate-id handling, missing-assistant-turn handling.
• --mia-method offline: White-box MIA baseline that requires NO shadow
  models. Uses sample mean/std of audit-set NLL as the OUT distribution.
  For each record: z = (nll - μ_out) / σ_out. Lower z = more likely member.
  Default threshold: -1.5 (configurable). Field names: offline_z,
  offline_mu_out, offline_sigma_out, offline_flagged.
• compute_offline_z() added to inversion.scoring (reuses
  zscore_normalize() for the σ=0 guard).
• 15 new tests (10 in test_score_shadow.py, 5 in test_offline_mia.py).
  Total: 475 passing, 0 skipped.

Bug fix

• pyproject.toml drift fixed: was 0.4.0, everything else was 0.4.2.
  Caught by bumpversion --check. The every-commit-equals-a-new-tag rule
  paid for itself again.

Workflow recipes

Full LiRA (K=16 shadows, ~16h on RTX 4080 for 3B):

for k in $(seq 0 15); do
    attacklm train --output models/shadow_$k
    python scripts/score_shadow.py \
        --model models/shadow_$k \
        --records data/audit_set.jsonl \
        --output-dir losses/ \
        --shadow-index $k
done
python -m inversion.shadow_train \
    --loss-dir losses/ --output shadow_params.json
python scripts/inversion_audit.py \
    --model target \
    --mia-method lira \
    --lira-params shadow_params.json \
    --mia-threshold-mode lrt

Quick baseline (1 minute, no shadows):

python scripts/inversion_audit.py \
    --model target \
    --mia-method offline

Prior art

Carlini 2022 §3.2 (reference attack with z-normalization).
This is NOT Sablayrolles (early planning mis-claimed the provenance);
it's the loss-z baseline with sample std.

Files

• data tarball (3.8 MB, 18 sources, 24,652 records, 13.1x compression)
• See docs/LIRA.md for the full workflow

attacklm-dataset v0.4.2 — Docs sync + version bump

attacklm-dataset v0.4.2 — Docs sync + version bump

Cumulative head of the v0.4.2 release. Docs-only release; no code change from v0.4.1.

What changed

All changes are documentation:

• README.md: test count badge 106+ → 457+; "23 extract scripts" → 21 (post lib split); attack class descriptions updated to reflect the new "loss on assistant turn" behavior in v0.4.1; CLI ref table corrected; output structure now mentions the new prompt_text and best_reconstruction fields; testing section updated to list 10 test files.
• CHANGELOG.md: new [Unreleased] entry documenting the v0.4.1 bug fixes.
• docs/LIRA.md: --lira-k description corrected (was misleadingly stating "K is derived from the file" — the old broken behavior).
• docs/AUDIT_RUNNER.md: 20× speedup note from Bug #4 fix added.
• docs/MIA_THRESHOLD_CALIBRATION.md: line numbers refreshed (410-468 → 613-710 after the --mia-threshold-mode refactor and the v0.4.1 bugfix commits); post-bugfix note added explaining assistant-turn-only scoring.

Version bump

__version__ bumped from 0.1.0 (stale for several sessions) to 0.4.2 for introspection correctness:

>>> import attacklm_dataset
>>> attacklm_dataset.__version__
'0.4.2'

Tests

457 passing, 0 skipped. Same as v0.4.1 (no test changes).

Data

The data tarball is unchanged from v0.4.0 (and v0.4.1, since v0.4.1 was code-only).

Install

git clone https://github.com/Veedubin/attacklm-dataset.git
cd attacklm-dataset
git checkout v0.4.2
pip install -e ".[inversion]"

Or upgrade from v0.4.1:

cd attacklm-dataset
git fetch origin
git checkout v0.4.2
pip install -e ".[inversion]"

attacklm-dataset v0.4.1 — Audit-harness bug fixes

attacklm-dataset v0.4.1 — Audit-harness bug fixes

A paper-vs-code audit (memory 2094e7cf) found 5 bugs in scripts/inversion/. All 5 fixed in this release.

What changed

• Bug #1 (correctness, MUST FIX): scoring.py:162 score_record was using _extract_full_text() which leaked the prompt into the NLL and biased all --mia-method reference scores by prompt length. Switched to _extract_assistant_turn() per MUSE 2023 default.
• Bug #2 (crash + wrong metadata, MUST FIX): lira.py:305 save_shadow_params hardcoded lira_k=0 in the JSON and crashed with NameError on empty params. Added explicit lira_k parameter; shadow_train.py now passes lira_k=len(shadow_losses).
• Bug #3 (evidence chain, MUST FIX): probe.py ProbeResult didn't include prompt_text or best_reconstruction, so memorization findings couldn't be recovered from the audit artifact without re-running the probe. Added the two fields; the audit artifact (inversion_results.jsonl, chmod 0600) is now self-contained.
• Bug #4 (performance): probe.py:144 generate_completions was doing K sequential model.generate() calls. Switched to single generate() with num_return_sequences. ~20× speedup on typical 14B + 256-token setups.
• Bug #5 (code smell): lira.py LiRAScore had a dead alpha field "reserved for future use". Removed the field and the alpha parameter from score_lira. Cleaner contract.

Data

The data tarball is unchanged from v0.4.0. Only the audit-harness code in scripts/inversion/ changed.

Tests

14 new tests in tests/test_audit_bugfixes.py. 443 → 457 passing.

Audit code status

Production-ready for a real defensive run. The three MUST-FIX bugs are all fixed. The audit code is safe to use.

Install

git clone https://github.com/Veedubin/attacklm-dataset.git
cd attacklm-dataset
git checkout v0.4.1
pip install -e ".[inversion]"

Or upgrade an existing v0.4.0 install:

cd attacklm-dataset
git fetch origin
git checkout v0.4.1
pip install -e ".[inversion]"

Audit usage

python scripts/inversion_audit.py \
    --model /path/to/model \
    --dataset-root /path/to/data/datasets/buckets/sources \
    --attack all \
    --mia-method per_token

For LiRA:

python scripts/inversion_audit.py \
    --model /path/to/model \
    --dataset-root /path/to/data/datasets/buckets/sources \
    --attack mia \
    --mia-method lira \
    --lira-params shadow_params.json