attacklm-dataset v0.4.1 — Audit-harness bug fixes

attacklm-dataset v0.4.1 — Audit-harness bug fixes

A paper-vs-code audit (memory 2094e7cf) found 5 bugs in scripts/inversion/. All 5 fixed in this release.

What changed

• Bug #1 (correctness, MUST FIX): scoring.py:162 score_record was using _extract_full_text() which leaked the prompt into the NLL and biased all --mia-method reference scores by prompt length. Switched to _extract_assistant_turn() per MUSE 2023 default.
• Bug #2 (crash + wrong metadata, MUST FIX): lira.py:305 save_shadow_params hardcoded lira_k=0 in the JSON and crashed with NameError on empty params. Added explicit lira_k parameter; shadow_train.py now passes lira_k=len(shadow_losses).
• Bug #3 (evidence chain, MUST FIX): probe.py ProbeResult didn't include prompt_text or best_reconstruction, so memorization findings couldn't be recovered from the audit artifact without re-running the probe. Added the two fields; the audit artifact (inversion_results.jsonl, chmod 0600) is now self-contained.
• Bug #4 (performance): probe.py:144 generate_completions was doing K sequential model.generate() calls. Switched to single generate() with num_return_sequences. ~20× speedup on typical 14B + 256-token setups.
• Bug #5 (code smell): lira.py LiRAScore had a dead alpha field "reserved for future use". Removed the field and the alpha parameter from score_lira. Cleaner contract.

Data

The data tarball is unchanged from v0.4.0. Only the audit-harness code in scripts/inversion/ changed.

Tests

14 new tests in tests/test_audit_bugfixes.py. 443 → 457 passing.

Audit code status

Production-ready for a real defensive run. The three MUST-FIX bugs are all fixed. The audit code is safe to use.

Install

git clone https://github.com/Veedubin/attacklm-dataset.git
cd attacklm-dataset
git checkout v0.4.1
pip install -e ".[inversion]"

Or upgrade an existing v0.4.0 install:

cd attacklm-dataset
git fetch origin
git checkout v0.4.1
pip install -e ".[inversion]"

Audit usage

python scripts/inversion_audit.py \
    --model /path/to/model \
    --dataset-root /path/to/data/datasets/buckets/sources \
    --attack all \
    --mia-method per_token

For LiRA:

python scripts/inversion_audit.py \
    --model /path/to/model \
    --dataset-root /path/to/data/datasets/buckets/sources \
    --attack mia \
    --mia-method lira \
    --lira-params shadow_params.json