Skip to content

attacklm-dataset v0.9.0 — first real defensive audit

Latest
Latest

Choose a tag to compare

View all tags
@Veedubin Veedubin released this 16 Jul 06:11
· 4 commits to main since this release
v0.9.0
0e2e08a

Highlights

  • First real defensive audit on AttackLM/uncensored (Qwen2.5-Coder-14B-Instruct-uncensored, non-finetuned). 51 records (17 per source × 3 sources).
  • MIA Track 1 calibration fix validated: threshold = 5th percentile of membership scores (default --mia-threshold-mode percentile --mia-percentile 5) → 3/51 flagged (5.9%) instead of the 50%-by-construction median threshold of the 2026-07-07 pilot.
  • 0 exact matches across all 3 sources. Per-source NLL/pt: atomic-red-team 1.63, metasploit-framework 2.62, sigma-hq 2.53.
  • Per-record evidence chain preserved (prompt_text, best_reconstruction, suffix_text, per_token NLL, membership score) per the v0.4.1 audit bug fix.

Install

The attacklm-dataset.tar.gz attached below is the canonical tarball used by attacklm init (via the AttackLM package). The versioned attacklm-dataset-v0.9.0.tar.gz is identical content.

pip install attacklm
attacklm --init
attacklm init  # downloads the tarball
attacklm audit --help

Methodology

  • Carlini 2021 prefix-completion extraction probe, K=5, max_new_tokens=128
  • MUSE 2023 per-token MIA scoring (the --mia-method per_token default)
  • Threshold calibration: 5th percentile of probed scores (per docs/MIA_THRESHOLD_CALIBRATION.md Track 1)
  • Full audit results: data/audit/2026-07-16-defensive-v1/ (chmod 0700 parent + 0600 on raw records)

Data layout

24,652 records across 18 source directories in per-source layout (data/datasets/buckets/sources/<source>/<bucket>/<tactic>/data.jsonl). No restricted sources (RTA, infection_monkey, BPL) included in the public tarball. No new sources since v0.8.0.

Per-record provenance

Every record carries base provenance (source, source_uri, license, license_uri, rights_contact) plus license-specific fields for BSD-3-Clause (Metasploit) and DRL 1.1 (Sigma).

History

  • v0.8.0 — held-out NLL evaluation suite (Sprint 4, MAI-Thinking-1 §2.3)
  • v0.7.0 — memorization-aware epoch capping (Sprint 3, MAI-Thinking-1 §2.5.4)
  • v0.6.0 — 20-gram fuzzy decontamination (Sprint 2, MAI-Thinking-1 §2.3.1 + §2.4.3)
  • v0.5.0 — closed-loop audit + LiRA (Sprint 1, MAI-Thinking-1 §5.2)
  • v0.4.3 — scripts/score_shadow.py + --mia-method offline (LiRA workflow closure)
  • v0.4.2 — 5 audit-harness bug fixes (3 MUST-FIX + 2 quality)
  • v0.4.1 — per-source fix for --max-records
  • v0.3.1 — per-source dataset layout, MIA threshold Track 1
Assets 4
Loading