Skip to content

AuditPoison v0.3.0 — Reproducible Model Evaluation Pipeline

Choose a tag to compare

View all tags
@anirudhnshandilya anirudhnshandilya released this 30 Jul 12:48
v0.3.0
a28491e

AuditPoison v0.3.0 introduces the first complete evaluation-ready version of the open-source benchmark for adversarial evidence attacks against LLM-based cybersecurity compliance auditors.

Included

  • 40 validated evidence bundles across 20 paired scenarios
  • Six adversarial evidence attack families
  • Six benign perturbation families
  • Balanced false-assurance evaluation boundary
  • Native Ollama model adapter
  • OpenAI-compatible API adapter
  • Provider-neutral command adapter
  • Strict structured-response parsing
  • Reproducible experiment manifests
  • Dataset, prompt, and prediction SHA-256 hashes
  • False Assurance Rate and paired attack metrics
  • Benign stability and citation-grounding metrics
  • Automatic Markdown and ACM-ready LaTeX result tables
  • Continuous integration across Python 3.10–3.13

Research status

This release is a development pilot intended to support ongoing research targeting ACM AsiaCCS 2027.

The current 40-bundle dataset should not yet be treated as statistically conclusive evidence about any model or provider. Future releases will expand the benchmark, evaluate additional models, introduce EvidenceShield defences, and add uncertainty analysis.

Verification

python -m pip install -e ".[dev]"
python scripts/validate_dataset.py
python -m pytest -q

Expected output:

AuditPoison validation PASSED: 40 bundles in 20 paired scenarios.
11 passed
Assets 2
Loading