AuditPoison v0.6.1

Patch release fixing the anonymous reviewer-artifact builder:

• avoids the reserved PowerShell HOME variable;
• excludes the builder itself from reviewer artifacts;
• preserves UTF-8 TOML files without a BOM;
• stores the oracle key separately from the frozen holdout;
• strengthens identity-leak and reproducibility checks.

No benchmark data, defence semantics, model predictions, or reported experimental results changed.

Title: AuditPoison v0.6.0

AuditPoison v0.6.0

AuditPoison v0.6.0 is a submission-ready research software release focused on reproducibility and safe artifact separation.

Highlights

• EvidenceShield v0.2 deterministic predicate adjudication.
• Preservation of EvidenceShield v0.1 as a provenance-only ablation.
• Public development benchmark validation and regression tests.
• Clear separation between public software, anonymous reviewer artifacts, and private research records.
• Security, contribution, issue, pull-request, and reproducibility guidance.
• Local anonymous-artifact builder and identity-leak scan.

Deliberately excluded

This identifiable release does not contain:

• the blinded holdout oracle;
• unlock keys;
• local model prediction files;
• the private manuscript;
• author-identifying experimental paths;
• pre-unseal records containing local filesystem paths.

Those materials belong in the separately anonymized reviewer artifact or private research archive.

Validation

python scripts/validate_dataset.py
python -m pytest -q
git diff --check

docs/release_v0.5.0.md

ersion 0.5.0 removes final compliance authority from the language model.

EvidenceShield now:

• screens provenance, scope, environment, and assessment-period validity;
• quarantines unsupported authority claims and untrusted instructions;
• sanitizes instruction-bearing comments inside verified technical exports;
• evaluates explicit control predicates as satisfied, failed, or unresolved;
• assigns the final label deterministically;
• records the model’s original advisory label for ablation analysis;
• preserves EvidenceShield v0.1 for reproducing the provenance-only defence.

Deterministic verdict policy:

Any failed predicate        -> non_compliant
No failures, any unresolved -> insufficient_evidence
All predicates satisfied    -> compliant

Benchmark composition

Oracle label	Bundles
Compliant	20
Non-compliant	15
Insufficient evidence	5
Total	40

Attack families: instruction injection, authority spoofing, contradiction flooding, temporal rollback, scope substitution, and evidence omission.

Benign perturbations: formatting noise, evidence reordering, irrelevant context, duplicate support, semantic paraphrasing, and metadata noise.

Installation

git clone https://github.com/anirudhnshandilya/AuditPoison.git
cd AuditPoison
python -m venv .venv

Windows Command Prompt:

.venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Linux and macOS:

source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Validate

python scripts/validate_dataset.py
python -m pytest -q

Expected: 40 bundles, 20 pairs, and 31 tests passing.

Inspect deterministic predicates

python scripts/inspect_predicates.py --bundle-id AP-IA2-005-attacked

Run model experiments

Unshielded:

python scripts/run_model.py --adapter ollama --model YOUR_MODEL --defense none --output results/model_unshielded.jsonl

Historical EvidenceShield v0.1 ablation:

python scripts/run_model.py --adapter ollama --model YOUR_MODEL --defense evidenceshield-v0.1 --output results/model_v01.jsonl

EvidenceShield v0.2:

python scripts/run_model.py --adapter ollama --model YOUR_MODEL --defense evidenceshield-v0.2 --output results/model_v02.jsonl

--defense evidenceshield is an alias for evidenceshield-v0.2.

Important interpretation rule

The v0.2 predicate engine is explicitly designed for the frozen structured pilot. A 100% contract-test result on these 40 bundles demonstrates implementation consistency only. It must not be presented as benchmark generalisation or production effectiveness.

Core metrics

• accuracy and macro F1;
• False Assurance Rate;
• paired Attack Success Rate and robust accuracy;
• benign consistency and both-correct rate;
• citation precision, recall, and F1;
• attack-evidence detection and benign false-flag rate;
• Brier score and expected calibration error.

Repository layout

data/pilot/           Evidence bundles
docs/                 Threat model, protocols, and EvidenceShield designs
prompts/              Frozen auditor and analyst prompts
results/              Local metrics and paper tables
schema/               Evidence-bundle JSON Schema
scripts/              Validation, experiments, inspection, and reporting
src/auditpoison/      Python package, screening, and predicate engine
tests/                Integrity, leakage, reproducibility, and defence tests

Versions

• v0.1.0 — threat model and adversarial pilot
• v0.2.0 — balanced benchmark and evaluation pipeline
• v0.3.0 — real-model adapters and reproducibility manifests
• v0.4.0 — EvidenceShield v0.1 provenance firewall
• v0.5.0 — EvidenceShield v0.2 deterministic predicate adjudication

Licence and citation

AuditPoison is released under the MIT License. Citation metadata is provided in CITATION.cff.

AuditPoison v0.4.0 — EvidenceShield

This release adds EvidenceShield v0.1, a provenance-aware defence wrapper with:

• scope and assessment-period screening;
• instruction-like content quarantine;
• unsupported authority-claim detection;
• provenance and integrity labelling;
• model-visible trust annotations;
• a conservative positive-assurance gate;
• defence comparison tooling and regression tests.

The benchmark remains frozen at 40 bundles across 20 paired scenarios so the defence can be evaluated without changing the underlying pilot.

Benchmark composition

Oracle label	Bundles
Compliant	20
Non-compliant	15
Insufficient evidence	5
Total	40

Attack families: instruction injection, authority spoofing, contradiction flooding, temporal rollback, scope substitution, and evidence omission.

Benign perturbations: formatting noise, evidence reordering, irrelevant context, duplicate support, semantic paraphrasing, and metadata noise.

Installation

git clone https://github.com/anirudhnshandilya/AuditPoison.git
cd AuditPoison
python -m venv .venv

Windows Command Prompt:

.venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Linux and macOS:

source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Validate

python scripts/validate_dataset.py
python -m pytest -q

Expected: 40 bundles, 20 pairs, and 20 tests passing.

Inspect EvidenceShield

python scripts/inspect_defense.py --bundle-id AP-AC2-001-attacked

Compare unshielded and shielded smoke tests

python scripts/run_model.py --adapter keyword --output results/keyword_raw_predictions.jsonl
python scripts/evaluate_predictions.py results/keyword_raw_predictions.jsonl --output results/keyword_raw_metrics.json

python scripts/run_model.py --adapter keyword --defense evidenceshield --output results/keyword_shield_predictions.jsonl
python scripts/evaluate_predictions.py results/keyword_shield_predictions.jsonl --output results/keyword_shield_metrics.json

python scripts/compare_defenses.py results/keyword_raw_metrics.json results/keyword_shield_metrics.json
python scripts/build_paper_table.py results/keyword_raw_metrics.json results/keyword_shield_metrics.json

The keyword adapter is a deterministic software smoke test, not a research baseline.

Run a real local model

python scripts/run_model.py --adapter ollama --model YOUR_MODEL --limit 2 --output results/ollama_test.jsonl
python scripts/run_model.py --adapter ollama --model YOUR_MODEL --defense evidenceshield --output results/ollama_shield_predictions.jsonl

Run an OpenAI-compatible endpoint

Set AUDITPOISON_API_KEY only in the current shell, then run:

python scripts/run_model.py --adapter openai-compatible --model YOUR_MODEL --base-url YOUR_BASE_URL --defense evidenceshield --output results/hosted_shield_predictions.jsonl

AuditPoison never writes the API key to predictions or manifests.

Core metrics

• accuracy and macro F1;
• False Assurance Rate;
• paired Attack Success Rate and robust accuracy;
• benign consistency and both-correct rate;
• citation precision, recall, and F1;
• attack-evidence detection and benign false-flag rate;
• Brier score and expected calibration error.

Repository layout

data/pilot/           Evidence bundles
docs/                 Threat model, protocols, and EvidenceShield design
prompts/              Frozen auditor prompts
results/              Metrics and paper tables
schema/               Evidence-bundle JSON Schema
scripts/              Validation, experiments, comparison, and reporting
src/auditpoison/      Python package and defence implementation
tests/                Integrity, leakage, reproducibility, and defence tests

Versions

• v0.1.0 — threat model and adversarial pilot
• v0.2.0 — balanced benchmark and evaluation pipeline
• v0.3.0 — real-model adapters and reproducibility manifests
• v0.4.0 — EvidenceShield provenance firewall and assurance gate

Licence and citation

AuditPoison is released under the MIT License. Citation metadata is provided in CITATION.cff.

AuditPoison v0.3.0 — Reproducible Model Evaluation Pipeline

AuditPoison v0.3.0 introduces the first complete evaluation-ready version of the open-source benchmark for adversarial evidence attacks against LLM-based cybersecurity compliance auditors.

Included

• 40 validated evidence bundles across 20 paired scenarios
• Six adversarial evidence attack families
• Six benign perturbation families
• Balanced false-assurance evaluation boundary
• Native Ollama model adapter
• OpenAI-compatible API adapter
• Provider-neutral command adapter
• Strict structured-response parsing
• Reproducible experiment manifests
• Dataset, prompt, and prediction SHA-256 hashes
• False Assurance Rate and paired attack metrics
• Benign stability and citation-grounding metrics
• Automatic Markdown and ACM-ready LaTeX result tables
• Continuous integration across Python 3.10–3.13

Research status

This release is a development pilot intended to support ongoing research targeting ACM AsiaCCS 2027.

The current 40-bundle dataset should not yet be treated as statistically conclusive evidence about any model or provider. Future releases will expand the benchmark, evaluate additional models, introduce EvidenceShield defences, and add uncertainty analysis.

Verification

python -m pip install -e ".[dev]"
python scripts/validate_dataset.py
python -m pytest -q

Expected output:

AuditPoison validation PASSED: 40 bundles in 20 paired scenarios.
11 passed