Skip to content

Skillware-Bench v0.3.2

Choose a tag to compare

View all tags
@rosspeili rosspeili released this 28 Oct 09:11
· 5 commits to main since this release
v0.3.2
ca68939

Executive Summary

We are proud to announce Skillware-Bench v0.3.2, hardening the evaluation platform with zero-cost CLI cryptographic report verification, tracked canonical execution telemetry across all 23 benchmarked models, formalized contribution protocols for both human engineers and autonomous logical systems, rigorous AURA Harness provenance grounding, and alignment with Skillware 0.6.0.

v0.3.2 eliminates trust assumptions in LLM evaluation by making every turn, token count, latency metric, and scoring verdict across 736 live experimental conditions independently auditable and tamper-evident using cryptographic SHA-256 receipts anchored to the AURA AuditSpine.

What's New in v0.3.2

1. Zero-Cost CLI Report Verification (skillware-bench verify)

  • Instant Verification: Researchers and auditors can cryptographically verify any published benchmark report JSON in under 1 second without calling external APIs, spending tokens, or requiring network access: 
    skillware-bench verify benchmarks/reports/google_gemma_4_31b_it.json
  • Tamper-Evident Detection: Independently recomputes the canonical SHA-256 digest across the JSON payload. Exploiting the cryptographic avalanche effect, altering even a single byte (e.g. flipping a verdict from 13/16 to 16/16 or modifying token counts) immediately triggers HASH_MISMATCH with exit code 1.

2. Tracked Canonical 23-Model Execution Reports (benchmarks/reports/)

  • Tracked Directly in Git: All 23 canonical 16-layer benchmark execution reports are tracked in benchmarks/reports/, preserving raw turn-by-turn telemetry, execution latencies, token consumption, disk diffs, and verdicts.
  • Master Receipts Index: Authored benchmarks/reports/README.md cataloging all 23 models, report files, run IDs, and anchored SHA-256 receipts alongside output verification examples.
  • Model Card Deep-Linking: Updated all 23 model cards (docs/results/models/*.md) with direct links to their canonical report JSON and CLI verification commands.

3. Rigorous AURA Harness Grounding & Provenance

  • Grounded in Authentic Architecture: Grounded all attestation documentation strictly in AURA Harness (Greek: αύρα / AVRA, PyPI: aura-harness), the runtime coat for agent loops developed by ARPA Hellenic Logical Systems.
  • Membrane Model: Accurately describes session-open ingress context normalization, host cavity execution (SkillwareHost), and egress policy boundary gating before tool dispatches leave the runtime cavity.
  • Live AuditSpine: Execution turns, schema preflight validations, mock API interactions, and environment mutations are recorded in real time as immutable AuraEvent records with sub-millisecond overhead (<50 µs per turn, <0.01% runtime tax).

4. Autonomous Logical System & Contributor Protocols (CONTRIBUTING.md)

  • Dual-Audience Protocol: Tailored for collaboration between human researchers and autonomous AI agents / reasoning models.
  • Skillware as the Core Evaluation Axis: Firmly establishes Skillware as the non-negotiable evaluation axis, strictly banning "markdown pseudo-skills" or unstructured prompt manuals in favor of authentic, typed, schema-validated Skillware skills.
  • The Ripple Effects Matrix: Provides an explicit mapping of cross-cutting file dependencies (crucibles, adapters, reports, exports, version bumps) to guarantee builds never fail on missed secondary updates.
  • The Five Crucible Invariants: Mandates Dual-Engine execution (--mock and --live), negative controls, ambiguous consent resistance, multi-vector state diff verification, and Port 0 hermetic mock servers.
  • Zero-Hotfix Preflight Checklist: Step-by-step verification protocol ensuring PRs pass CI on the first attempt.

5. Ecosystem Branding & Badging Alignment

  • Skillware 0.6.0 Alignment: Added soft pastel blue (#D4E4F1) badge and dedicated "Governed Execution via Skillware" section with official logo.
  • AURA Splash & ARPA Branding: Embedded left-aligned AURA splash logo and centered ARPA maintainer footer logo matching aura/README.md.
  • PyPI Trusted Publisher Configuration: Confirmed reserved name skillware-bench and automated OIDC trusted publishing workflow.

6. Hardened CI Workflows & Contribution Templates

  • CI Link & Report Auditing: .github/workflows/ci.yml now automatically verifies markdown link integrity (scripts/audit_md_links.py) and tests report receipt verification across all matrix Python versions (3.10-3.13) and clean virtualenv wheel smoke tests.
  • Template Alignment: Synchronized .github/PULL_REQUEST_TEMPLATE.md and .github/ISSUE_TEMPLATE/ with the Zero-Hotfix checklist, Five Crucible Invariants, and added a dedicated benchmark_report_submission.md template.

Quick Verification Example

Verify the authenticity of any benchmark report:

# Verify Google Gemma 4 31B IT report:
skillware-bench verify benchmarks/reports/google_gemma_4_31b_it.json

# Verify Meta Llama 4 Maverick report:
skillware-bench verify benchmarks/reports/meta_llama_4_maverick.json

Sample output:

Verifying AURA benchmark report receipt: benchmarks/reports/google_gemma_4_31b_it.json...

[PASS] Benchmark Execution Report Verification Succeeded
  * Run ID: tiered_ablation_20261018_152532
  * Model: google/gemma-4-31b-it
  * Conditions Evaluated: 32 (Vanilla: 13 passed, Skillware: 16 passed)
  * AURA Cryptographic Receipt (SHA-256): cf7c13b4e74273ec745f0afda62f26e0f0cf93105c49c5c0ef89a2439e285ae0
  * Attestation Engine: AURA AuditSpine / SHA-256
  * Integrity Status: AUTHENTIC / UNTAMPERED

Upgrade Guide from v0.3.1

To upgrade from v0.3.1:

pip install --upgrade skillware-bench

Verify your installation:

skillware-bench --version
skillware-bench check-env

License

MIT License. Copyright (c) 2026 ARPA Hellenic Logical Systems.

Assets 2
Loading