Skillware-Bench v0.3.3

Executive Summary

We are pleased to announce Skillware-Bench v0.3.3, bringing key empirical publication exhibits directly into the primary root documentation, standardizing repository citation infrastructure with the permanent Zenodo Concept DOI, refining ecosystem badge aesthetics to match our pastel design system, and deepening the foundational Skillware architecture specification.

v0.3.3 enhances repository visual transparency for visitors and researchers by embedding the critical empirical proof points: run-to-run determinism, cross-model pass rate uplift trajectories, granular per-call token ledger decompositions, and tool call survival flows.

---

What's New in v0.3.3

1. Key Publication Figures in Root Documentation (README.md)

Four high-impact 3x-Retina publication figures have been integrated directly into README.md under Key Empirical Findings with detailed context and empirical captions:

• Figure 14 — Run-to-Run Determinism & Variance Contraction:
  • Highlights variance contraction across 40 seeded replays under identical prompts for an 8B compact model.
  • Contrasts unassisted Vanilla execution scattering across an erratic score distribution (mean 61.0, $\sigma = 4.1$) against Skillware's 5.1× tighter dispersion (mean 88.2, $\sigma = 0.8$), proving enterprise release gating predictability.
• Figure 18 — Equip-Any-Model Pass Rate Uplift Trajectories:
  • Visualizes trajectory lifts across 14 representative models, demonstrating how compact sovereign models achieve capability parity with unassisted frontier LLMs (e.g. Meta Llama 4 Maverick jumping from 37.5% to 93.8% across 16 tasks).
• Figure 20 — Token Ledger Breakdown per Tool Call (Office Mail):
  • Decomposes token expenditure per tool call during binary attachment extraction, demonstrating how Skillware preserves core reasoning (1,340 tokens) while excising documentation down to typed schemas (−88%) and eliminating error repair loops entirely (−100%), reducing per-call token tax by 70.6% (1,620 vs 5,520 tokens).
• Figure 33 — 368-Layer Benchmark Call Survival Flow:
  • Details tool invocation survival normalized across 1,000 emitted calls per arm. Illustrates how Skillware's preflight typing delivers 94.1% clean first-pass calls (vs 71.2% in Vanilla) with only 7 terminal aborts (vs 57 in Vanilla).

2. Standardized Zenodo Concept DOI Badge & Citation Infrastructure

• Permanent Concept DOI: Minted and permanently linked the Zenodo Concept DOI 10.5281/zenodo.23725543, ensuring all future benchmark releases resolve to a persistent, citable scientific record.
• Shields.io Design Alignment: Standardized the Zenodo badge to match the repository's flat-square design system with #007ec6 blue styling (DOI | 10.5281/zenodo.23725543).
• Metadata Synchronization: Synchronized the permanent Concept DOI across README.md, CITATION.cff, and pyproject.toml ([project.urls].DOI).
• Complete BibTeX Citation: Added ready-to-copy BibTeX block in README.md citing v0.3.3.

3. Streamlined Ecosystem Navigation & Badges

• AURA Versioned Badge: Updated attestation badge to AURA | 0.3.6 in pastel peach (#ffdac1), positioned in second place immediately following Skillware | 0.6.0 (#D4E4F1).
• Cleaned Header Navigation: Streamlined the top navigation links row in README.md by removing external project links, keeping the table of contents focused on benchmark sections.

4. Grounded Skillware Framework Architecture

• 5-Layer Skill Anatomy: Updated the Skillware section in README.md to thoroughly explain Skillware's core philosophy (decoupling capability from intelligence) and five integrated bundle layers:
  1. Contract (manifest.yaml): Strict constitutional safety boundaries, dry-run guarantees, and typed I/O schemas.
  2. Effect (skill.py): Deterministic Python handlers executing verified actions directly.
  3. Directive (instructions.md): Structured cognitive guidance and system instructions adapted to host LLMs.
  4. Assurance (test_skill.py): Offline verification suites ensuring effects honor contracts before registry admission.
  5. Interface (card.json): Universal tool adapters standardized across Gemini, Claude, OpenAI, Ollama, and OpenAI-compatible runtimes.

---

Upgrade & Verification

Upgrade via pip:

pip install --upgrade skillware-bench==0.3.3

Verify package version and CLI environment:

skillware-bench --version
skillware-bench check-env

Cryptographically verify any model execution report without API keys:

skillware-bench verify benchmarks/reports/google_gemma_4_31b_it.json

Skillware-Bench v0.3.2

Executive Summary

We are proud to announce Skillware-Bench v0.3.2, hardening the evaluation platform with zero-cost CLI cryptographic report verification, tracked canonical execution telemetry across all 23 benchmarked models, formalized contribution protocols for both human engineers and autonomous logical systems, rigorous AURA Harness provenance grounding, and alignment with Skillware 0.6.0.

v0.3.2 eliminates trust assumptions in LLM evaluation by making every turn, token count, latency metric, and scoring verdict across 736 live experimental conditions independently auditable and tamper-evident using cryptographic SHA-256 receipts anchored to the AURA AuditSpine.

---

What's New in v0.3.2

1. Zero-Cost CLI Report Verification (skillware-bench verify)

• Instant Verification: Researchers and auditors can cryptographically verify any published benchmark report JSON in under 1 second without calling external APIs, spending tokens, or requiring network access:

  skillware-bench verify benchmarks/reports/google_gemma_4_31b_it.json

• Tamper-Evident Detection: Independently recomputes the canonical SHA-256 digest across the JSON payload. Exploiting the cryptographic avalanche effect, altering even a single byte (e.g. flipping a verdict from 13/16 to 16/16 or modifying token counts) immediately triggers HASH_MISMATCH with exit code 1.

2. Tracked Canonical 23-Model Execution Reports (benchmarks/reports/)

• Tracked Directly in Git: All 23 canonical 16-layer benchmark execution reports are tracked in benchmarks/reports/, preserving raw turn-by-turn telemetry, execution latencies, token consumption, disk diffs, and verdicts.
• Master Receipts Index: Authored benchmarks/reports/README.md cataloging all 23 models, report files, run IDs, and anchored SHA-256 receipts alongside output verification examples.
• Model Card Deep-Linking: Updated all 23 model cards (docs/results/models/*.md) with direct links to their canonical report JSON and CLI verification commands.

3. Rigorous AURA Harness Grounding & Provenance

• Grounded in Authentic Architecture: Grounded all attestation documentation strictly in AURA Harness (Greek: αύρα / AVRA, PyPI: aura-harness), the runtime coat for agent loops developed by ARPA Hellenic Logical Systems.
• Membrane Model: Accurately describes session-open ingress context normalization, host cavity execution (SkillwareHost), and egress policy boundary gating before tool dispatches leave the runtime cavity.
• Live AuditSpine: Execution turns, schema preflight validations, mock API interactions, and environment mutations are recorded in real time as immutable AuraEvent records with sub-millisecond overhead (<50 µs per turn, <0.01% runtime tax).

4. Autonomous Logical System & Contributor Protocols (CONTRIBUTING.md)

• Dual-Audience Protocol: Tailored for collaboration between human researchers and autonomous AI agents / reasoning models.
• Skillware as the Core Evaluation Axis: Firmly establishes Skillware as the non-negotiable evaluation axis, strictly banning "markdown pseudo-skills" or unstructured prompt manuals in favor of authentic, typed, schema-validated Skillware skills.
• The Ripple Effects Matrix: Provides an explicit mapping of cross-cutting file dependencies (crucibles, adapters, reports, exports, version bumps) to guarantee builds never fail on missed secondary updates.
• The Five Crucible Invariants: Mandates Dual-Engine execution (--mock and --live), negative controls, ambiguous consent resistance, multi-vector state diff verification, and Port 0 hermetic mock servers.
• Zero-Hotfix Preflight Checklist: Step-by-step verification protocol ensuring PRs pass CI on the first attempt.

5. Ecosystem Branding & Badging Alignment

• Skillware 0.6.0 Alignment: Added soft pastel blue (#D4E4F1) badge and dedicated "Governed Execution via Skillware" section with official logo.
• AURA Splash & ARPA Branding: Embedded left-aligned AURA splash logo and centered ARPA maintainer footer logo matching aura/README.md.
• PyPI Trusted Publisher Configuration: Confirmed reserved name skillware-bench and automated OIDC trusted publishing workflow.

6. Hardened CI Workflows & Contribution Templates

• CI Link & Report Auditing: .github/workflows/ci.yml now automatically verifies markdown link integrity (scripts/audit_md_links.py) and tests report receipt verification across all matrix Python versions (3.10-3.13) and clean virtualenv wheel smoke tests.
• Template Alignment: Synchronized .github/PULL_REQUEST_TEMPLATE.md and .github/ISSUE_TEMPLATE/ with the Zero-Hotfix checklist, Five Crucible Invariants, and added a dedicated benchmark_report_submission.md template.

---

Quick Verification Example

Verify the authenticity of any benchmark report:

# Verify Google Gemma 4 31B IT report:
skillware-bench verify benchmarks/reports/google_gemma_4_31b_it.json

# Verify Meta Llama 4 Maverick report:
skillware-bench verify benchmarks/reports/meta_llama_4_maverick.json

Sample output:

Verifying AURA benchmark report receipt: benchmarks/reports/google_gemma_4_31b_it.json...

[PASS] Benchmark Execution Report Verification Succeeded
  * Run ID: tiered_ablation_20261018_152532
  * Model: google/gemma-4-31b-it
  * Conditions Evaluated: 32 (Vanilla: 13 passed, Skillware: 16 passed)
  * AURA Cryptographic Receipt (SHA-256): cf7c13b4e74273ec745f0afda62f26e0f0cf93105c49c5c0ef89a2439e285ae0
  * Attestation Engine: AURA AuditSpine / SHA-256
  * Integrity Status: AUTHENTIC / UNTAMPERED

---

Upgrade Guide from v0.3.1

To upgrade from v0.3.1:

pip install --upgrade skillware-bench

Verify your installation:

skillware-bench --version
skillware-bench check-env

---

License

MIT License. Copyright (c) 2026 ARPA Hellenic Logical Systems.