AgentEval v0.3.0 — Failure Memory, Replay and Minimization

AgentEval v0.3.0 — Failure Memory, Replay and Minimization

Commit: 06966d8
Package: nishanttyagi-agenteval==0.3.0
Import: agenteval

This release ships production Failure Memory (product V2 / V2.1): turn real agent failures into human-approved golden regressions that run on every future change.

Highlights

• Production Failure Memory workflow — local SQLite store and agenteval memory CLI for ingest → classify → cluster → review → export → CI
• Secure trace ingestion and redaction — JSONL ingest with best-effort redaction before persistence (not full DLP)
• Deterministic failure clustering — taxonomy and fingerprint clustering without embeddings
• Human-approved golden tests — explicit review/approve gates; no automatic promotions
• Sync and async replay — deterministic local replay against adapters (FakeReplayAdapter for offline demos)
• Deterministic failure minimization — delta-debug style payload reduction
• Minimized golden export — export only human-approved minimized cases
• Recurrence and resurfacing analytics — occurrence tracking, recurring/novel fingerprints, coverage signals
• CI Failure Memory coverage policies — opt-in workflow and agenteval memory coverage gate
• OTel-compatible JSON interchange — OpenTelemetry-shaped helpers for trace interchange
• SQLite schema migrations — versioned path through schema v3 (V2.1 tables) with later concurrency hardening
• Zero-network flagship demo — examples/failure_memory_demo_v21/run_demo.py
• 1115-test verified main suite — 1115 passed, 1 skipped; Failure Memory suite 59 passed on release main

Notes

• Failure Memory databases, traces, generated goldens, and temporary artifacts are not packaged.
• See CHANGELOG.md and docs/failure-memory.md for full documentation.
• PyPI publish uses GitHub Actions Trusted Publishing (OIDC) on this release event — no manual API token upload.

Artifacts

Attached wheel and sdist were built and verified from commit 06966d8 before release (SHA-256 hashes recorded in the release preparation process). The Publish to PyPI workflow rebuilds and publishes from this tag via trusted publishing.

v0.2.0

SQL agent safety scanner (Tiers 1-5): scan, diff-runs, import, structural/policy/execution/session/semantic rules.

v0.1.0 — Initial release

What's Changed

• Build the real regression gate and standalone foundation by @nishanttyagi28 in #1
• Add reviewable adversarial generation and break-rate dashboard by @nishanttyagi28 in #3
• docs: strengthen AgentEval portfolio presentation by @nishanttyagi28 in #5
• feat(flakiness): add opt-in repeat-based consistency detection by @nishanttyagi28 in #6
• fix(provenance): resolve CI checkout SHA by @nishanttyagi28 in #7
• docs: add MIT license by @nishanttyagi28 in #8
• feat(trajectory): add observability-only step scoring by @nishanttyagi28 in #9
• feat(packaging): add installable CLI and trusted publishing by @nishanttyagi28 in #10

New Contributors

• @nishanttyagi28 made their first contribution in #1

Full Changelog: https://github.com/nishanttyagi28/agenteval/commits/v0.1.0