Skip to content

AgentEval v0.3.0 — Failure Memory, Replay and Minimization

Latest
Latest

Choose a tag to compare

View all tags
@nishanttyagi28 nishanttyagi28 released this 26 Jul 19:56
v0.3.0
06966d8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

AgentEval v0.3.0 — Failure Memory, Replay and Minimization

Commit: 06966d8
Package: nishanttyagi-agenteval==0.3.0
Import: agenteval

This release ships production Failure Memory (product V2 / V2.1): turn real agent failures into human-approved golden regressions that run on every future change.

Highlights

  • Production Failure Memory workflow — local SQLite store and agenteval memory CLI for ingest → classify → cluster → review → export → CI
  • Secure trace ingestion and redaction — JSONL ingest with best-effort redaction before persistence (not full DLP)
  • Deterministic failure clustering — taxonomy and fingerprint clustering without embeddings
  • Human-approved golden tests — explicit review/approve gates; no automatic promotions
  • Sync and async replay — deterministic local replay against adapters (FakeReplayAdapter for offline demos)
  • Deterministic failure minimization — delta-debug style payload reduction
  • Minimized golden export — export only human-approved minimized cases
  • Recurrence and resurfacing analytics — occurrence tracking, recurring/novel fingerprints, coverage signals
  • CI Failure Memory coverage policies — opt-in workflow and agenteval memory coverage gate
  • OTel-compatible JSON interchange — OpenTelemetry-shaped helpers for trace interchange
  • SQLite schema migrations — versioned path through schema v3 (V2.1 tables) with later concurrency hardening
  • Zero-network flagship demo — examples/failure_memory_demo_v21/run_demo.py
  • 1115-test verified main suite — 1115 passed, 1 skipped; Failure Memory suite 59 passed on release main

Notes

  • Failure Memory databases, traces, generated goldens, and temporary artifacts are not packaged.
  • See CHANGELOG.md and docs/failure-memory.md for full documentation.
  • PyPI publish uses GitHub Actions Trusted Publishing (OIDC) on this release event — no manual API token upload.

Artifacts

Attached wheel and sdist were built and verified from commit 06966d8 before release (SHA-256 hashes recorded in the release preparation process). The Publish to PyPI workflow rebuilds and publishes from this tag via trusted publishing.

Assets 5
Loading