Skip to content

v0.8.0 — Diagnostic Reports & CLI Renderer

Choose a tag to compare

View all tags
@Mathews-Tom Mathews-Tom released this 30 Apr 17:03
· 866 commits to main since this release
v0.8.0
ef08b86
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

This release transforms Attest reports from flat score dumps into structured failure diagnostics. Every failed assertion now carries a layer tag (1–8), the trace path it bound to, expected vs actual evidence, judge audit metadata (model, rubric, sample-score variance), and a suggested next action — visible in PR comments, pytest output, and vitest output alike.

Engine

  • Hierarchical Markdown report: summary header (counts, cost, P50/P95 latency), failure section with per-assertion detail blocks, and a per-layer breakdown.
  • JSON report v2 with report_version: 2, per-layer indices, latency percentiles, and a simulated flag. The legacy v1 emitter is preserved behind ReportOptions{Version: ReportVersionV1} for one minor release.
  • Diagnostic fields on AssertionResult: Layer, Type, TraceNodePath, Expected, Actual, SuggestedAction, plus a new JudgeMetadata struct (model, rubric, prompt hash, sample scores, mean, stddev, high-variance flag).

Python SDK (attest-ai 0.8.0)

  • Mirror of the new diagnostic fields and JudgeMetadata on AssertionResult with backwards-compatible round-trip for legacy v1 payloads.
  • New attest.cli.diagnostics module — render_assertion_failure, render_diagnostics, render_summary with optional ANSI colour.
  • AgentResult.render_diagnostics() for the canonical pytest pattern: assert result.passed, result.render_diagnostics().
  • pytest plugin enriched: pytest_terminal_summary auto-renders the diagnostic block for failed Attest runs; --attest-diagnostics flag also enumerates passing assertions.

TypeScript SDKs (@attest-ai/core 0.8.0, @attest-ai/vitest 0.8.0)

  • JudgeMetadata interface and the new optional fields on AssertionResult.
  • New diagnostics.ts module exporting renderAssertionFailure, renderDiagnostics, renderSummary with ANSI colour support.
  • AttestCostReporter (vitest) renders the diagnostic block for failed runs in addition to the cost summary.

Dependency updates

  • vitest 4.1.4 → 4.1.5
  • modernc.org/sqlite 1.48.2 → 1.50.0
  • github.com/yalue/onnxruntime_go 1.27.0 → 1.28.0

Backwards compatibility

All new AssertionResult fields are JSON-omitempty, Python-default-None, TypeScript-optional. v0.7.x clients see no change. The JSONReport v1 schema and golden-file tests are preserved.

Engine binaries

Six platforms — Darwin/Linux/Windows × amd64/arm64 — with SHA-256 checksums in checksums-sha256.txt.

What's next

Phase 4 of the roadmap: OpenTelemetry GenAI parity (current span name set, agent-hierarchy attributes, OTel-export round-trip).

Assets 9
Loading