Skip to content

VerifyCraft v0.2.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 07 Nov 15:39
· 1 commit to main since this release
v0.2.0
197ffff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

VerifyCraft v0.2.0 Release Notes

VerifyCraft v0.2.0 introduces semantic, requirements-aware automated code review, universal model provider integration, source coordinate correctness, and an auditable evaluation benchmark with maximum-cardinality assignment.

This release extends VerifyCraft from test and hypothesis verification into full semantic code review, bridging pull request diffs, explicit requirements documents, and reviewer model outputs.

What's New in v0.2.0

1. Requirements-Aware Semantic Review Engine (verifycraft review)

  • Semantic Code Review Pipeline: Review changes between Git revisions against explicit requirement files (verifycraft review --base <ref> --head <ref> --requirements <path>).
  • Conventional Comments Schema: Formats candidate review findings into standard conventional comments labels (issue, suggestion, question, thought, nit) with decorators (blocking, security).
  • Exact Source Coordinate Preservation: Preserves original line coordinate mappings throughout Git context collection, preventing line-shift bugs between diff hunks and full file context.
  • Deterministic Semantic Anchor Validator: Validates that all candidate citations map directly to existent, verified lines in the target revisions.
  • Context Budget Invariants: Strict byte and token accounting preventing prompt truncation while ensuring deterministic analysis retains access to full source content.

2. Universal Model Provider Ecosystem

  • Universal Provider Adapters: First-class support for major foundation model families:
    • OpenAI (gpt-4o, gpt-4o-mini)
    • Anthropic (claude-3-5-sonnet, claude-3-5-haiku)
    • Google Gemini (gemini-1.5-pro, gemini-1.5-flash)
    • DeepSeek (deepseek-chat)
  • Enterprise & Local Inference:
    • AWS Bedrock runtime adapter for enterprise compliance.
    • Local inference support via OpenAI-compatible endpoints (Ollama, vLLM, LM Studio).
    • Model-free review mode for deterministic counterexample checks and existing finding ingestion.

3. Requirement Compliance & Ambiguity Classification

  • RFC 2119 Semantics: Distinguishes mandatory constraints (must, shall) from recommendations (should).
  • Defect vs Ambiguity Separation: Prevents unresolved specification questions or non-blocking observations from erroneously forcing not_satisfied. Unresolved specification precedence yields unclear.
  • Compensating Controls: Identifies surrounding and related files that mitigate apparent diff-level defects, correctly preserving appears_satisfied.

4. Empirical Review Benchmark & Maximum Bipartite Matching

  • Versioned Review Dataset: 16 real-world evaluation cases across dev and held_out splits with human-reviewed ground truth.
  • Genuine Maximum Bipartite Matching: Replaced greedy first-fit matching with deterministic Kuhn-Munkres maximum-cardinality assignment to adjudicate findings against expected defects.
  • Fail-Closed Release Gate: 5-stage release gate with multi-variant validation (Direct-Diff, Direct-Context, VerifyCraft, Replay), verifying per-case execution traces and token accounting.

Known Limitations in v0.2.0

The following two limitations are explicitly acknowledged for the v0.2.0 release:

  1. Heuristic Requirement Evidence Matching (_has_requirement_evidence):
    • _has_requirement_evidence() matches candidate findings to requirements using constraint token extraction, generic domain vocabulary filtering (auth, user, data), and cosmetic/documentation observation filters.
    • While effective at eliminating false positives from documentation nits while preserving genuine functional defects, this mechanism remains lexical and rule-based.
    • A structured semantic evidence graph (binding AST entities and call-graph traces) is scheduled for v0.3.0.
  2. Replay Mismatch on REQ-PERF-01 (dev_05_ambiguous_requirement_timeout):
    • In dev_05_ambiguous_requirement_timeout, the recorded DeepSeek Variant B evaluation reports raw status satisfied, causing offline replay to return appears_satisfied.
    • The human-reviewed ground truth specifies unclear because the specification omits timeout acceptance criteria.
    • This known model evaluation discrepancy is preserved in the benchmark dataset without artificial ground truth adjustments.

Installation

VerifyCraft 0.2.0 is available on PyPI:

# Using pip
pip install verifycraft==0.2.0

# Using uv
uv add verifycraft==0.2.0

Verify installation:

verifycraft --version
verifycraft review --help
Assets 4
Loading