fix(verify): deterministic fix verification + developer dismissal handling - #22
Merged
Conversation
…per-file LLM calls The agent-based verifier used a sliding window of 3 turns, causing it to forget previously read files and loop endlessly — re-reading the same 5-6 files across 57 turns while never reaching the remaining 20+. This made verification unreliable (findings marked "still_present" without checking) and expensive (836K tokens / $1.78). Switch to the deterministic fix_verifier which makes one LLM call per file with all findings for that file. Key improvements: - Model: configurable, defaults to gpt-5-codex (was hardcoded gpt-4o-mini) - File content cap: 50K chars (was 8K, truncating large files) - Git diff: included in prompt so LLM sees exactly what changed - Prompt: judges whether underlying problem is resolved, not exact match - Path fallback: glob-based file resolution when ADO paths don't match - Usage tracking: token counts and cost reported in findings.json - Graceful diff failure: when git diff fails (shallow clone), verifies all files via LLM instead of marking everything "not modified" Result: 77K tokens / $0.19 (10x cheaper), all 31 findings verified, 10 correctly identified as fixed vs 0 before.
When a developer replies to a CodeHawk finding with a technical reason (e.g., "spreading undefined is valid JS", "sequential uploads are intentional"), the fix verifier now evaluates the dismissal using an LLM call with the full thread conversation and code context. Key changes: - fetch_pr_comments_activity: add get_developer_replies() returning full conversation threads labeled [CodeHawk] / [Developer Name] - fix_verifier: post-processing step evaluates still_present findings that have developer replies; accepted dismissals get status="dismissed" with 0 penalty (same as fixed) - Prompt includes .codereview.md for project context (versions, conventions) - Collaborative tone: benefit of the doubt for plausible technical reasons; reject only bare "Invalid" or provably wrong claims - Token usage from dismissal LLM calls tracked in total cost - JSON parser handles single-object responses (not just arrays) - post_findings: displays dismissed findings with resolution rate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Fix verification overhaul (
fix_verifier.py,batch_review_job.py)Developer dismissal (
fix_verifier.py,fetch_pr_comments_activity.py).codereview.mdproject contextScoring & display (
post_findings.py,review_models.py)"dismissed"status alongside fixed/still_present/not_relevantTest plan
🤖 Generated with Claude Code