Count parse failures in enhance telemetry - #196
Merged
Conversation
…ors']) In the non-agentic enhancer (ContextEnhancer.enhance_unit), the parse-failure else-branch stored _get_default_context(error=...) on the unit but did NOT increment self.stats['errors'] (the exception branch already does). So the [Enhance] Errors telemetry under-reported parse failures. The unit still carries the error and is analyzed with default context - this is a telemetry-honesty fix, not a coverage change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
July 26, 2026 20:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Count parse failures in enhance telemetry
Branch:
pr-2b-enhancer-telemetry(1 commit offmaster/d4caf8a) · Target:knostic/OpenAnt:masterDiff: 2 files, +42 (fix +4, test +38) · Not yet pushed — awaiting approval.
Problem
In the non-agentic enhancer (
utilities/context_enhancer.py::ContextEnhancer.enhance_unit), when the LLM response can't be parsed, theelsebranch stores_get_default_context(error={"type":"parse_error"})on the unit but does not incrementself.stats["errors"]— while the exception branch already does. So the[Enhance] Errors: Nsummary under-reports parse failures.This is telemetry-only: the unit still carries
llm_context.errorand is analyzed with default context (not dropped). Benign for coverage, wrong for the error tally.Fix
Add
self.stats["errors"] += 1to the parse-failureelsebranch, mirroring the exception branch. One line, purely additive; no verdict/coverage behavior change.Testing
tests/test_enhancer_parse_failure_counted.py(2 tests): a parse failure incrementserrors(and is not counted asunits_enhanced); a successful parse does not. Passes on thisd4caf8abase.Blast radius (checked)
ContextEnhancer.stats["errors"]is the enhance-stage counter (surfaced in the[Enhance] Errorsline). It is a different counter from the analysis-stagemetrics["errors"]thatcore/reporter.py:442reads ("units_analyzed": total_units - metrics.get("errors", 0), at thed4caf8abase) — so this fix does not changeunits_analyzed.Provenance (direct re-derivation — NOT the KB grade-A record)
This fix stands on a direct re-derivation at the PR base
d4caf8a: the parse-failureelsebranch stores_get_default_context(error={"type":"parse_error"})but does notself.stats["errors"] += 1, while theadjacent
exceptbranch does — verified atd4caf8a, and the guard test fails on the pre-fixd4caf8asource.Do NOT cite the KB
enhancer-failed-context-nograde-A record as corroboration (per fable review): thatrecord describes a different symptom — "failed enhance stored with no error key" — which is the
c81c0e1state and is already fixed at
d4caf8a(the else-branch there carrieserror={parse_error}). Its grade row'sstated basis ("byte-identical blob at c81c0e1") is also false —
context_enhancer.pyblobs differ (f90d29b≠e5365606).So the grade-A record neither matches this PR's target nor transfers by identity. (Both are KB-provenance defects
flagged for feedback in
KB-STUDY.md.) This PR's narrower counter-defect is real and residual atd4caf8a.Rollback
Git-revert-clean: one commit,
context_enhancer.py(+ its test), disjoint from all other changes. Revertingrestores the prior (under-counting) telemetry with zero coverage/verdict impact.