docs(evaluation): Error localization (concept)#720
Merged
khushalsonawat merged 7 commits intoJul 10, 2026
Conversation
khushalsonawat
requested changes
Jul 9, 2026
- drop About heading and --- dividers, open at a real first H2 - bold error localization on first mention, link judge model/eval template/Observe - drop the fake configured-threshold framing; state the real pass/fail vs low-score trigger - soften code-type/non-scorable rows to what's grounded, not the private ee engine's internals - compress How to enable it to the checkbox + the real OR-of-two-flags inheritance - call it the Error Localization checkbox, not the API field name - reword Review/Where results appear to the confirmed surfaces (trace detail eval view, dataset row drawer, Playground run tests) - fix stale link to guides/running-evaluations, cut the backward Understanding-evaluation link - Next steps -> Keep exploring, as cards TH-6675
…e-agi/docs into docs/eval-error-localization
…ew (error localization)
…s/eval-error-localization
…to docs/eval-error-localization
…model rename, drop false skip condition) Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New Evaluation concept page: Error localization — what it is, when it runs, where results appear, and how to enable it.
Explains that error localization runs a deeper analysis on a failed eval result to pinpoint which input field drove the failure, returning
error_analysisandselected_input_key.Grounding (verified against the product repo)
model_hub/services/error_localizer_service.pyerror_analysis+selected_input_key—model_hub/models/error_localizer_model.pyerror_localizer_enabledon the eval template, inherited by each run —model_hub/models/evals_metric.py:189,573Checks
astro devdocs/evaluation-revamp; title matches the sidebar labelTH-6675