fix(ai-improve): tolerate prose-wrapped JSON from the model - #31
Merged
Conversation
Reproduced 3/3 on Data-Science-Toolkit: the selection call got a valid
response, but Haiku sometimes reasons in prose ("Looking at this
codebase, I need to identify...") instead of emitting bare JSON, despite
the existing "ONLY valid JSON" instruction. ask_json() only tried
json.loads() on the whole trimmed response, so any surrounding prose
made it discard an otherwise-usable answer and skip the run.
- Strengthen SELECT_PROMPT and WRITE_PROMPT: explicitly forbid
reasoning/commentary, since a caller parses the response
programmatically
- Add a fallback in ask_json(): if the whole response isn't valid JSON,
scan for the first balanced {...} object anywhere in it and parse
that instead of giving up
- Print more of the raw response on a genuine parse failure (800 -> 2000
chars) for easier debugging next time this happens
There was a problem hiding this comment.
Checked the fallback JSON extractor, the prompt escaping, and the downstream call sites. The balanced-brace scan only runs after a plain json.loads failure, so the happy path is unchanged; its one weakness (brace counting ignores JSON string context) fails closed — a premature } yields an unterminated string that json.loads rejects, returning None exactly as before, so it can never hand back a silently truncated file_content. Extracting a stray object from prose is still gated downstream by safe_target(), the required-keys check, and the SHRINK_FLOOR guard, and the new prompt text correctly doubles {{/}} for the .format() templates. Scope is one file and matches the stated intent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reproduced 3/3 in a row on Data-Science-Toolkit while verifying the pipeline: the selection call succeeded (context built, API responded), but Haiku sometimes reasons in prose ("Looking at this codebase, I need to identify...") instead of emitting bare JSON, despite
SELECT_PROMPT's existing "ONLY valid JSON" instruction.ask_json()only triedjson.loads()on the whole trimmed response, so any surrounding prose made it discard an otherwise-usable answer and skip the run entirely — safely (no bad file written), but silently.What changed
SELECT_PROMPTandWRITE_PROMPT: explicitly forbid reasoning/commentary, spell out that a caller parses the response programmaticallyask_json(): if the full response isn't valid JSON on its own, scan for the first balanced{...}object anywhere in it and parse that instead of giving upSame file, byte-identical across all 11 repos before this change — opening the same fix everywhere rather than just on the repo that surfaced it.