spine-probes: support thinking models in run_probes and judge#9
Merged
TheTom merged 1 commit intoJul 26, 2026
Merged
Conversation
Blackwellboy
force-pushed
the
spine-probes-thinking-model-fixes
branch
from
July 26, 2026 11:58
fcf509e to
97cdfed
Compare
run_probes.py: read thinking from message.reasoning as well as reasoning_content. Lanes exposing the former logged 0 in the reasoning column, which is the signal the README tells people to watch. judge.py: suppress thinking on the judge. max_tokens 8 is consumed by the reasoning block on a thinking model, so every verdict scored UNPARSED.
TheTom
added a commit
that referenced
this pull request
Jul 26, 2026
Generalizing the fix in #9. Some OpenAI-compatible servers expose thinking on 'reasoning' rather than 'reasoning_content', and one vLLM lane has no 'reasoning_content' key at all, so the probe runner read zero reasoning on all 42 rows: exactly the false signal the README tells people to watch for. This is the same confusion that produced two wrong conclusions earlier in this guide's history, so the verify block now tells readers to log both keys plus raw content, and says plainly that an empty field can mean wrong-key rather than did-not-reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two one-line fixes from #8, both for running spine-probes against a thinking model. Scope kept to the mechanical bugs; the third finding in that issue (the heuristic missing silent folds) is a judgement call about the classifier and the README, so it is not touched here.
run_probes.py: read thinking fromreasoningas well asreasoning_content. Our vLLM Qwen NVFP4 lane has noreasoning_contentkey at all, so the reasoning column read 0 on all 42 rows, which is the signal your README tells people to watch. Post-patch the same run logs a mean of 5,920 reasoning chars per response.judge.py: suppress thinking on the judge.max_tokens: 8is entirely consumed by the reasoning block on a thinking model, content comes back empty, and every verdict scores UNPARSED (42/42 for us). Turning thinking off is cheaper than raising the budget, and the judge only needs to emit one word. Post-patch, 0 UNPARSED.Both are additive fallbacks, so behaviour on servers that already populate
reasoning_content, and on non-thinking judges, is unchanged.check_guides.pypasses.Full transcripts, the deliberately-kept UNPARSED run, and the runner carried with these same two patches plus SHA256SUMS: https://github.com/Blackwellboy/laguna-s21-lab/tree/main/spine-probes
Drafted with AI assistance (Claude); reviewed and submitted by @Blackwellboy.
Happy to adjust scope or drop this if you would rather fold it your own way.