Make the harvest classifier answer every candidate - #119
Merged
Conversation
The classifier was dropping most of its input in silence. Against one session's ten candidates the old prompt answered five and kept none, five runs running. The prompt never said every message needs an answer, and it opened by telling the model to skip boilerplate, which pushed it toward blanket refusal. max_tokens of 500 also truncated a ten-line answer mid-summary. Nothing noticed, because only KEEP lines were parsed: an all-SKIP reply, a truncated reply and a reply that never came were the same empty list. State the batch size, demand one line per message in order, reframe the criteria as what to keep and what to skip, and raise the token cap. Parse SKIP lines to count coverage and warn when a batch comes back partly answered. Same ten candidates now answer 10/10 and keep 9, 9, 9, 9, 8 across five runs. Unanswered candidates are still dropped rather than regex-kept: inventing keepers from an answer the model never gave would swap a silent loss for silent noise. Closes #117
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #117.
What was wrong
The classifier was dropping most of its input without saying so. Measured against one omp session's 10 pre-filtered candidates, the old prompt answered 5 of 10 and kept 0, five runs in a row. An earlier sample had produced 5, then 0, then 0 keeps, which read as randomness; the real shape is that the model answers about half the batch and the rest is discarded silently.
Two causes, both in
_llm_classify:max_tokenswas 500, which truncates a 10-line answer mid-summary.Nothing detected either. Only
KEEPlines were parsed, so an all-SKIPreply, a truncated reply and a reply that never arrived were the same empty list. An empty result was indistinguishable from a genuinely insight-free session, which is why a daily harvest could capture nothing and log nothing.The fix
The prompt now states the batch size, demands exactly one line per message in order, forbids merging and reordering, and reframes the criteria as what to keep and what to skip rather than an instruction to be sparing.
max_tokensgoes to 2000 so a full answer fits.SKIPlines are now parsed too, purely to count coverage. When a batch comes back partly answered the classifier logs how many candidates were answered and how many were dropped unclassified. Silence was the actual defect.Unanswered candidates are still dropped rather than regex-kept: inventing keepers from a reply the model did not give would trade a silent loss for silent noise.
Measured
Same 10 candidates, same model (
gemma4-e4b-text, temperature 0.1), five consecutive runs through the real_llm_classify:The single 8 answered all ten and skipped one on judgement; no coverage warning fired. That is ordinary model judgement, not the format failure this issue was about.
Gate
uv run ruff check src/ tests/exit 0.uv run pytest -qexit 0, 808 passed (+3). New tests pin the count instruction and the token floor, assert the partial-answer warning names both figures, and assert a fully answered all-SKIPreply logs nothing so the warning cannot cry wolf.