Skip to content

Make the harvest classifier answer every candidate - #119

Merged
raphasouthall merged 1 commit into
mainfrom
fix/117-classify-determinism
Aug 25, 2026
Merged

Make the harvest classifier answer every candidate#119
raphasouthall merged 1 commit into
mainfrom
fix/117-classify-determinism

Conversation

@raphasouthall

Copy link
Copy Markdown
Owner

Closes #117.

What was wrong

The classifier was dropping most of its input without saying so. Measured against one omp session's 10 pre-filtered candidates, the old prompt answered 5 of 10 and kept 0, five runs in a row. An earlier sample had produced 5, then 0, then 0 keeps, which read as randomness; the real shape is that the model answers about half the batch and the rest is discarded silently.

Two causes, both in _llm_classify:

  • The prompt led with the keep/skip criteria and offered the two line shapes as alternatives, never stating that every message needs an answer. It also opened with "Skip boilerplate, status updates, and routine tool output", which pushed the model toward blanket refusal — a shortened prompt without that line kept 7 of the same 10.
  • max_tokens was 500, which truncates a 10-line answer mid-summary.

Nothing detected either. Only KEEP lines were parsed, so an all-SKIP reply, a truncated reply and a reply that never arrived were the same empty list. An empty result was indistinguishable from a genuinely insight-free session, which is why a daily harvest could capture nothing and log nothing.

The fix

The prompt now states the batch size, demands exactly one line per message in order, forbids merging and reordering, and reframes the criteria as what to keep and what to skip rather than an instruction to be sparing. max_tokens goes to 2000 so a full answer fits.

SKIP lines are now parsed too, purely to count coverage. When a batch comes back partly answered the classifier logs how many candidates were answered and how many were dropped unclassified. Silence was the actual defect.

Unanswered candidates are still dropped rather than regex-kept: inventing keepers from a reply the model did not give would trade a silent loss for silent noise.

Measured

Same 10 candidates, same model (gemma4-e4b-text, temperature 0.1), five consecutive runs through the real _llm_classify:

before after
answered 5 / 10 10 / 10
kept 0, 0, 0, 0, 0 9, 9, 9, 9, 8

The single 8 answered all ten and skipped one on judgement; no coverage warning fired. That is ordinary model judgement, not the format failure this issue was about.

Gate

uv run ruff check src/ tests/ exit 0. uv run pytest -q exit 0, 808 passed (+3). New tests pin the count instruction and the token floor, assert the partial-answer warning names both figures, and assert a fully answered all-SKIP reply logs nothing so the warning cannot cry wolf.

The classifier was dropping most of its input in silence. Against one
session's ten candidates the old prompt answered five and kept none, five
runs running.

The prompt never said every message needs an answer, and it opened by
telling the model to skip boilerplate, which pushed it toward blanket
refusal. max_tokens of 500 also truncated a ten-line answer mid-summary.
Nothing noticed, because only KEEP lines were parsed: an all-SKIP reply, a
truncated reply and a reply that never came were the same empty list.

State the batch size, demand one line per message in order, reframe the
criteria as what to keep and what to skip, and raise the token cap. Parse
SKIP lines to count coverage and warn when a batch comes back partly
answered.

Same ten candidates now answer 10/10 and keep 9, 9, 9, 9, 8 across five
runs. Unanswered candidates are still dropped rather than regex-kept:
inventing keepers from an answer the model never gave would swap a silent
loss for silent noise.

Closes #117
@raphasouthall
raphasouthall merged commit 173abcd into main Aug 25, 2026
5 checks passed
@raphasouthall
raphasouthall deleted the fix/117-classify-determinism branch August 25, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harvest LLM classification is nondeterministic on identical input

1 participant