v0.2.2
A focused fix to the model-validation eval.
Fixed
-
The off-topic check no longer false-fails strong models. The "refuses off-topic questions" check requires a real retrieval tool call, but the production system prompt both names the wiki's subject (through its worked example) and explicitly permits declining obvious trivia without searching — so a capable model like
gpt-4ocorrectly refused "what is the capital of France?" with zero tool calls and was intermittently marked as failing.The off-topic check now runs under a domain-blind, strict-search prompt: with the subject hidden and the decline-without-searching shortcut removed, the model must retrieve before it may decline, so the tool-call requirement is meaningful and non-flaky. The two citation checks still run under the real production prompt (whose worked example is what makes citation reliable). Verified:
gpt-4onow passes 3/3 across runs, with 1–2 tool calls on the off-topic refusal.
Full changelog: https://github.com/Clod/llmwiki-marimo/blob/v0.2.2/CHANGELOG.md