Skip to content

Judge learner answers before generating the tutor move #45

Description

@rosshd

Parent: #41

Follow-up to: #42

Problem

The documented tutor loop says to classify and judge the learner answer, update learner state, select a move, and then generate the tutor response.

The current ask_topic flow generates the tutor response first and runs update_learning_metadata afterward.

As a result, the response cannot reliably use the current answer score, misconception, prerequisite gap, or difficulty tier.

The metadata extractor also sees newly generated tutor material, which can reassign the learner's attempt to a concept introduced after the answer.

The Phase 0 dogfood run demonstrated this when a return-value answer was recorded under the newly introduced parameters-and-arguments focus.

Goal

Make the structured answer judgment and learner-state transition authoritative inputs to the tutor move for the same turn.

Scope

  • Classify the learner message before answer judging so navigation and unrelated requests are not graded as answers.
  • Judge answers against the prior pending question and prior concept focus.
  • Apply the resulting score, status, misconception, gap, gaming signal, and tier before generating the tutor response.
  • Generate the response from the updated state and selected move.
  • Keep tutor-generated new material from changing which concept received the attempt.
  • Define safe behavior when the judge call fails without inventing mastery or losing the learner's answer.
  • Preserve local-first storage, event logging, streaming behavior where practical, and provider compatibility.

Acceptance criteria

  • A learner answer is judged against the prior pending question and prior focus before response generation.
  • The same-turn tutor prompt receives the current judgment, misconception or gap, and resulting difficulty tier.
  • Newly introduced tutor material cannot reattribute the answer to a different concept.
  • Explicit navigation requests still bypass answer grading and advance according to policy.
  • Judge failures do not create mastery evidence or corrupt pending-answer state.
  • Tests cover correct, partial, incorrect, navigation, unrelated request, and provider-failure turns.
  • Tutor behavior evidence shows that correct_full_answer remains attributed to return values and prerequisite_gap is treated as struggling before remediation is generated.
  • make check passes.

Priority

P0 production correctness blocker.

This should land before prompt tuning because prompt changes cannot reliably compensate for stale same-turn learner state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Release blocker for v1.0enhancementNew feature or request

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions