Mastery-gated progressive knowledge base and answer/learn modes

[evan188199-tech]

@wait-yyy Thanks for the thoughtful comment on #498. I think this idea has enough design depth that it may be easier to explore as a separate discussion. I have tried to structure the underlying proposal below. Please feel free to correct anything I have misunderstood, and maintainers/community members are very welcome to suggest additions or a better scope.

Mastery-Gated Progressive Knowledge Base and Answer/Learn Modes

Problem

DeepTutor currently combines broad model knowledge, user-selected knowledge bases, memory, Mastery Path, and agentic tool use. This is powerful, but it can make the tutor feel globally omniscient. It also makes learning progression less explicit: the system may answer a question even when the learner has not yet encountered its prerequisite concepts, and the knowledge available to the tutor is not necessarily tied to the learner's current stage.

The original suggestion was to limit the active knowledge scope so the tutor starts in a narrower world, then introduce additional knowledge modules when the learner's needs or mastery reach the next stage. Reasoning could expose what is known, what is missing, and what candidate next-step knowledge should be selected. This also suggests a separation between an immediate answer mode and a pedagogical learn mode.

Proposal

1. Scoped, stage-aware knowledge access

Introduce a per-course, per-topic, or per-learning-path knowledge scope. In learn mode, retrieval should preferentially use only the sources associated with the current stage. If a requested answer depends on out-of-scope knowledge, the tutor should say so instead of silently filling the gap with internal model knowledge.

This does not mean suppressing model capabilities everywhere. DeepTutor could offer:

• Answer mode: use the normal broad tool surface, selected KBs, memory, and web/tools as configured today.
• Learn mode: use a stage-scoped knowledge set, reveal prerequisites, and route unsupported requests into candidate future modules rather than directly answering them.

2. Learner-state and mastery gates

Use the existing Mastery Path surface as the natural integration point. Each knowledge module could define:

• prerequisite concepts;
• target concepts;
• evidence sources;
• example tasks or quizzes;
• mastery criteria;
• unlock conditions for the next module.

The learner advances by demonstrating enough evidence on current-stage material, not merely by asking about a later topic. If they do ask about a later topic, the system can diagnose which prerequisite gap matters and offer either a brief answer-mode response or a guided learning detour.

3. CoT as a missing-knowledge detector, not an ingestion authority

Chain-of-thought or an agentic planning step can be used to classify what the current reasoning needs:

1. already covered by the active knowledge scope;
2. covered by learner memory or prior mastered modules;
3. missing from the current scope and needed for the current goal;
4. speculative or unsupported.

Only category 3 should become a candidate knowledge module. The agent should not automatically ingest its own reasoning as truth. Candidate modules should require source evidence, review, deduplication, and conflict handling before they become part of the durable KB.

4. Candidate knowledge modules

When a missing prerequisite or target concept appears, the tutor can create a temporary learning proposal containing:

• why it was triggered;
• the relevant learner question or task;
• prerequisite links;
• source candidates;
• proposed assessment items;
• whether it should be personal, course-scoped, or global.

Depending on the deployment, these candidates could be approved by the learner, teacher, workspace administrator, or an automated evidence verifier.

Why This Is Grounded

This direction overlaps with several established research areas:

• Retrieval-augmented generation / grounded QA: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
• Mastery learning: Bloom, Learning for Mastery, 1968; Kulik, Kulik & Bangert-Drowns, 1990.
• Knowledge spaces and prerequisite structures: Doignon & Falmagne, Spaces for the Assessment of Knowledge, 1985.
• Curriculum learning: Bengio et al., Curriculum Learning, 2009.
• Knowledge tracing: Corbett & Anderson, 1995; Piech et al., Deep Knowledge Tracing, 2015.
• Reasoning plus tool/retrieval loops: Wei et al., Chain-of-Thought Prompting, 2022; Yao et al., ReAct, 2023; Trivedi et al., IRCoT, 2023; Jiang et al., Active Retrieval Augmented Generation, 2023; Asai et al., Self-RAG, 2024.

The educational-theory papers support staged progression and learning from known concepts to new concepts. The LLM papers support evidence retrieval, explicit reasoning steps, and deciding when to retrieve or abstain. The combination is still an engineering hypothesis, so it should be validated through concrete learning outcomes rather than assumed to work because the components exist.

Integration with DeepTutor

A v1 could stay narrow:

1. Add a learn/answer mode switch to Mastery Path or a new capability.
2. In learn mode, restrict RAG to the current course/topic/path sources.
3. Represent stages as modules with prerequisites, evidence, and mastery criteria.
4. Emit a scoped-answer event when evidence is unavailable.
5. Store missing-knowledge observations as reviewable candidates, not directly in memory or the KB.

Later versions could add:

• automatic prerequisite extraction from uploaded materials;
• learner-specific mastery estimates;
• teacher/admin approval workflows;
• conflict-aware KB updates;
• audits for every KB expansion;
• benchmark tasks measuring whether staged scoping improves learning outcomes without hurting useful answers.

Questions

1. Should learn mode hard-suppress out-of-scope model knowledge, or should it allow a brief answer while marking it as outside the learning scope?
2. Should new modules live in the learner's personal memory graph, the course KB, or a temporary learning-path module?
3. Who should approve knowledge expansion in the different deployment modes?
4. How should this interact with existing Mastery Path gating, RAG engines, and memory surfaces?
5. What mastery threshold or evidence mix is appropriate: quiz score, solved problems, teacher confirmation, or a combination?

Success Criteria

• Learners receive stage-appropriate guidance.
• Unsupported or out-of-scope answers are visibly scoped rather than silently generated.
• New knowledge is unlocked through evidence-based mastery gates.
• KB growth remains auditable, reversible, and linked to evidence.
• Learning outcomes improve compared with unrestricted chat/RAG on comparable tasks.