Fix context-rot and citation quality issues in the chat agent - #190
Merged
Conversation
Rewrites all prompt templates to be cleaner and more instruction-precise (AGENT_SYSTEM_PROMPT, SYSTEM_PROMPT, SOURCED_ANSWER, REPHRASE, GENERATE_NEW_QUESTIONS, reformulate/standalone prompts). Fixes get_new_questions() to actually use the detected language when formatting the GENERATE_NEW_QUESTIONS template (was previously ignored), and corrects history slicing from broken [::-2][:2] to [-2:]. Fixes reformulate_user_query() to call the LLM via run_llm_with_json_parsing instead of returning a hardcoded stub. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…l-call fix AGENT_SYSTEM_PROMPT: anti-sycophancy, stronger length limit (1-2 sentences for openers), one tool call per response with comprehensive query, explicit prohibition on invented facts/examples, character-for-character URL copy instruction. Language detection (get_new_questions): when history exists, detect language from the last 4 history messages instead of the current short query (more reliable for langdetect). Accept optional ui_language (lang) param for the empty-chat case where there is no query to detect from. Thread lang through Context → ContextOut → endpoint → service. Agent sources (agent_response): collect artifacts from ALL ToolMessages instead of only the last one, so the right panel shows the complete set of retrieved docs when the agent makes multiple tool calls. Agent iteration cap (agent_message): add recursion_limit=5 to RunnableConfig to prevent runaway multi-call loops; complements the prompt instruction. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes blank lines after function/class definitions in several files, applied automatically by black during lint run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… on follow-up" This reverts commit 7bdcfc7.
…l on follow-up" This reverts commit c8a562c.
…case Both the agent's own end-of-turn question and the separate /reformulate/questions suggestions were framed generically around "continuing to learn about SDGs" rather than WeLearn's actual purpose: helping professors and course designers move from understanding sustainability topics to applying them in their own teaching.
Citations occasionally rendered as plain "[Doc N]" text instead of clickable links when the model didn't follow the formatting instruction. Adds a regex-based safety net (linkify_missing_citations in helpers.py) that wraps any bare marker with the correct <a> tag using the URL already available from the retrieved docs, applied to both the streaming and non-streaming agent endpoints.
…ibility Nested f-strings reusing the outer quote character only became legal in Python 3.12. Extracting the joined string into a variable first keeps this working on earlier versions.
…tion rules
Team review flagged several ambiguous references in AGENT_SYSTEM_PROMPT
("in this same conversation turn", "(see above)", "the retrieval tool")
and asked for harder "never" language plus repeated/reinforced citation
rules. Also adds a new section so requests for a specific deliverable
(e.g. a detailed learning activity) don't fall back to overly long,
externally-linked output.
…context rot Only the most recent get_resources_about_sustainability call's documents stay in the checkpointed conversation history; older ones are permanently cleared (not just hidden per-call) before summarization runs, so summarization no longer has to compress bulk that's already gone. Lowered the summarization trigger since the model was starting to ignore its own system prompt in long threads, and added a middleware that re-states a condensed version of the full prompt on every turn to keep those instructions salient. Also allows citing the same retrieved docs across turns when the topic hasn't shifted, and adds a vouvoiement rule for French replies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
External sources were sometimes named (title/journal) without being retrieved, since the prompt explicitly allowed unlinked plain-text mentions of non-retrieved sources — now forbidden outright. Combined citation markers like "[Docs 3 et 5]" are unlinkable (no single URL to resolve to), so the prompt now requires one document per marker, and the model is told to verify a claim is actually in the document it cites. Also switch citation format from HTML anchors to Markdown links ([Doc N](URL)), matching the rest of a Markdown reply and reducing formatting mistakes. linkify_missing_citations follows suit, but note it only patches the non-streaming endpoint and the saved chat record — it never reaches the live streamed view, so the prompt rules are the only real fix for what streams to the user. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rkdown rendering A plain Markdown link [Doc N](URL) renders with its brackets and URL consumed as syntax, showing only the bare text "Doc N" — this is correct Markdown behavior (per CommonMark), not the model dropping the citation. Since the brackets need to stay visible, wrap the whole marker in an extra pair: [[Doc N]](URL). The outer brackets are the link syntax (invisible once rendered), the inner "[Doc N]" is the literal label that survives rendering. Updated AGENT_SYSTEM_PROMPT and AGENT_REMINDER_PROMPT with the double-bracket format and a concrete wrong/correct example, and linkify_missing_citations's fallback wrapping to match, without re-wrapping a marker that's already correctly double-bracketed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
abst_chat.py still had literal unresolved merge-conflict markers around the langsmith/langgraph import block from the rebase onto main — both imports were actually needed (traceable from main's newer tracing instrumentation, REMOVE_ALL_MESSAGES for the persisted tool-clearing middleware), so this just keeps both lines and drops the markers. test_abst_chat.py still imported ReformulatedQueryResponse, which main's "Remove rephrase API endpoints" (#189) deleted along with the rephrase/reformulate tests that used it — the tests were already gone from this branch, just the now-dead import survived. Removed it. Both were breaking the module import outright (SyntaxError / ImportError). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jmsevin
approved these changes
Sep 10, 2026
jmsevin
left a comment
jmsevin
left a comment
Collaborator
There was a problem hiding this comment.
Looks good for a current fix. We'll probably have to improve this again in the future, after our UX audit, f.e. by implementing a "resources" state by thread, where all used resources could be stored and dynamically indexed.
sandragjacinto
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[[Doc N]](URL)) once we confirmed a plain[Doc N](URL)renders with its brackets stripped by design (that's how Markdown works, not a model error).linkify_missing_citations) to match — worth noting this fallback only patches the non-streaming endpoint and the saved chat record; it can't fix what's already been streamed live to the user..gitignoreentries for local scratch files).Known issue (not fixed here): a cited document link can occasionally resolve to a URL from an earlier search earlier in the same conversation rather than the current one — investigated, backend and frontend both ruled out as the direct cause, tabled for the team to dig into further.
Test plan
flake8/isort/blackclean on all changed files_PersistClearedToolUses,_ReinforceHardConstraints) and updatedlinkify_missing_citationsbehavior, run directly viaunittest(bypasses a pre-existing, unrelated brokenconftest.py/environment issue — not from this branch) — all passinglint-and-testrun (this branch's local dev environments couldn't run the whole suite; deferring to CI)[Doc N]links, no external sources get named, and instruction adherence holds up in a long thread🤖 Generated with Claude Code