Skip to content

fix(convex): stop agents echoing <untrusted_source> tags into replies#2149

Merged
larryro merged 1 commit into
mainfrom
fix/untrusted-source-tag-leak
Jun 25, 2026
Merged

fix(convex): stop agents echoing <untrusted_source> tags into replies#2149
larryro merged 1 commit into
mainfrom
fix/untrusted-source-tag-leak

Conversation

@larryro

@larryro larryro commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Problem

Agents occasionally reproduced raw <untrusted_source ...>...</untrusted_source> wrapper tags in their user-facing replies — sometimes once, sometimes around every fourth sentence. A teammate flagged it in a live chat (the model wrapped each cited fact in the literal tag).

Root cause

This is a prompt-design gap, not model misbehavior:

  • We wrap every external tool result (web search, integrations like Tavily, RAG over external sources) in <untrusted_source> tags via wrapUntrusted() in lib/untrusted_content.ts. That's a deliberate prompt-injection defense — the tags tell the model "this is data, not instructions."
  • But the system.untrusted_content TRUST RULES prompt only said "quote the URL attribute of the enclosing tag" and never stated that the tags are internal markers that must not appear in output.
  • Models read "quote the enclosing tag" as a citation format and copied the whole wrapper through. It's intermittent because it depends on how the model chooses to cite that turn.

Fix

  • system.untrusted_content: replace the ambiguous line with two explicit rules — cite via the url attribute as a normal markdown link, and never reproduce the <untrusted_source> opening/closing tags.
  • researcher.json: same correction in the agent's inline TRUST RULES copy (it repeats the instruction and hits Tavily every turn, so it's the most visible offender).
  • render.test.ts.snap: update the cache-critical prompt snapshot to match. ⚠️ This template feeds the prompt-cache stable prefix — any byte change invalidates caches platform-wide; the snapshot test guards it.

Scope of effect

The shared UNTRUSTED_CONTENT_SYSTEM_PROMPT covers every agent whose tools return external content, so fixing it in one place (plus the researcher's inline copy) cleans it up everywhere.

Tests

  • render.test.ts — 14 passed (snapshot matches)
  • example-agents-normalized.test.ts — 2 passed (researcher.json still valid/normalizes)

Not in scope

The separate question of why Tavily ran without being statically bound (composer capability-pill → per-turn dynamic binding via mergeCapabilityBindings) is untouched — can follow up separately if desired.

The TRUST RULES system prompt told the model to 'quote the URL attribute
of the enclosing tag' when citing, but never said the tags themselves are
internal markers. Models read that as a citation format and reproduced the
whole <untrusted_source ...>...</untrusted_source> wrapper in user-facing
replies (intermittently, depending on how they chose to cite).

Add an explicit rule to system.untrusted_content (and the researcher
agent's inline copy) forbidding reproduction of the tags and pointing
citations at normal markdown links. Update the cache-critical prompt
snapshot accordingly.
@larryro larryro force-pushed the fix/untrusted-source-tag-leak branch from 651ecc4 to f2ff4ff Compare June 25, 2026 10:49
@larryro larryro merged commit 1edf37a into main Jun 25, 2026
50 checks passed
@larryro larryro deleted the fix/untrusted-source-tag-leak branch June 25, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant