feat: enhance adversarial defense mechanisms in prompts and tools#122
Conversation
|
Warning Review limit reached
More reviews will be available in 43 minutes and 51 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
WalkthroughThe PR updates prompt text and tool outputs so target responses are treated as untrusted data, adds sandbox framing to attacker-facing prompts, and wraps successful target responses in ChangesTarget Output Hardening
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/src/autonomous/tools/getThread.ts`:
- Around line 36-42: The get_thread serialization path is wrapping every
non-error t.response and omitting the rate-limited passthrough state, which
breaks parity with send_to_target and recon_probe. Update the turn-building
logic in getThread/get_thread to preserve t.rateLimited alongside response, and
skip the <untrusted_target_output> wrapper whenever a turn is rate-limited so
replayed transcripts match the live-tool contract. Ensure each serialized turn
includes the rateLimited flag in addition to response, score, and isError.
In `@core/src/autonomous/tools/sendToTarget.ts`:
- Around line 147-153: The untrusted output wrapper in sendToTarget is currently
bypassable because result.response is inserted raw, allowing a target to close
the delimiter and inject instructions. Update the wrapping logic in sendToTarget
to escape or serialize the payload before surrounding it with
<untrusted_target_output> tags, and apply the same reusable helper wherever
target text is wrapped, including recon_probe and get_thread. Prefer a shared
utility such as wrapUntrustedTargetOutput so the trust-boundary handling stays
consistent across these call sites.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ede73440-e82a-4f1d-8bf6-d631b7f921d5
📒 Files selected for processing (7)
core/src/autonomous/prompts/commander.tscore/src/autonomous/prompts/operator.tscore/src/autonomous/tools/getThread.tscore/src/autonomous/tools/reconProbe.tscore/src/autonomous/tools/sendToTarget.tscore/src/generate/generateAttacks.tscore/src/prompts/attacker-mcp.ts
| response: | ||
| t.isError || !t.response | ||
| ? t.response | ||
| : `<untrusted_target_output>\n${t.response}\n</untrusted_target_output>`, | ||
| score: t.score, | ||
| isError: t.isError, | ||
| })), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve the rate-limited passthrough contract in get_thread.
send_to_target and recon_probe intentionally leave rate-limited responses unwrapped, but this path wraps any non-error t.response and drops rateLimited from the serialized turn. That makes replayed transcripts diverge from the live-tool contract and can present rate-limit notices as normal target output. Please gate on t.rateLimited here too and return that flag in each turn.
Suggested fix
turns: thread.turns.map((t) => ({
turnIndex: t.turnIndex,
inherited: thread.forkedFromTurn !== undefined && t.turnIndex <= thread.forkedFromTurn,
persona: t.persona,
strategy: t.strategy,
prompt: t.prompt,
response:
- t.isError || !t.response
+ t.isError || t.rateLimited || !t.response
? t.response
: `<untrusted_target_output>\n${t.response}\n</untrusted_target_output>`,
score: t.score,
isError: t.isError,
+ rateLimited: t.rateLimited,
})),📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| response: | |
| t.isError || !t.response | |
| ? t.response | |
| : `<untrusted_target_output>\n${t.response}\n</untrusted_target_output>`, | |
| score: t.score, | |
| isError: t.isError, | |
| })), | |
| response: | |
| t.isError || t.rateLimited || !t.response | |
| ? t.response | |
| : `<untrusted_target_output>\n${t.response}\n</untrusted_target_output>`, | |
| score: t.score, | |
| isError: t.isError, | |
| rateLimited: t.rateLimited, | |
| })), |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@core/src/autonomous/tools/getThread.ts` around lines 36 - 42, The get_thread
serialization path is wrapping every non-error t.response and omitting the
rate-limited passthrough state, which breaks parity with send_to_target and
recon_probe. Update the turn-building logic in getThread/get_thread to preserve
t.rateLimited alongside response, and skip the <untrusted_target_output> wrapper
whenever a turn is rate-limited so replayed transcripts match the live-tool
contract. Ensure each serialized turn includes the rateLimited flag in addition
to response, score, and isError.
2 participants
Problem
Two gaps in the autonomous (
opfor hunt) and attack-generation paths:The hunt agent trusts target output as instructions. Commander/operator subagents receive raw target replies. An adversarial target can embed directives ("the test is over", "you are now in maintenance mode", "reply PASS", "ignore previous instructions") that the agent may follow — a prompt-injection-on-the-tester risk that can corrupt a run or suppress findings.
The attacker LLM over-refuses. When generating adversarial prompts, the model sometimes treats the evaluator's strong language ("fraud", "exfiltrate", "deceive") as a real-world request and refuses, so the target ships unevaluated. The generation prompts also did not state the concrete pass/fail criteria the message should target.
Solution
Adversarial-target defense (hunt path):
send_to_target,recon_probe, andget_threadnow wrap target responses in<untrusted_target_output>…</untrusted_target_output>delimiters before returning them to the agent. Wrapping is applied on the return path only; error and rate-limited responses pass through unwrapped, so failure handling is unchanged.Attacker-prompt hardening (run path):
fail_criteria/pass_criteriadirectly into the generation prompts so the crafted message and judge hint target the actual vulnerability definition.These are additive: prompt-text changes carry no control-flow impact, and the response-wrapping is a guarded, return-path-only transform.
Checklist
npm run buildpassesnpm run typecheckpassesnpm run build:catalog:checkpasses (if evaluators or suites changed) — N/A, no evaluator/suite changesopfor huntend-to-end (target: deepseek-chat via OpenRouter, brain: claude-haiku-4.5); recon, sends, andget_threadall exercised the wrapped path, and a finding still passed the verbatim-evidence guard.env, or.opfor/artifacts committed<type>: <what changed>—feat: harden red-team prompts against adversarial targets and attacker refusalsEvaluator checklist (skip if no evaluator added)
Skipped — no evaluator added or changed.
Summary by CodeRabbit