Added model summary and risk assessment for commands that violate sandbox policy #5536
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…dbox policy and require user approval
etraut-openai
changed the title
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
jif-oai
reviewed
Oct 23, 2025
pakrym-oai
reviewed
Oct 23, 2025
pakrym-oai
reviewed
Oct 23, 2025
pakrym-oai
approved these changes
Oct 23, 2025
* Moved prompt into its own file and switched it to use askama for templating * Refactored sandbox_retry_data trait for simplification * Fixed otel telemetry so assessment conversation doesn't appear as a new task * Added otel telemetry point for recording latency of assessment * Removed defensive JSON parsing of assessment response Removed new experimental config key from public documentation for now. We're going to roll this out internally first to get feedback.
* Simplified config handling by leveraging "features" mechanism * Moved approvals-related schemas from protocol.rs to simplify
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for a model-based summary and risk assessment for commands that violate the sandbox policy and require user approval. This aids the user in evaluating whether the command should be approved.
The feature works by taking a failed command and passing it back to the model and asking it to summarize the command, give it a risk level (low, medium, high) and a risk category (e.g. "data deletion" or "data exfiltration"). It uses a new conversation thread so the context in the existing thread doesn't influence the answer. If the call to the model fails or takes longer than 5 seconds, it falls back to the current behavior.
For now, this is an experimental feature and is gated by a config key
experimental_sandbox_command_assessment.Here is a screen shot of the approval prompt showing the risk assessment and summary.