Sandbox_integration: Skipping sandbox integration tests to unblock.#312537
Sandbox_integration: Skipping sandbox integration tests to unblock.#312537dileepyavan merged 21 commits intomainfrom
Conversation
Agent-Logs-Url: https://github.com/microsoft/vscode/sessions/ec5cf3c2-6c7b-4577-bdbb-8ac3d42bdfb0 Co-authored-by: dileepyavan <52841896+dileepyavan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Disables several sandbox-on integration tests for the run_in_terminal chat tool in vscode-api-tests to avoid current build failures in sandbox integration.
Changes:
- Marks five sandbox-on test cases as
test.skip(...)(unconditional). - Effectively removes coverage for network allowlisting, unsandboxed retry semantics, sandbox file access, and
$TMPDIRbehavior in CI.
Show a summary per file
| File | Description |
|---|---|
| extensions/vscode-api-tests/src/singlefolder-tests/chat.runInTerminal.test.ts | Unconditionally skips multiple “sandbox on” integration tests to unblock failing builds. |
Copilot's findings
Comments suppressed due to low confidence (4)
extensions/vscode-api-tests/src/singlefolder-tests/chat.runInTerminal.test.ts:335
- Marking this test as
test.skipunconditionally disables coverage forrequestUnsandboxedExecutionbehavior. Please add a tracking issue reference (similar to other skips in this repo) and/or make the skip conditional on the environment that is actually failing so the test still runs where supported.
test.skip('requestUnsandboxedExecution preserves sandbox $TMPDIR', async function () {
extensions/vscode-api-tests/src/singlefolder-tests/chat.runInTerminal.test.ts:373
- This change makes the “read files outside the workspace” sandbox test always skipped, which leaves the sandbox file-access boundary untested. If it’s flaky, please document the reason with an issue link and consider a narrower/conditional skip rather than disabling it everywhere.
test.skip('can read files outside the workspace', async function () {
extensions/vscode-api-tests/src/singlefolder-tests/chat.runInTerminal.test.ts:400
- Unconditionally skipping this test removes coverage for sandbox $TMPDIR writability. If CI is failing here, please add a tracking issue (or inline comment) and prefer conditional skipping (
this.skip()) based on the detected unsupported environment so it continues to run elsewhere.
test.skip('$TMPDIR is writable inside the sandbox', async function () {
extensions/vscode-api-tests/src/singlefolder-tests/chat.runInTerminal.test.ts:414
- Skipping this test unconditionally disables coverage for the non-allowlisted network confirmation flow. Please either link to the build failure/flakiness issue and add a TODO to re-enable, or make the skip conditional so we still validate this behavior in environments where sandboxing works reliably.
test.skip('non-allowlisted domains trigger unsandboxed confirmation flow', async function () {
- Files reviewed: 1/1 changed files
- Comments generated: 1
| test.skip('network requests to allowlisted domains succeed in sandbox', async function () { | ||
| this.timeout(60000); | ||
|
|
There was a problem hiding this comment.
This newly changes the test to an unconditional skip, which removes coverage for allowlisted-domain network behavior. If this is to unblock CI due to a known failure, please gate the skip (e.g., with this.skip() under the failing environment condition) and/or add a tracking issue link/comment explaining when it can be re-enabled.
This issue also appears in the following locations of the same file:
- line 335
- line 373
- line 400
- line 414
| test.skip('network requests to allowlisted domains succeed in sandbox', async function () { | |
| this.timeout(60000); | |
| test('network requests to allowlisted domains succeed in sandbox', async function () { | |
| this.timeout(60000); | |
| // Allow CI to gate this test explicitly in environments where sandboxed | |
| // outbound network access is known to fail, while keeping coverage enabled by default. | |
| if (process.env['VSCODE_SKIP_SANDBOX_ALLOWLIST_NETWORK_TEST'] === 'true') { | |
| this.skip(); | |
| } |
Skipping sandbox integration tests causing build failure.