test: skip flaky chat sandbox home-read smoke test - #325575
Merged
Merged
Conversation
The 'allows reading a home directory file configured in allowRead' smoke test remains flaky after the app restart added in #325532: the restart tears down the warmed-up chat participant and its mock LLM connection, so the probe message can race an unready chat and time out (observed on release/1.129 build 455304, and 9/20 in the flaky-test pipeline 455265). Skip it to unblock the build while a proper fix (re-warming chat after the restart) is prepared. Tracked by microsoft/vscode-engineering#3280. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR temporarily disables a flaky chat sandbox smoke test by marking it as skipped, to unblock release/1.129 builds while a proper fix (re-warming chat after restart) is prepared separately.
Changes:
- Mark the “home directory file configured in allowRead” chat sandbox smoke test as
it.skip. - Add an inline comment explaining the flake cause (restart tears down warmed chat/mock LLM) and linking the tracking issue.
Show a summary per file
| File | Description |
|---|---|
| test/smoke/src/areas/chat/chatSandbox.test.ts | Skips the flaky home-read allowRead smoke test and documents why/where it’s tracked. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Low
Giuspepe
marked this pull request as ready for review
July 13, 2026 09:28
Giuspepe
enabled auto-merge (squash)
July 13, 2026 09:29
deepak1556
approved these changes
Jul 13, 2026
Giuspepe
added a commit
that referenced
this pull request
Jul 13, 2026
…start #325575 skipped the "allows reading a home directory file configured in allowRead" smoke test because it went flaky after the app restart added in #325532. This re-enables it with a proper fix. restartWithUpdatedSandboxSettings restarts the app so the sandbox settings reload deterministically. A full restart tears down the extension host along with the warmed-up chat participant and its mock LLM connection, but the helper only waited for the chat view DOM (waitForChatView) before sending the probe. On loaded CI agents the first probe races an unready chat and never receives a response: Timed out waiting for response matching /HOME_READ_ALLOWED_EXIT_CODE=(\d+)/ Re-run the existing warmUpChat retry loop after the restart so the probe is only sent once chat can reliably reach the mock LLM server. Verified locally against the installed Insiders build: with the post-restart warm-up disabled the exact failure reproduces 3/8 runs; with the fix it does not occur across 13 runs (8 on main, 5 here). Fixes microsoft/vscode-engineering#3280 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Giuspepe
added a commit
that referenced
this pull request
Jul 13, 2026
…restart (#325594) Re-enable Chat Sandbox home-read smoke test and re-warm chat after restart #325575 skipped the "allows reading a home directory file configured in allowRead" smoke test because it went flaky after the app restart added in #325532. This re-enables it with a proper fix. restartWithUpdatedSandboxSettings restarts the app so the sandbox settings reload deterministically. A full restart tears down the extension host along with the warmed-up chat participant and its mock LLM connection, but the helper only waited for the chat view DOM (waitForChatView) before sending the probe. On loaded CI agents the first probe races an unready chat and never receives a response: Timed out waiting for response matching /HOME_READ_ALLOWED_EXIT_CODE=(\d+)/ Re-run the existing warmUpChat retry loop after the restart so the probe is only sent once chat can reliably reach the mock LLM server. Verified locally against the installed Insiders build: with the post-restart warm-up disabled the exact failure reproduces 3/8 runs; with the fix it does not occur across 13 runs (8 on main, 5 here). Fixes microsoft/vscode-engineering#3280 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
Chat Sandbox (darwin) > allows reading a home directory file configured in allowReadsmoke test is flaky and is red onrelease/1.129(build 455304), and failed 9/20 in the flaky-test pipeline (build 455265).This unblocks the build. It is a stop-gap — a proper fix (re-warming chat after the restart) is being prepared separately.
Root cause (for context)
This is the only active test that calls
app.restart()(viarestartWithUpdatedSandboxSettings, added in #325532 to make theallowReadsetting reload deterministically). The restart tears down the extension host along with the warmed-up chat participant and its mock-LLM connection, but the helper only waits for the chat view DOM (waitForChatView) — it never re-runs thewarmUpChatretry loop that the cold-startbeforehook relies on (which retries for up to 180s). So the first probe message races an unready chat and the response never renders → 120s timeout.Two distinct failure signatures observed across the builds above:
HOME_READ_ALLOWED_EXIT_CODE=1(theallowReadsetting had not been reloaded).Change
Marks the test
it.skipwith an explanatory comment. No other behavior changes.Tracked by https://github.com/microsoft/vscode-engineering/issues/3280.
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
cc @dileepyavan