Retry integration tests when they fail #262
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integration tests have been flaky. I tracked down the problem to the vscode electron binary sometimes segfaulting after all tests have been successfully run. The information the test runner gets from
vscode-test
unfortunately does not distinguish this segfault from test failure. See comments for what we could do about this.I tried bisecting which integration test, if any, was causing the problem, without much success --- locally I get a segfault rate of about 0.5%-5% depending on which tests are present. It seems like the fewer tests are included in the suite, the fewer segfaults, but artificially duplicating tests doesn't make the segfaults tractably common. A minimal commandline to (possibly) trigger the segfault, after
npm run integration
has been run at least once, isI wrapped this in a bash while loop waiting for
$?
to be 139. The.vscode-test/vscode-1.42.1/VSCode-linux-x64/code
binary downloaded byvscode-test
is stripped of debugging symbols so I wasn't able to get any insight from the coredump.