Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry integration tests when they fail #262

Merged
merged 2 commits into from
Mar 5, 2020

Conversation

jcreedcmu
Copy link
Contributor

Integration tests have been flaky. I tracked down the problem to the vscode electron binary sometimes segfaulting after all tests have been successfully run. The information the test runner gets from vscode-test unfortunately does not distinguish this segfault from test failure. See comments for what we could do about this.

I tried bisecting which integration test, if any, was causing the problem, without much success --- locally I get a segfault rate of about 0.5%-5% depending on which tests are present. It seems like the fewer tests are included in the suite, the fewer segfaults, but artificially duplicating tests doesn't make the segfaults tractably common. A minimal commandline to (possibly) trigger the segfault, after npm run integration has been run at least once, is

$PATH_UP_TO/vscode-codeql/.vscode-test/vscode-1.42.1/VSCode-linux-x64/code  \
  --disable-extensions \
  --extensionDevelopmentPath=$PATH_UP_TO/vscode-codeql/extensions/ql-vscode \
  --extensionTestsPath=$PATH_UP_TO/vscode-codeql//extensions/ql-vscode/out/vscode-tests/no-workspace/index

I wrapped this in a bash while loop waiting for $? to be 139. The .vscode-test/vscode-1.42.1/VSCode-linux-x64/code binary downloaded by vscode-test is stripped of debugging symbols so I wasn't able to get any insight from the coredump.

Copy link
Contributor

@aeisenberg aeisenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A very small comment. Looks fine to me, but I don't really have the background on this yet to approve.

@henrymercer
Copy link
Contributor

No docs impact

@henrymercer henrymercer merged commit 35f619e into github:master Mar 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants