Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reliable CI tests #2032

Closed
wants to merge 8 commits into from
Closed

Reliable CI tests #2032

wants to merge 8 commits into from

Conversation

ahlinc
Copy link
Contributor

@ahlinc ahlinc commented Jan 15, 2023

Some Tree-sitter tests are flaky and this from time to time affects CI with random fails in unexpected places.
At this moment I know two such cases:

  • Tests that use timeouts may fail.
  • Corpus tests that test random edits may fail on some random seed that produces an edit that changes parsing tree structure.

This PR marks such tests as #[ignored] by default and uses another nextest test runner to run such marked test separately with 10 retries.

@ahlinc ahlinc force-pushed the reliable-ci-tests branch 6 times, most recently from cd248c0 to 43a44fc Compare January 16, 2023 10:31
@sogaiu
Copy link

sogaiu commented Jan 17, 2023

I tried out 43a44fc a bit.

In the 15+ invocations I tried of ./script/test I didn't see a single failure :)

@ahlinc
Copy link
Contributor Author

ahlinc commented Jan 18, 2023

@sogaiu I didn't patched the ./script/test and with regular cargo test runner it just skips flaky tests.
I found the approach in the PR not very useful as CI and cargo test works differently and there is no much more benefit from cargo nextest in CI. I'll rework this PR in another way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants