Skip to content

test: Deal with incomplete tests in Jepsen tool #5804

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 9, 2020

Conversation

martinmr
Copy link
Contributor

@martinmr martinmr commented Jul 2, 2020

This PR does two things:

  1. Add an option --refresh-cluster to down and up the test cluster before each test. I noticed
    that most errors that lead to incomplete tests go away if this is done.
  2. Retry incomplete tests. Most incomplete tests fail early so it's not very expensive to retry
    the test. The tool now only returns an exit code of 1 if there's a test failure or a test was incomplete
    even after retries.

Related to DGRAPH-971


This change is Reviewable

Docs Preview: Dgraph Preview

@github-actions github-actions bot added area/integrations Related to integrations with other projects. area/testing/jepsen labels Jul 2, 2020
@martinmr martinmr changed the title test: Add option to recreate cluster for every Jepsen test. test: Deal with incomplete tests in Jepsen tool Jul 3, 2020
@martinmr martinmr marked this pull request as ready for review July 3, 2020 01:23
@martinmr martinmr requested a review from a team July 6, 2020 19:09
Copy link
Contributor

@parasssh parasssh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor. Feel free to resolve.

// Retry incomplete tests. Sometimes tests fail due to temporary errors.
tries++
if tries == maxRetries {
defer os.Exit(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a log here before exiting.

Copy link
Contributor Author

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 files reviewed, 1 unresolved discussion (waiting on @manishrjain, @parasssh, and @vvbalaji-dgraph)


contrib/jepsen/main.go, line 480 at r1 (raw file):

Previously, parasssh wrote…

maybe a log here before exiting.

Done.

@martinmr martinmr merged commit 5e7f443 into master Jul 9, 2020
@martinmr martinmr deleted the martinmr/jepsen-refresh branch July 9, 2020 19:46
dna2github pushed a commit to dna2fork/dgraph that referenced this pull request Jul 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/integrations Related to integrations with other projects. area/testing/jepsen
Development

Successfully merging this pull request may close these issues.

2 participants