Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide visibility to flaky tests #21

Closed
2 tasks done
agis opened this issue Aug 26, 2020 · 0 comments · Fixed by #26
Closed
2 tasks done

Provide visibility to flaky tests #21

agis opened this issue Aug 26, 2020 · 0 comments · Fixed by #26
Labels
enhancement New feature or request
Milestone
0.2.0

Comments

@agis
Copy link
Collaborator

agis commented Aug 26, 2020 •
edited

Flaky tests, while not causing the build to fail, should still be fixed. Otherwise they can impact build times, since they can silently compound and cause many retries in each build.Consider a test suite that over time has 30 flaky tests. These could easily cause 90 additional example retries.

We should provide visibility into flaky tests (after the fact we determined they are flaky). Now that #16 is merged, we can report them to Sentry.

  • print to stdout of reporter
  • submit to Sentry
@agis agis added the enhancement New feature or request label Aug 26, 2020
@agis agis added this to the 0.2.0 milestone Aug 28, 2020
@agis agis changed the title Submit flaky tests to Sentry Provide visibility to flaky tests Aug 28, 2020
agis added a commit that referenced this issue Aug 28, 2020
@agis
Implement Queue#flaky_jobs
d02e327 
This method can be called after a build is complete and returns the jobs
that were flaky (i.e. failed initially but passed upon a retry).

This enables us to provide visibility to flaky tests so that test
implementers have a chance to actually fix them.

Part of #21
agis added a commit that referenced this issue Aug 28, 2020
@agis
reporter: Print flaky jobs in build summary
f0b30d3 
Example output (the new part is just below "Spec execution time"):

    Failed examples:

      bin/rspec ./test/sample_suites/flaky_job_detection/spec/legit_failure_spec.rb:5 # is expected to equal true

    Total results:
      13 examples (6 jobs), 1 failures, 0 errors

    Spec execution time: 00:00:00

    Flaky jobs detected (count=2):
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:3]
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:1]

Part of #21
agis added a commit that referenced this issue Aug 28, 2020
@agis
reporter: Report flaky jobs to Sentry
1be0009 
Part of #21
@agis agis mentioned this issue Aug 28, 2020
Merged
agis added a commit that referenced this issue Aug 28, 2020
@agis
Implement Queue#flaky_jobs
14b49d0 
This method can be called after a build is complete and returns the jobs
that were flaky (i.e. failed initially but passed upon a retry).

This enables us to provide visibility to flaky tests so that test
implementers have a chance to actually fix them.

Part of #21
agis added a commit that referenced this issue Aug 28, 2020
@agis
reporter: Print flaky jobs in build summary
f878a21 
Example output (the new part is just below "Spec execution time"):

    Failed examples:

      bin/rspec ./test/sample_suites/flaky_job_detection/spec/legit_failure_spec.rb:5 # is expected to equal true

    Total results:
      13 examples (6 jobs), 1 failures, 0 errors

    Spec execution time: 00:00:00

    Flaky jobs detected (count=2):
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:3]
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:1]

Part of #21
agis added a commit that referenced this issue Aug 28, 2020
@agis
reporter: Report flaky jobs to Sentry
4c2cac5 
Part of #21
agis added a commit that referenced this issue Aug 30, 2020
@agis
Implement Queue#flaky_jobs
28ed7a3 
This method can be called after a build is complete and returns the jobs
that were flaky (i.e. failed initially but passed upon a retry).

This enables us to provide visibility to flaky tests so that test
implementers have a chance to actually fix them.

Part of #21
agis added a commit that referenced this issue Aug 30, 2020
@agis
reporter: Print flaky jobs in build summary
4ea3aae 
Example output (the new part is just below "Spec execution time"):

    Failed examples:

      bin/rspec ./test/sample_suites/flaky_job_detection/spec/legit_failure_spec.rb:5 # is expected to equal true

    Total results:
      13 examples (6 jobs), 1 failures, 0 errors

    Spec execution time: 00:00:00

    Flaky jobs detected (count=2):
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:3]
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:1]

Part of #21
agis added a commit that referenced this issue Aug 30, 2020
@agis
reporter: Report flaky jobs to Sentry
4c3e681 
Part of #21
agis added a commit that referenced this issue Aug 31, 2020
@agis
Implement Queue#flaky_jobs
5610bff 
This method can be called after a build is complete and returns the jobs
that were flaky (i.e. failed initially but passed upon a retry).

This enables us to provide visibility to flaky tests so that test
implementers have a chance to actually fix them.

Part of #21
agis added a commit that referenced this issue Aug 31, 2020
@agis
reporter: Print flaky jobs in build summary
d5c0b9c 
Example output (the new part is just below "Spec execution time"):

    Failed examples:

      bin/rspec ./test/sample_suites/flaky_job_detection/spec/legit_failure_spec.rb:5 # is expected to equal true

    Total results:
      13 examples (6 jobs), 1 failures, 0 errors

    Spec execution time: 00:00:00

    Flaky jobs detected (count=2):
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:3]
      ./test/sample_suites/flaky_job_detection/spec/flaky_spec.rb[1:1]

Part of #21
agis added a commit that referenced this issue Aug 31, 2020
@agis
reporter: Report flaky jobs to Sentry
e527e75 
Part of #21
@agis agis closed this as completed in #26 Aug 31, 2020
@agis agis mentioned this issue Oct 15, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
0.2.0
Development

Successfully merging a pull request may close this issue.

Add visibility to flaky jobs skroutz/rspecq
1 participant
@agis