Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support error logging before jest retry #12201

Merged

Conversation

zackasaurus
Copy link
Contributor

@zackasaurus zackasaurus commented Dec 30, 2021

Summary

Currently, to my knowledge Jest does not support logging errors before a retry event happens. This makes it difficult to debug why a test failed, especially if the code the test is testing is not deterministic (it will fail once, but will pass during the first retry), and thus no error will be visible as why it failed during the first time.

The need for visibility is a big driver here as I'd like to know if the reason that the retry occurred was due to a Jest.setTimeout error or an error that occurred in the test itself.

Test plan

Visuals

Screen Shot 2022-02-14 at 10 20 58 PM

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Dec 30, 2021

Hi @zackasaurus!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Dec 30, 2021

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@zackasaurus zackasaurus marked this pull request as ready for review Dec 30, 2021
Copy link
Collaborator

@SimenB SimenB left a comment

Thanks for the PR! I like the idea.

can you update the documentation as well?

jest.retryTimes(3);
jest.setLogTestErrorsBeforeRetry(true);
Copy link
Collaborator

@SimenB SimenB Dec 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
jest.retryTimes(3);
jest.setLogTestErrorsBeforeRetry(true);
jest.retryTimes(3, {logErrorsBeforeRetry: true});=

I'd rather we just add an options bag to the existent call than adding a new method entirely

Copy link
Contributor Author

@zackasaurus zackasaurus Jan 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, I've updated it. Do we want to handle the Symbol differently as well? Right now, there is still an additional symbol for the logErrorsBeforeRetry.

@codecov-commenter
Copy link

codecov-commenter commented Jan 3, 2022

Codecov Report

Merging #12201 (521dd6c) into main (2953433) will decrease coverage by 0.01%.
The diff coverage is 33.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main   #12201      +/-   ##
==========================================
- Coverage   66.93%   66.92%   -0.02%     
==========================================
  Files         329      329              
  Lines       17344    17352       +8     
  Branches     5061     5063       +2     
==========================================
+ Hits        11610    11612       +2     
- Misses       5702     5708       +6     
  Partials       32       32              
Impacted Files Coverage Δ
packages/jest-circus/src/run.ts 5.81% <0.00%> (-0.22%) ⬇️
packages/jest-types/__typetests__/jest.test.ts 0.00% <0.00%> (ø)
packages/jest-runtime/src/index.ts 55.25% <50.00%> (-0.08%) ⬇️
packages/jest-circus/src/types.ts 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2953433...521dd6c. Read the comment docs.

@zackasaurus zackasaurus requested a review from SimenB Jan 3, 2022
e2e/__tests__/testRetries.test.ts Outdated Show resolved Hide resolved
e2e/__tests__/testRetries.test.ts Outdated Show resolved Hide resolved
e2e/__tests__/testRetries.test.ts Outdated Show resolved Hide resolved
packages/jest-circus/src/run.ts Outdated Show resolved Hide resolved
@SimenB
Copy link
Collaborator

SimenB commented Feb 9, 2022

@zackasaurus ping 🙂

@zackasaurus
Copy link
Contributor Author

zackasaurus commented Feb 10, 2022

not a fan of using console here. Can we update the result object and reporter to include the information instead?
@zackasaurus ping 🙂

Hey @SimenB !
Just making sure I'm on the right track here, but basically add something like this in packages/jest-core/src/ReporterDispatcher.ts =>

async onTestRetry(
    results: AggregatedResult,
    options: ReporterOnTestRetryOptions,
  ): Promise<void> {
    for (const reporter of this._reporters) {
      reporter.onTestRetry && (await reporter.onTestRetry(results, options));
    }
  }

and have packages/jest-circus/src/run.ts dispatch that method?

 while (numRetriesAvailable > 0 && test.errors.length > 0) {
      if (logErrorsBeforeRetry) {
        // dispatch test retry reporter method ?
      }
      // Clear errors so retries occur
      await dispatch({name: 'test_retry', test});

      await _runTest(test, isSkipped);
      numRetriesAvailable--;
    }

@SimenB
Copy link
Collaborator

SimenB commented Feb 10, 2022

I think we should just add something to the test result that allows the reporter to print some context, which the reporter can then print after the tests is done. I don't think the reporter needs to be told when it happens? So essentially, look at where circus currently handles the test_retry event (it just clears errors) and have it push those errors into some retryReasons array or some such. Then when the test is done, you can ensure it's on the test result. Then in the reporter when it's printing the results it can inspect this new array and then print something.

Makes sense?

@zackasaurus
Copy link
Contributor Author

zackasaurus commented Feb 15, 2022

@SimenB I've yet to update the tests for this feature, but do the newest changes reflect your vision? I could only get the errors to show up correctly when using jest-serializer, so I'm passing an array of buffers to the reporter through the test result, and had to modify the deepCyclicCopy so the buffer data type would not change. I also included a visual of what the reporter displays when running a test. Let me know what you think! Thanks for all your help. :)

@SimenB
Copy link
Collaborator

SimenB commented Feb 16, 2022

@zackasaurus I need to re-process this PR, might be a few days. Gut feeling is that this solution is more complex than it needs to be, but I need to dig into it.

I'll add it to the milestone so I don't forget 🙂

@SimenB SimenB added this to the Jest 28 milestone Feb 16, 2022
CHANGELOG.md Outdated Show resolved Hide resolved
@SimenB
Copy link
Collaborator

SimenB commented Apr 21, 2022

@zackasaurus I tweaked the logging a bit, thoughts?

image

SimenB
SimenB approved these changes Apr 21, 2022
Copy link
Collaborator

@SimenB SimenB left a comment

I'm happy with the current state of this 👍

@SimenB SimenB merged commit 811228d into facebook:main Apr 23, 2022
89 checks passed
F3n67u pushed a commit to F3n67u/jest that referenced this pull request May 2, 2022
Co-authored-by: Zachary Bogard <zbogard@microsoft.com>
@github-actions
Copy link

github-actions bot commented May 24, 2022

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants