Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support test retries #3968

Open
wants to merge 78 commits into
base: develop
from

Conversation

5 participants
@Bkucera
Copy link
Member

commented Apr 15, 2019

support mocha retries: fix #1313

Tasks

  • bug fix 'runnable:after:run:async' / screenshots in hooks after pass
    fix #3744 fix #3918

  • add .snapshot() function for agents (spies and stubs) to reduce memory
    fix #3849

  • allow test runner to be ran in an isolated window for ui tests

  • isolated test runner specs, snapshot tests for emitted events

  • add prevAttempts onto the test runnable instances in driver and server-reporter

  • reporter ui changes to display previous test attempts

  • allow retries configuration value

  • send _currentRetry and _retries runnable props in events (to recreate runnables)

  • update snapshot tests for module API and dashboard results

  • fix failing specs

  • pass along 'retry' event from mocha retry event is only fired in mocha 6+

  • fix merge conflicts from develop

  • disable retries by default in open mode, support config: enableTestRetriesInOpenMode: false (default)

  • rename retries config to numTestRetries

  • throw custom error when user sets retry via this.retries or on hook

  • screenshot filenames on retry, add tests (address Bkucera/cypress-plugin-retries#10)

  • emit a test retry event (for plugin authors)

other

  • upgrade chai devDep in packages/server, fix resulting errors in tests
  • fix flaky cli/unit test
  • seemingly broken css after merge

Docs / changes

  • you can set numTestRetries in cypress.json OR Use Use Cypress.config('numTestRetries')
  • to enable retries in open mode, use config enableTestRetriesInOpenMode (enabled in run mode by default if numTestRetries is set)

Migrating from retries plugin:

  • remove plugin-retries devDependencies and plugin-retries code in support files
  • remove usage of Cypress.currentTest
  • remove usage of this.retries(n) (not supported)
  • Use Cypress.config('numTestRetries') instead of Cypress.env('RETRIES') (or see cypress.json instructions above

jennifer-shehane and others added some commits Feb 20, 2019

Add 'attempts' component
- Style attempt headers in new scss file
- Create collapsible space within attempt to nest hooks
Fix automatically loading fixtures with more than one . in the name (#…
…3606)

Fixes #1402

Also fixes a few TODO comments in the fixtures.coffee file along the way
bump tough-cookie (#3659)
- breaking change: no longer supports node version < 6
- bugfixes
Bump cachedir from 1.3.0 -> 2.1.0 (#3660)
- Breaking change: Drop support for Node.js < 6.x]
- Use os.homedir() instead of os-homedir module (reduced dependency and
size)
chore(deps): update dependency @babel/core to version 7.3.4 🌟 (#3263)

Co-authored-by: Renovate Bot <bot@renovateapp.com>
Co-authored-by: Jennifer Shehane <shehane.jennifer@gmail.com>
fix references to globals
Co-authored-by: Brian Mann <brian@cypress.io>
fix references to globals
Co-authored-by: Brian Mann <brian@cypress.io>
test code for isolated runner
Co-authored-by: Brian Mann <brian@cypress.io>

@Bkucera Bkucera requested a review from brian-mann Apr 15, 2019

Bkucera added some commits Apr 15, 2019

https://on.cypress.io/test-retries
"""

This comment has been minimized.

Copy link
@Bkucera

Bkucera Apr 17, 2019

Author Member

pls review these error messages

@@ -887,6 +905,9 @@ module.exports = {
else
msg += "all of the remaining tests."

if (obj.hookName is 'after all' or obj.hookName is 'before all') and obj.retries > 0
msg += "\n\nAlthough you have test retries enabled, we do not retry 'before all' or 'after all' hooks"

This comment has been minimized.

Copy link
@Bkucera

Bkucera Apr 17, 2019

Author Member

pls review this error message

Bkucera added some commits Apr 17, 2019

@lukeapage

This comment has been minimized.

Copy link

commented Apr 29, 2019

Will this feature cover this scenario: Bkucera/cypress-plugin-retries#10

I can work-around it (I think) by listening to screenshot events, recording the filenames and then if I can detect a retry in the before each (same spec name?) I can delete the files before the test starts.. but I'd like to know if I should implement a work-around or if you plan to ship this working with this initial release, I'll just wait for it.

@Bkucera

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

@lukeapage yes, thank you for that. I'm adding that task here, the final attempt should produce the same artifacts as a normal passing test.

Edit: after considering we would have to rename files on retry, I think we will simply increment the filename with (attempt 2) and so on for retried attempts

@lukeapage

This comment has been minimized.

Copy link

commented May 1, 2019

That’s a shame. Any reason why? I can grep but the logic is a bit more complicated than ignoring files based on a regex (as we can with failed). I might have a go at deleting the files myself then.

And will the logic work like this..

Testname-1
Testname-2
Testname (failed)
Testname-1 (attempt 2)
Testname-2 (attempt 2)
Testname-3 (attempt 2)

@lukeapage

This comment has been minimized.

Copy link

commented May 1, 2019

And if I do delete the files myself, will you still add on (attempt 2) on the second attempt?

@Bkucera

This comment has been minimized.

Copy link
Member Author

commented May 8, 2019

@lukeapage so in plugins you will be able to listen to after:screenshot and rename any screenshot with (attempt ...) to overwrite the original file

but will add (attempt ...) regardless if a file exists

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.