Retries hangs on failed assertions #8363

zbigniewkalinowski · 2020-08-20T10:19:42Z

Current behavior:

Sometimes Cypress hangs to eternity on failed assertions while Retries functionality is active.
I described it once in this ticket:
kuceb/cypress-plugin-retries#64 (comment)

Now I'm experiencing similar issue in 5.0.0 version.
It's probably a problem with .should('contain', xxx) assertion.

Test code to reproduce

Versions

5.0.0

The text was updated successfully, but these errors were encountered:

jennifer-shehane · 2020-08-20T13:00:29Z

Can you provide a reproducible example? We have quite a few tests we run through, so there’s likely something specific about the way the tests are written or run that will trigger this.

This will be hard to track down without that, especially if it only fails sometimes. Thanks!

boringricardo · 2020-08-20T14:33:10Z

Same happened for us. Usually our test suite finished in about 3 or 4 minutes max (success/failure). Nothing changed, we just enabled retries runMode: 3 and then our last CI cypress run lasted ran for 01:32:22 according to the Cypress Dashboard, but actually the Github Action was still running after 4h 34m.

It hits the first failure and then just hangs forever, it literally does nothing else:

(Attempt 1 of 4) test name omitted here
(Attempt 1 of 4) test name omitted here
(Attempt 2 of 4) test name omitted here

^ this is the very last line, there is nothing else. edit: The duplicate "Attempt 1 of 4" is also for the same test, which is weird.

this happened on all of our CI runners for parallel tests (each one just hit the first error and then the same thing like above happens).

edit: Just removed the retries from our cypress config, and the test execution finished again in 4 minutes.

lukeapage · 2020-08-20T17:52:00Z

Same is happening for us, both with the plugin and with 5.0.0.

I did an investigation to track down what was hanging, you can see the issue on cypress plugin retries for that, assuming it’s the same problem, but I wasn’t able to work out the exact problem only what was happening.

Some of the failed tests hang, whilst others are able to retry 3 times and then fail so I think it is something specific to certain tests. I’ve already spent way too long this week debugging cypress but if I find a reproduction case I will post it.

jennifer-shehane · 2020-08-21T04:56:30Z

I'm unable to recreate this with a .should('contains', 'foo') assertion alone.

We will really need a reproducible example to be able to track this down - there could be a lot of things contributing, such as other defined hooks or things in the project support/plugins.

My test code

index.html

<html>
  <body>
    <div>bar</div>
  </body>
</html>

spec.js

it('test', () => {
  cy.visit('index.html')
  cy.get('div').should('contain', 'foo')
})

cypress.json

{
  "retries": {
    "runMode": 3,
    "openMode": 3
  }
}

Runs 4 times and correctly ends the test.

lukeapage · 2020-08-21T08:00:42Z

It only happens in run mode and I have a test case I can reproduce fairly consistently locally and on the build server.
Its failing on the first attempt to retry (test runs once, fails and then stalls). But once I start removing code from the test it starts failing on 2nd retry.
my test that is failing does not have should('contain', 'foo') assertion in it.
no additional hooks
I'm trying to make a test case but its very fragile - remove one assertion or another and it starts working or fails on 2nd retry - its like the length of the test makes a difference on whether retry will work

zbigniewkalinowski · 2020-08-21T08:41:49Z

Today my test hanged on click also.

jennifer-shehane · 2020-08-21T09:28:02Z

Does anyone have a test:after:run defined in their tests anywhere? #2271

lukeapage · 2020-08-21T09:36:35Z

@jennifer-shehane no hooks are needed.
I made a reproduction here: cypress-io/cypress-test-tiny#62
Does that reproduce for you? I know your on mac and I'm on windows but 🤞

jennifer-shehane · 2020-08-21T09:50:00Z

Yah, it reproduces it when running via cypress run, not during cypress open.

lukeapage · 2020-08-21T11:56:09Z

Is that enough to investigate? I ask because this issue is still labeled "Needs information"

jennifer-shehane · 2020-08-24T06:46:14Z

It is, but it's going to take a lot longer to determine the cause with such a large example.

Roger-Fang-CS · 2020-08-26T03:20:50Z

Same happened for us. Usually our test suite finished in about 3 or 4 minutes max (success/failure). Nothing changed, we just enabled retries runMode: 3 and then our last CI cypress run lasted ran for 01:32:22 according to the Cypress Dashboard, but actually the Github Action was still running after 4h 34m.

It hits the first failure and then just hangs forever, it literally does nothing else:

(Attempt 1 of 4) test name omitted here
(Attempt 1 of 4) test name omitted here
(Attempt 2 of 4) test name omitted here

^ this is the very last line, there is nothing else. edit: The duplicate "Attempt 1 of 4" is also for the same test, which is weird.

this happened on all of our CI runners for parallel tests (each one just hit the first error and then the same thing like above happens).

edit: Just removed the retries from our cypress config, and the test execution finished again in 4 minutes.

I meet the similar problem. after upgrade to 5.0.0, deprecate cypress-plugin-retries and use built-in retry function, my test cases fail all the time then hang for serveral hours when run github action.

print like below
(Attempt 1 of 3) tset1 name
(Attempt 2 of 3) test1 name
1) test1 name
........
(Attempt 1 of 3) tsetn name
(Attempt 2 of 3) testn name
n) testn name

but my code runs well on local machine. the only change is upgrade
(with "cypress open")

I run with "cypress run" on local machine again. And It hangs. Maybe it is not about github action.

jennifer-shehane · 2020-08-26T10:38:45Z

@lukeapage I narrowed down the reproducible example to this single spec file. I think the issue is with your overwriting cy.get(). We really don't recommend doing this - you should be using overwrite to overwrite commands.

https://github.com/cypress-io/cypress-test-tiny/pull/62/files#diff-05da80f0f815af9c2b8279ce29b86790

Run via cypress run --headed --no-exit to see the test stall.

lukeapage · 2020-08-26T11:05:03Z

There’s only a single spec file and spec in that Pr isn’t there?

How did you narrow it down to overwriting get ? I can try refactoring that.

Unfortunately we got so many issues with overwrite that we don’t use it.

lukeapage · 2020-08-26T13:39:57Z

Sorry I see what you did now. I will take a look later.

lukeapage · 2020-08-26T19:50:53Z

@jennifer-shehane I pushed some commits that simplify it further. I've removed the overwrite of get and also I replaced the "waitForTimers" with a simpler pause - I added the commits in stages because a 1s pause is more likely that it might suddenly stop reproducing.. but for me I still reproduce with the simplified example.

cypress-io/cypress-test-tiny@ae67c76
cypress-io/cypress-test-tiny@e52a63d
cypress-io/cypress-test-tiny@739d2c8

jennifer-shehane · 2020-08-27T06:22:41Z

@lukeapage K, thanks, thats helpful.

kuceb · 2020-09-04T14:16:35Z

I think I found the source of the issue, hoping to get a patch out in the next release.

kuceb · 2020-09-08T23:13:51Z

what reproduces the bug is having a test that fails and retries with an existent DOM node as the subject of an assertion e.g.:

cy.get('.element-exists').should('not.exist')

our logic only pretty formatted err.actual on test failure, not retry, so the browser hangs trying to serialize an entire DOM node.

cypress-bot · 2020-09-14T17:23:47Z

The code for this is done in cypress-io/cypress#8527, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

cypress-bot · 2020-09-15T14:10:31Z

Released in 5.2.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v5.2.0, please open a new issue.

jennifer-shehane added the stage: needs information Not enough info to reproduce the issue label Aug 20, 2020

rsi21 mentioned this issue Aug 21, 2020

Calling mochawesome addContext throws error in TypeScript spec files. #8374

Closed

cypress-bot bot added stage: needs investigating Someone from Cypress needs to look at this and removed stage: needs information Not enough info to reproduce the issue labels Aug 24, 2020

jennifer-shehane added v5.0.0 🐛 Issue present since 5.0.0 type: bug labels Aug 24, 2020

zbigniewkalinowski mentioned this issue Aug 25, 2020

cypress-plugin-retries causing problems with hanging assertions kuceb/cypress-plugin-retries#64

Closed

jennifer-shehane added the internal-priority label Aug 26, 2020

jennifer-shehane mentioned this issue Aug 27, 2020

Retry hangs #8363 cypress-io/cypress-test-tiny#62

Closed

jennifer-shehane mentioned this issue Aug 28, 2020

Unable to retry tests while using the Cypress run mode #8434

Closed

sync-by-unito bot assigned kuceb Sep 1, 2020

lukeapage mentioned this issue Sep 3, 2020

Retries with failures stops processing tests #8455

Closed

cypress-bot bot added stage: ready for work The issue is reproducible and in scope and removed stage: needs investigating Someone from Cypress needs to look at this labels Sep 4, 2020

kuceb mentioned this issue Sep 8, 2020

fix(driver): ensure errors dont get serialized w/ actual/expected when no showDiff #8527

Merged

2 tasks

cypress-bot bot added stage: work in progress There is an open PR for this issue [WIP] and removed stage: ready for work The issue is reproducible and in scope labels Sep 8, 2020

cypress-bot bot added stage: needs review The PR code is done & tested, needs review and removed stage: work in progress There is an open PR for this issue [WIP] labels Sep 14, 2020

chrisbreiding closed this as completed in #8527 Sep 14, 2020

cypress-bot bot added stage: pending release There is a closed PR for this issue and removed stage: needs review The PR code is done & tested, needs review labels Sep 14, 2020

cypress-bot bot removed the stage: pending release There is a closed PR for this issue label Sep 15, 2020

cypress-bot bot locked as resolved and limited conversation to collaborators Sep 15, 2020

Retries hangs on failed assertions #8363

Retries hangs on failed assertions #8363

zbigniewkalinowski commented Aug 20, 2020 •

edited by sync-by-unito bot

jennifer-shehane commented Aug 20, 2020

boringricardo commented Aug 20, 2020 •

edited

lukeapage commented Aug 20, 2020

jennifer-shehane commented Aug 21, 2020 •

edited

lukeapage commented Aug 21, 2020

zbigniewkalinowski commented Aug 21, 2020

jennifer-shehane commented Aug 21, 2020

lukeapage commented Aug 21, 2020

jennifer-shehane commented Aug 21, 2020

lukeapage commented Aug 21, 2020

jennifer-shehane commented Aug 24, 2020

Roger-Fang-CS commented Aug 26, 2020 •

edited

jennifer-shehane commented Aug 26, 2020 •

edited

lukeapage commented Aug 26, 2020

lukeapage commented Aug 26, 2020

lukeapage commented Aug 26, 2020 •

edited

jennifer-shehane commented Aug 27, 2020

kuceb commented Sep 4, 2020

kuceb commented Sep 8, 2020

cypress-bot bot commented Sep 14, 2020

cypress-bot bot commented Sep 15, 2020

Retries hangs on failed assertions #8363

Retries hangs on failed assertions #8363

Comments

zbigniewkalinowski commented Aug 20, 2020 • edited by sync-by-unito bot

Current behavior:

Test code to reproduce

Versions

jennifer-shehane commented Aug 20, 2020

boringricardo commented Aug 20, 2020 • edited

lukeapage commented Aug 20, 2020

jennifer-shehane commented Aug 21, 2020 • edited

lukeapage commented Aug 21, 2020

zbigniewkalinowski commented Aug 21, 2020

jennifer-shehane commented Aug 21, 2020

lukeapage commented Aug 21, 2020

jennifer-shehane commented Aug 21, 2020

lukeapage commented Aug 21, 2020

jennifer-shehane commented Aug 24, 2020

Roger-Fang-CS commented Aug 26, 2020 • edited

jennifer-shehane commented Aug 26, 2020 • edited

lukeapage commented Aug 26, 2020

lukeapage commented Aug 26, 2020

lukeapage commented Aug 26, 2020 • edited

jennifer-shehane commented Aug 27, 2020

kuceb commented Sep 4, 2020

kuceb commented Sep 8, 2020

cypress-bot bot commented Sep 14, 2020

cypress-bot bot commented Sep 15, 2020

zbigniewkalinowski commented Aug 20, 2020 •

edited by sync-by-unito bot

boringricardo commented Aug 20, 2020 •

edited

jennifer-shehane commented Aug 21, 2020 •

edited

Roger-Fang-CS commented Aug 26, 2020 •

edited

jennifer-shehane commented Aug 26, 2020 •

edited

lukeapage commented Aug 26, 2020 •

edited