Cypress hung up trying to run a spec - won't resolve #17627
Comments
|
Runs automatically cancel after 1.5 hours of not getting a signal from CI for the current run. Could you run Cypress in debug mode mode and print the entire set of logs here for a hanging spec? I can't directly see any relation to 8.2.0 knowing what's in there. Were there any other changes made to your code/app/CI settings? |
jennifer-shehane
added
the
stage: needs information
|
We've also started seeing this very sporadically since upgrading from 8.0.0 -> 8.2.0. I've seen it 2x in ~50 runs, so it's definitely not something I can reliably replicate yet. |
|
debug mode seems to only work locally, not in circleci where I'm seeing the issue |
|
I have noticed the same problem on my attempt to upgrade to 8.2.0, on 100% of my 8.2.0 builds on Jenkins (but not always on the same spec). The time between starting Cypress and the failure happening isn't consistent (ranges from 5 to 15 minutes). Is there something else I can look into to help identify the cause of this problem? |
|
Some more notes:
|
|
Seeing a lot more occurrences of this error after upgrading from 8.2.0 to 8.3.0. Affected almost every run. |
|
We're having this exact same issue when running in CI after upgrading from 6.6.0 to 8.0.0. Tried upgrading further to 8.3.0 but the problem remains. The weird part is that the issue seems to not happen when we run with the debug mode. Normally it occurs almost every run. Some of the plugins we're using are cypress-cucumber-preprocessor and cypress-visual-regression if the information would be of any use. |
|
I can confirm the fact that it doesn't seem to happen with debug mode enabled (tried a dozen or so test runs). We do not use |
|
We found a workaround. Could not reproduce the problem anymore after disabling video recording (setting |
|
The Error happens in our Azure Pipeline as well (Cypress 8.3.1). |
|
Noticed the same on GitHub Action runs. Failed at the same point (while loading our catalog/services.spec.js file) 5 or 6 times in the last 10 days, however some test runs also completed successfully within that period. |
|
Random tests have been failing with the same |
Disabling videos did not work for us. |
|
We can reproduce this issue on every run with Cypress 8.2.0 and above. As already mentioned in #17998 we use Jenkins with Docker running on image cypress/base:14.17.0. Maybe someone is able to get more out of the debug log? What I see is that every test scenario ends with |
|
We have seen this issue as well and it disturbed our CI flow in Github Actions. We had one day last week where every PR, even on multiple reruns, was hitting this. Before that, it happened only occasionally for a period of about a week and half leading up. A few things we tried:
|
|
I noticed this failing on almost every scheduled GitHub Action run last week for one project but since about a day of four it is working again consistently. Was the problem on the dashboard end maybe? |
|
I've looked again on our logs to find a solution or at least a workaround as this issue blocks us from upgrading. I share here a log of one test (as an example of many others) where everything works as expected. The steps are usually executed in ~700ms:
If I compare it to the test where it fails:
My guess would be that Electron wasn't able to start within 30sec, but can it also be that the proxy is not ready in time? Free memory is is still enough, also free disk space. CPU is also not more than 30%. |
|
Ok I did around 30 rund and only 3 have been successful. The last successful run was with
What I see in the logs on usual start of a test is the following: At the point where it crashes it does the following: What I read from these two logs is that after the video capture is initialized Cypress starts to connect to a socket @jennifer-shehane does this information (and maybe my other posts) help to tackle this issue? the full logs have a size between 20MB and 1.3GB ... I think that's not very helpful. I can try also some different settings if you want. |
|
Ooh, nice capture, @rebel-l! Just from a quick inspection, it seems like it's getting hung opening Electron - it's making it to here, but getting hung up somewhere in here. It seems useful if you were able to rerun the same tests using: (just adding debugging of |
|
Actually... I think I might have spotted something. If debug is enabled for If I'm right about that, adding the debug string I suggested above will make it impossible to reproduce the issue (and hence, could be a more targeted temporary fix than |
|
@rbayliss I did another run with the debug string you suggested. The result was that it failed. The debug output itself when it fails is the following: It seems it can start and connect to electron but then the connection gets lost after it recieved the browser version. The logs around a test where Electron was started successfully looks like: I hope that helps you in your investigation. Let me know if I should try something else or provide more of the logs. |
|
Thanks, @rebel-l. So where you're getting hung up specifically is here (so unsurprisingly, we're getting hung up waiting for the initial page to load). I think we can disregard what I said above - that specific debug string obviously isn't the magic cure. At this point, I'm at a loss. There's a timeout error being caught by Electron itself after ~30 seconds. In my testing trying to find a reproducible test case, I've been unable to make that happen (only sporadically, in CI, with our full test suite and configuration, and with debug off). I think this needs further investigation from the Cypress team, who will need a reproducible test case. |
|
I can confirm we having this issue failing ~%50 of our test runs. Retries eventually pass. We are running on AWS CodeBuild with AWS Linux. I tried mitigating it by providing much bigger machines but it doesn't seem to resolve the issue. This has been the same for many 8.x.x Cypress versions we tried. |
|
We ran into this issue as well when upgrading to 8.x and have rolled back to 7.7.0. I tried a bunch of 8.x versions, but they all had the same issue. We'll have to stay on 7.x until this is resolved. |
|
Have the same issue on 8.6.0. Have someone resolved the issue? |
Nope, everyone is doing a workaround. Most of the people go back to latest 7.x version. As we switched from 6.x to 8.5.0 currently and spend a lot effort to get there, we decided to turn off video captuirng as soon as it is fixed. It's a very bad solution as we need a second run with only failed test and video capturing on and pray that this is fast enough that it don't break. I hope the people from Cypress will give that more priority soon. Again: I can offer my help to run a development version on our setup and share logs afterwards. |
|
@Fgerthoffert, i haven't been able to reproduce since that last run. I think you're ok to migrate since we haven enough to attempt a recreation in my forked repo. Thanks so much for all the info and help you've provided thus far. I'll be out until the week after the 4th of july and attempt to re-create again. |
|
@mjhenkes sounds good, we'll continue with the migration. This issue is quite critical for us, it's pretty impactful on our other repo. |
So this is fixed in 10 version , isn't it? |
|
This issue may be related to #22631. Still working on getting a reliable recreation. |
|
@romankhomitskyi, what makes you say this issue is fixed in version 10? |
Sorry, misunderstood comments |
cypress-bot
bot
added
stage: investigating
|
Some updates here. I've been able to get a semi reliable recreation, about 1 failure for every 4 builds. I was also able to get debug logs (attached are a successful and unsuccessful run) The pertinent sections start at line 17506 for the successful run and 17553 of the unsuccessful run Successful Failed Note the lack GET for |
|
@mjhenkes I'm not really surprised that some of the calls are missing in the failed logs if we assume that the failure is related to some process crashing. Cypress "crash" or "hangs", therefore does not execute what it's supposed to afterwards. The issue here is why are we having such issues? what are we doing that would be triggering this type of behavior? |
|
Surely some of the staff at cypress could weigh in on what they are running on their enterprise platform for their paying users? Im amazed how cypress themselves dont battle with this issue on their own platform because every single team ive worked with the biggest complaint is the amount of crashes and failures when running cypress in CI |
|
They don't battle with many more important issues: like Safari, runner performance, adequately preserve local storage and cookies, cy.origin, iframe, native events etc I really have no idea why they prioritized component testing (you can see a lot of opened Pull requests with that stuff), since cypress has beed forced as e2e testing tool |
|
I've narrowed this issue to having been introduced with version 8.6.0 based on my CI runs https://app.circleci.com/pipelines/github/mjhenkes/jcontent?branch=master Next I'll narrow down which commit in 8.6.0 specifically causes the hang. |
|
We have this issue with Cypress 8.1 as well as 8.5 😬 |
|
@mjhenkes how can we help from our side even if its just confirming something in logs or something? I have a private repo where I can pretty much replicate the failure 100% by just upgrading to cypress 10 |
|
@Antman261, It's likely that there are multiple "hanging" issues combined into this single thread. The example I currently have to work with was definitely introduced in cypress 8.6.0. The issues you're seeing in 8.1 and 8.5 are likely a different problem. @mattvb91, the best thing I can have is a reproducible example I can test with and verify corrections. Barring that, you can do what I'm doing, narrow in on what release/commit caused the problem. With the example I currently have I've been methodically pairing down what the problem could be. First by determining which dependency update introduced the issue (Cypress), then which version (8.6.0) and now which commit. It would be harder for an outside contributor to test each commit but I'm happy to help get temporary test versions of cypress built. Finally providing debug logs ( For your hang in particular, was it introduced in Cypress 10? Lots of code changed between Cypress 9 and cypress 10, which will make it tough to narrow down. Do you have an issue for your crash? (Sorry in advance if i or someone else closed it as a dupe of this one 😳. |
|
It seems this change is causing the hang: #18392 See my CI run evidence here: Now to dig into exactly what this change is causing. |
|
I have a PR out to fix this hang: #23001 A big thanks to @Fgerthoffert. Without your help getting a reproducible example, I wouldn't have gotten anywhere. This specific hang happened on Cypress versions 8.6.0 and greater when using electron and recording video. I believe there are more hangs reported in this thread than just the one I am fixing. Help me get a reproducible example and I'll see what I can do. I may close this issue with this fix and use new issues for additional hangs. If you cannot upgrade to Cypress 10.4.0, you'll have to lock into cypress 8.5.0 to avoid this hang. If there are enough of you that cannot upgrade, we can see about getting a 8.x or 9.x patch out, but I'd prefer it if we could avoid that. @stychu, you may be right, once the PR is merged you can test the pre-release or wait for 10.4.0 to get released next week. |
|
The code for this is done in cypress-io/cypress#23001, but has yet to be released. |
cypress-bot
bot
removed
the
stage: investigating
|
Released in This comment thread has been locked. If you are still experiencing this issue after upgrading to |
and others
Current behavior
Can't reproduce this locally - but intermittently when running specs via circleci (so far 50% of the time) - one of the specs (never the same one), spins causing circleci to timeout. When I look at the run in the Cypress dashboard it shows all the rest of the specs have passed except for that one and it's just constantly saying it is running - but shows no test results. https://dashboard.cypress.io/projects/zb7814/runs/1686/overview is an example. When I rerun the tests they all pass again so it looks like there's an issue with the cypress runner. This just started happening when we upgraded to 8 - specifically 8.2
The error from circleci is:
After saying it was still running for several hours on the Cypress dashboard - it finally stopped and showed this error message:
"The run started, but never completed. This can happen when the run is canceled from CI or when Cypress crashes during running tests.
Check your CircleCI #557d91f7-1bed-4c21-9f3c-9a4850bc64ad logs for more information. Archive this run to remove it from the runs list and analytics."
Desired behavior
Cypress not to get hung up
Test code to reproduce
Since it fails on a different spec each time, it doesn't seem to be related to actual test code
Cypress Version
8.2.0
Other
No response
The text was updated successfully, but these errors were encountered: