Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cypress fails on the 8th test in CI #2265

Closed
kristiehowboutdat opened this issue Aug 2, 2018 · 13 comments

Comments

4 participants
@kristiehowboutdat
Copy link

commented Aug 2, 2018

Current behavior:

Our cypress tests intermittently fail to open / execute a file, only on version 3.0.x.

When they do fail, it is always the 8th test. I even created a very simple test file as the 8th one to confirm this suspicion.

Sample failure:

Running: 08_empty_test_should_pass.js...                            (8 of 14)
--
  | 🚨 Error: The command exited with status -1

Contents of 08_empty_test_should_pass.js:

describe('empty test', () => {});

Desired behavior:

Tests do not hang when opening the 8th (or any other) test file.

Steps to reproduce:

I'm not really sure what to put here since I can't repro it reliably, but here's the environment.

Buildkite with a docker container (cypress/base:8)

Set up more than 8 tests.
Run the tests a bunch of times.
If the tests fail, notice that it's always the 8th test, no matter what is in that file.

// cypress.json
{
  "video": true,
  "videoUploadOnPasses": false
}

Versions

Cypress: 3.0.3. We've been experiencing this issue on every 3.0.x version. We've had to revert to 2.x where this issue has never happened.

  | │ Cypress: 3.0.3 │
  | │ Browser: Electron 59 (headless) │
  | │ Specs: 14 found │

@jennifer-shehane

This comment was marked as outdated.

Copy link
Member

commented Aug 2, 2018

My first suspicion is that there is something at the end of the 7th spec file that may be causing this, could you isolate running the 7th file and the empty 8th file to see if this still occurs?

You can run specific specs using the --spec flag as documented here: https://on.cypress.io/command-line#Run-tests-specifying-multiple-test-files-to-run

@brian-mann

This comment has been minimized.

Copy link
Member

commented Aug 2, 2018

@kristiehowboutdat this is unfortunately now a known issue. We've spent a lot trying to debug it and understand why it's happening. We issued a PR in 3.0.3 after being able to reproduce similar conditions locally that we were hopeful that it fixed it.

Unfortunately, it did not. The challenge here is that it's extremely difficult to reproduce and seems to only happen on various CI providers. The only way we can gain an understanding is with the Cypress DEBUG logs.

To clarify, you haven't done anything wrong on your end, it's an issue with Cypress - however it's only an issue with Electron and if you switched to Chrome in CI, it would just work. Our docker containers have Chrome built in.

If you can reliably recreate this problem, then by enabling the DEBUG logs it will help us track it down.

To enable those run: DEBUG=cypress:* cypress run ...

You can check these other issues as they are all users with the same problem:

@kristiehowboutdat

This comment has been minimized.

Copy link
Author

commented Aug 6, 2018

Thanks for your feedback. I've switched to use --browser chrome and turned on the debug logs for 3.0.3.

Will report back if there are any updates!

@jbpallingayan

This comment has been minimized.

Copy link

commented Feb 8, 2019

hello any updates on this bug, were currently experiencing this on our third test. we try to change hierarchy of the file but it always failed on third. were using cypress 3.1.5

@brian-mann can you please help us. my test always hang on the 3rd test.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Feb 14, 2019

@jbpallingayan We cannot begin to look at this issue until someone provides us with a reproducible example, so no, there are no updates on this bug. Can anyone provide an example of test code + app code we can run to see this bug?

@jbpallingayan

This comment has been minimized.

Copy link

commented Feb 14, 2019

here is my temporary solution on this bug, i created two spec file that has no assertions it only runs a test my test will pass and it will not get sa same error again. i run with debug mode but i get the same error code waiting for response but when i try to run in gui electron i catch the bug and our keeps hanging on the same line of code.
screenshot from 2019-02-14 16-32-30

@jennifer-shehane update on this we experience this only on electron 59, is there a way that we can run our test on ci headless? or any recommendations that we can use chrome to run at our ci?

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Feb 27, 2019

@jbpallingayan You can pass a --browser flag to cypress run to run the tests within Chrome. You will no longer get recording of videos however.

@jbpallingayan

This comment has been minimized.

Copy link

commented Mar 8, 2019

we cannot move to our project because of random hang of cypress we cannot find any issues on both it will just hang randomly, even on headless chrome and electron.

we cannot share our app code or i can share my screen to you if you have time. @jennifer-shehane

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

@jbpallingayan Is this during cypress run or cypress open that this is happening?

@jbpallingayan

This comment has been minimized.

Copy link

commented Mar 8, 2019

sigh.. @jennifer-shehane we fix it and its only our fault. thank you for helping me on this. :)

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

@jbpallingayan Great! Any updates on how you fixed the issue would be helpful to add here - as I'm sure someone else will run into the same issue.

@jbpallingayan

This comment has been minimized.

Copy link

commented Mar 11, 2019

its on our app, error message is hidden when the user spam login on our app. im very sorry for this. its not a bug on cypress. @jennifer-shehane

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

Unfortunately we'll have to close this issue if no reproducible example is provided. Can anyone provide a way to reproduce this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.