Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple tests pass in opened UI but fail on command line (application error) #1297

Closed
ndonnellan opened this issue Feb 12, 2018 · 18 comments

Comments

@ndonnellan
Copy link

commented Feb 12, 2018

Current behavior:

Simple tests against web-app work, but when I run them in the headless mode, they fail and the error given is an application error that seems unrelated (or an indication that the web-page didn't finish loading.
Uncaught SyntaxError: Unexpected token (

basic e2e tests -- logging in -- should be able to login -- before each hook

If I replace the 'uncaught:exception' event in cypress like the documentation suggests,

Cypress.on('uncaught:exception', (err, runnable) => {
  return false
})

I get a totally different error complaining about my usage of get

CypressError: Timed out retrying: Expected to find element: 'input[name=email]', but never found it.
I suspect this is because the page never finished loading, so it's probably not important.

basic e2e tests -- logging in -- should be able to login

Desired behavior:

My tests should run in both interactive mode and headless mode.

How to reproduce:

I'm not sure how to reproduce except with my app (React/Redux with Apollo). I have confirmed that I can run the example cypress tests fine (in a clean state) both from UI and from command-line.

Test code:

const DEFAULT_URL = 'http://localhost:3000'

describe('Basic e2e tests', function() {
  it('.should() - assert that <title> is correct', function() {
    cy.visit(`${DEFAULT_URL}/login`)
    cy.title().should('include', 'React App') // <----- this works
  })

  context('Logging in', function() {
    beforeEach(function() {
      cy.visit(`${DEFAULT_URL}/login`)
    })

     it('Should be able to login', function() {
      cy.get('input[name=email]') // <------ failure here

I tried adding a wait, but it seems like it never loads. The screenshot definitely makes it look like it never loads.

Additional Info (images, stack traces, etc)

Full stack trace of the code where I let the error propagate:

CypressError: Timed out retrying: Expected to find element: 'input[name=email]', but never found it.
      at Object.cypressErr (http://localhost:3000/__cypress/runner/cypress_runner.js:65978:11)
      at Object.throwErr (http://localhost:3000/__cypress/runner/cypress_runner.js:65943:18)
      at Object.throwErrByPath (http://localhost:3000/__cypress/runner/cypress_runner.js:65970:17)
      at retry (http://localhost:3000/__cypress/runner/cypress_runner.js:60079:16)
      at http://localhost:3000/__cypress/runner/cypress_runner.js:52597:18
      at tryCatcher (http://localhost:3000/__cypress/runner/cypress_runner.js:6268:23)
      at Promise._settlePromiseFromHandler (http://localhost:3000/__cypress/runner/cypress_runner.js:4290:31)
      at Promise._settlePromise (http://localhost:3000/__cypress/runner/cypress_runner.js:4347:18)
      at Promise._settlePromise0 (http://localhost:3000/__cypress/runner/cypress_runner.js:4392:10)
      at Promise._settlePromises (http://localhost:3000/__cypress/runner/cypress_runner.js:4467:18)
      at Async._drainQueue (http://localhost:3000/__cypress/runner/cypress_runner.js:1200:16)
      at Async._drainQueues (http://localhost:3000/__cypress/runner/cypress_runner.js:1210:10)
      at Async.drainQueues (http://localhost:3000/__cypress/runner/cypress_runner.js:1084:14)
  • Operating System: macOS Sierra 10.12.6 (16G1212)
  • Cypress Version:1.4.2
  • Browser Version:Chrome?
@brian-mann

This comment has been minimized.

Copy link
Member

commented Feb 12, 2018

I am almost 100% positive this is because when running headlessly we use Electron (Chromium 53) - and when you launch the browser it is a much newer version of Chrome.

The difference is that the older browser likely does not have newer JS features which is why it's failing. Turning off uncaught exceptions won't actually make the page render - it just hides catching that error. If you were to open your dev tools when running from Electron you would see the error.

This document explains all of this: https://docs.cypress.io/guides/guides/launching-browsers.html

However we are about to release 2.0.0 very soon - like today or tomorrow which bumps Chromium 53 to 59 - which is much newer and these types of differences will be covered up.

I'm closing the issue assuming this is the problem.

@brian-mann brian-mann closed this Feb 12, 2018

@brian-mann

This comment has been minimized.

Copy link
Member

commented Feb 12, 2018

You can also switch the browser to Electron in the GUI to inspect the error, or make it run in headed mode with --headed when running from the CLI.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Feb 12, 2018

Or you could also just run in Chrome from the command line with --browser chrome argument.

https://docs.cypress.io/guides/guides/command-line.html#

@mkatrenik

This comment has been minimized.

Copy link

commented Feb 15, 2018

@brian-mann I have same problem - piece if code in headed electron it works, in headless electron it fails. But there is 1 strange thing - it fails on selector which returns 1 match, this is visible in inspector, but in UI it says NO matches.

screen shot 2018-02-15 at 7 32 15 pm

screen shot 2018-02-15 at 7 32 42 pm

@brian-mann

This comment has been minimized.

Copy link
Member

commented Feb 15, 2018

@mkatrenik there's no way based on what you've provided to be of any assistance. In headless you get a full recorded video + screenshots. You'd need to post all of those, plus your test code.

If I were to guess it's likely a problem of elements being re-rendered while Cypress code is running. It's possible that you can lead Cypress down a path of elements, deeper and deeper, and then a parent element is rerendered which detaches all of the children. In those cases it's impossible to proceed and it will fail.

Best practice is to always perform actions as high up in the chain as possible. In your case you have a get and then find and then click. It's best to do get and then click to avoid this as much as possible.

Another option is to add more assertions and "guards" around Cypress so it doesn't proceed until the state of your application has settled first.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Feb 15, 2018

There is little difference between running headed and headlessly in Electron - except that headless is optimized for long runs and certain things are disabled which means it will technically run faster than headed. This is where it could deviate. However, it could just be that you've written flaky test code and that if you were to repeat the same test say 100 times, you'll see the same failures in the UI too. That's another best practice - when you have inconsistent results, repeat the test (in a loop) so you can see what it looks like the moment that it fails.

@mkatrenik

This comment has been minimized.

Copy link

commented Feb 15, 2018

So for others who may have same issue - I tried to run it in docker/browsers image headlessly and there it works. Looks like some difference between osx and linux electron... Anyway thanks for suggestions.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Feb 15, 2018

Different environments will stress the browser in slightly different ways - there are different network configurations, timings, etc.

From our experience, this is most likely an issue with the way you've written your test and will likely crop up even in the environments where its primarily passing.

@ndonnellan

This comment has been minimized.

Copy link
Author

commented Feb 15, 2018

I didn't respond to the closing, but transpiling my code to a lower ES version made my issue go away so I don't think your bug (@mkatrenik ) is the same.

@ryan-mulrooney

This comment has been minimized.

Copy link

commented May 28, 2018

I'm having a similar issue... When I run headlessly I incur essentially the same error. If i run via the CLI but use -b chrome the test fails again, but freezes when typing in an input (i have 4 inputs in an iframe, and it always fails when typing in one of them, a different one each time).

However, when I run the test in the test runner, it always passes. I guess there's some fundamental difference between running the test via the CLI compared to the test runner - is it a combination of the memory allocation between that, and the local version of my app that's under test @brian-mann ?

@juddfranklin1

This comment has been minimized.

Copy link

commented Jun 7, 2018

First of all, thanks for this great testing tool!

I am having the same issue.

My version of chrome is Version 66.0.3359.181 (Official Build) (64-bit)
I am running OSX High Sierra 10.13.5 (17F77)

When I flag chrome as the browser, all my tests pass. When I do not, most tests fail, because they rely upon a link click and electron's interpreter appears to be handling element positioning in unexpected ways.

Would it be possible to use a more stable release of chrome for headless tests? It seems that in-window and headless browser tests should be run in as similar an environment as possible, and ideally a stable one to ensure realistic results.

Error details below
(included for clarification. I understand that these reflect issues with the electron interpreter, not cypress.io)

Here's an error that sets off a chain of failed tests:

     CypressError: Timed out retrying: cy.trigger() failed because this element is not visible:

<a href="#show">show ev...</a>

This element '<a>' is not visible because it has CSS property: 'position: fixed' and its being covered by another element:

<div class="eventful-toggle-tab">...</div>

Fix this problem, or use {force: true} to disable error checking.

While a parent element of that link is position fixed, this is what Chrome's inspector gives me for the element in question:

.eventful-toggle-tab a {
    position: relative;
    z-index: 100000;
    color: white;
    text-decoration: none;
    transition: color 1s;
    -webkit-transition: color 1s;
    text-transform: capitalize;
}

The computed style is position: relative.

The containing element, which is being interpreted as "hiding" the link within it is <div class="eventful-toggle-tag">, and it has the following styles:

.eventful-toggle-tab {
    display: block;
    width: 80px;
    height: 60px;
    padding: 10px;
    position: absolute;
    text-align: center;
    background: #b1d;
    transition: background-color 1s;
    -webkit-transition: background-color 1s;
    border: 1px solid #b1d;
}

The element is intended to be a visible tab fixed to the edge of the screen, which, when clicked reveals the app.

The generated video is also quite perplexing, because the element appears as it should and it is not clear what, if anything, is actually hiding it. Here is a screenshot to illustrate. The purple box that says "show eventful" houses the tag in question.

screenshot 2018-06-07 15 02 14

I hope that these details help.

Thanks again!

@z0d14c

This comment has been minimized.

Copy link

commented Jun 23, 2018

I have a test that passes in electron/chrome (headless or no) on Mac OS but fails on headless ubuntu :(

@raelgc

This comment has been minimized.

Copy link

commented Aug 30, 2018

In my experience, most of these cases (when tests fails only in headless mode) it's due the faster speed.

Example: in my app, after the login, we have a request to get more info about the user, and only then store some data in local storage and cookies.

While testing in Chrome, or even with Electron, it works. Headless, it was randomly failing (because the request was not completed, so user info was not saved, redirecting to login again).

To fix, I just include a check for some element that should be displayed only after the user request is finished, like one of item of the logged user menu, like cy.contains('Home').

@predictiveXu

This comment has been minimized.

Copy link

commented Sep 12, 2018

I just found that running headless never waited when the page was loading. The issue for me is that running tests in chrome or headed only works for the first 5 tests. After that, the chrome or electron became very slow, and sometimes chrome would be broken and show the error message 'Aw, Snap'. RIP

@amiral-jion

This comment has been minimized.

Copy link

commented Jan 31, 2019

I'm having the same issue, the tests passed correctly in the with open and it fails sometimes (in an indeterministic way) with run

@Vindexus

This comment has been minimized.

Copy link

commented Mar 14, 2019

I am having the same issue as well.

@akosipax

This comment has been minimized.

Copy link

commented Apr 22, 2019

I had this issue. It was failing on cypress run as well as on cypress open if my browser is Electron. The test consistently passes on an open UI using Chrome/Canary even if I loop the test 2000 times.

I followed @brian-mann's advice and rewrote my tests so that it "waits" until the element I want to assert is visible. My test now consistently passes across all environments.

    cy.get(element).then(element => {
        const actual = element.text();
        const expected = "sometext";
        expect(actual).to.equal(expected);
      });

to

cy.get(element)
      .should("be.visible")
      .then(element => {
        const actual = element.text();
        const expected = "sometext";
        expect(actual).to.equal(expected);
      });
@vedmant

This comment has been minimized.

Copy link

commented Jul 15, 2019

I have the same issue with latest Cypress, is there an official workaround for this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.