Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`cy.visit` in 3.3.1 is 2x-4x as slow as 3.2.0 #4313

Closed
fr0 opened this issue May 24, 2019 · 37 comments

Comments

8 participants
@fr0
Copy link

commented May 24, 2019

Current behavior:

Running my tests in 3.2.0 takes 1 hour 54 minutes.
Running my tests in 3.3.1 takes 4 hours 41 minutes.

(This is running on headless electron using cypress.run with the module API; I haven't measured it with cypress open).

Both cases are running on the same WIndows 10 VM, and not using parallelization.

Desired behavior:

I'd like my tests to take less than 2 hours.

Versions

Cypress 3.3.1

@fr0

This comment has been minimized.

Copy link
Author

commented May 24, 2019

I tried downgrading from 3.3.1 to 3.2.0 and my tests once again ran faster (under 2 hours).

@flotwig

This comment has been minimized.

Copy link
Member

commented May 24, 2019

Hey @fr0, are you using the built-in Electron browser? Do you have a proxy set up?

@fr0

This comment has been minimized.

Copy link
Author

commented May 24, 2019

Thanks for the quick response. I am using the built-in Electron browser, but I don't have a proxy set up.

@flotwig

This comment has been minimized.

Copy link
Member

commented May 24, 2019

This could be a related issue to #4298 - does your app rely on any kind of non-200 status code, any redirects or anything like that?

@fr0

This comment has been minimized.

Copy link
Author

commented May 24, 2019

It doesn't, but that's a good thought. It does make heavy use of WebSockets. Also, my tests rely on some setup pre-work that calls that WebSocket API from the beforeEach handler (and I use cy.wrap + cypress pipe for this).

I'm not sure if any of that is related, though.

Is there any additional logging I can turn on that would help to debug this issue? I'd love to send you guys the code so you can reproduce it, but unfortunately it's all proprietary (including the back-end).

@fr0

This comment has been minimized.

Copy link
Author

commented May 24, 2019

I guess technically the WebSocket upgrade is a 101, so the answer to your question may actually be "yes".

@flotwig

This comment has been minimized.

Copy link
Member

commented May 24, 2019

You can get full debug logs by following these steps and share it to shed some light on the issue: https://docs.cypress.io/guides/guides/debugging.html#Print-DEBUG-logs

That will help to debug the issue. Make sure to scrub any sensitive data before sharing them.

@fr0

This comment has been minimized.

Copy link
Author

commented May 24, 2019

This is just with a single spec file, but it's more than enough to show off the difference (59 seconds in 3.2 vs 114sec in 3.3.1).

debug-cypress-3.2.0.log
debug-cypress-3.3.1.log

@exan-atsui

This comment has been minimized.

Copy link

commented Jun 3, 2019

We've updated to 3.3.1 from 3.1.5 and we're also seeing this behavior. Things are much slower when running our tests in run.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Jun 3, 2019

We're triangulating a few different issues in the 3.3.x releases and several may all be related to the same underlying problem. Likely will have a patch in the next few days and then we can go from there.

@nthum

This comment has been minimized.

Copy link

commented Jun 4, 2019

We are seeing same behavior when upgraded from 3.2.0

@ChuckOp

This comment has been minimized.

Copy link

commented Jun 4, 2019

This is a major problem for us.

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

@nthum @ChuckOp @exan-atsui Hey, thanks for letting us know. Can you provide some additional details to help us debug? What operating system are you using? Are you on Cypress 3.3.1 or 3.3.0? Are you behind a proxy? What browser are you using?

@nthum

This comment has been minimized.

Copy link

commented Jun 4, 2019

@flotwig We are running 3.3.1 on Windows 10 & using headless electron. We don't have proxy

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 4, 2019

I am fairly certain that this is not OS-specific, as I've seen it both on Windows 10 and Mac OS.

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

I wrote a small test case to benchmark cy.visit:

describe '', ->
  it 'benchmark cy.visit', ->
    Cypress._.times 100, ->
      cy.visit('/index2.html')

    cy.wrap ->

In Cypress 3.2.0, this executes in 17.25 seconds.

In Cypress 3.3.1, this executes in 61.58 (!) seconds.

So it does look like the issue is with cy.visit(). I recorded a short video of the benchmark which shows that every few cy.visit()s, it seems to hang for a couple of seconds before continuing: https://drive.google.com/file/d/15a26L9dC9EQHFJKd-hQfvxkxMnFgmPmN/view?usp=sharing

Looks like #4349 is a duplicate of this issue, then. I'll close that issue and log future progress on tracking down the cy.visit slowness here.

@flotwig flotwig changed the title 3.3.1 is twice as slow as 3.2.0 `cy.visit` in 3.3.1 is 2x-4x as slow as 3.2.0 Jun 4, 2019

@flotwig flotwig self-assigned this Jun 4, 2019

@bahmutov

This comment has been minimized.

Copy link
Collaborator

commented Jun 4, 2019

@danceric0919

This comment has been minimized.

Copy link

commented Jun 11, 2019

Hi
is there a time line to fix this issue?
we are suffering from this for couple days.

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 11, 2019

@danceric0919 We have a fix in place, it will be out with the next patch release (3.3.2).

@cypress-bot

This comment has been minimized.

Copy link

commented Jun 17, 2019

The code for this is done in cypress-io/cypress#4385, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 18, 2019

Would it be possible to create a beta release in npm so I can verify that the fix addresses my issue?

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 18, 2019

@fr0 You can use the latest beta build for Windows:

In Git bash:

CYPRESS_INSTALL_BINARY=https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip npm i https://cdn.cypress.io/beta/npm/3.3.2/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.tgz

In other shells, you'll need cross-env to set the environment variable:

npm i -g cross-env
cross-env CYPRESS_INSTALL_BINARY=https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip npm i https://cdn.cypress.io/beta/npm/3.3.2/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.tgz
@fr0

This comment has been minimized.

Copy link
Author

commented Jun 18, 2019

I tried that, and at first it looked promising:

⚠ Warning: Forcing a binary version different than the default.

  The CLI expected to install version: 3.3.2

  Instead we will install version: https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip

  These versions may not work properly together.

Installing Cypress (version: https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip)

 ✔  Downloaded Cypress
 ✔  Unzipped Cypress
 ✔  Finished Installation /Users/crfrolik/Library/Caches/Cypress/3.3.2

You can now open Cypress by running: node_modules/.bin/cypress open

But then when I actually tried to run...

yarn run v1.16.0
$ /Users/crfrolik/Documents/atlas-dev-tools-proto/node_modules/.bin/cypress open
No version of Cypress is installed in: /Users/crfrolik/Library/Caches/Cypress/3.3.2/Cypress.app

Please reinstall Cypress by running: cypress install

----------

Cypress executable not found at: /Users/crfrolik/Library/Caches/Cypress/3.3.2/Cypress.app/Contents/MacOS/Cypress

----------

Platform: darwin (17.7.0)
Cypress Version: 3.3.2
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 18, 2019

Oh! I thought you were running on Windows, sorry. Here's the correct command for macOS:

CYPRESS_INSTALL_BINARY=https://cdn.cypress.io/beta/binary/3.3.2/darwin-x64/circle-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-123895/cypress.zip npm i https://cdn.cypress.io/beta/npm/3.3.2/circle-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-123894/cypress.tgz

You might have to delete /Users/crfrolik/Library/Caches/Cypress/3.3.2 to get rid of the Windows package.

I know it's kinda a mess, we're working on making this more accessible to end-users.

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 18, 2019

npm lets you tag a release as "pre-release", so users won't get it unless they specifically opt in. It makes this sort of thing very easy. :)

@bahmutov

This comment has been minimized.

Copy link
Collaborator

commented Jun 18, 2019

yeah @fr0 , we just don't want to pollute NPM with 10k builds of Cypress per month (like TypeScript), and that's why we take advantage of static folders to install pre-release builds. But we know some tricks to make it simpler, just did not get to implement them yet

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 18, 2019

You definitely don't want to publish every single CI build as a pre-release to npm, true.

Back on-topic: At first glance, the nightly build looks like it did not fix my issue. I'll try again on a different environment to be sure.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Jun 19, 2019

@fr0 i think the PR that's going to fix your issue hasn't been merged into develop yet. This is likely the one https://github.com/cypress-io/cypress/pull/4358/files

When you see that get merged in then you can see if it fixes your problem.

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 19, 2019

@fr0 i think the PR that's going to fix your issue hasn't been merged into develop yet. This is likely the one https://github.com/cypress-io/cypress/pull/4358/files

When you see that get merged in then you can see if it fixes your problem.

Thanks, I'll take a look. Is there a way to get the cdn beta link for a given develop commit?

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 19, 2019

Right now you have to pull it out of CI:

  1. Go to the "Mac build" CI stage:
    image
  2. Click on "mac" to get to the workflow:
    image
  3. The package CDN url is in Mac NPM package, the binary URL is in Mac binary:
    image
  4. In Mac NPM package, the package URL is in the "upload NPM package" step:
    image
  5. In Mac binary, the binary URL is in the "upload unique binary" step:
    image
  6. Then it's just CYPRESS_INSTALL_BINARY=<binary .zip URL> npm install <package .tgz URL>

The process is similar for Linux (also on Circle CI) and for Windows (except you need to go to Appveyor for the Windows builds)

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 20, 2019

I tried the latest build (from c7b340) and the issue is not resolved.

EDIT: Looks like I did get back some performance, but it's definitely still slower than 3.2.0.

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 20, 2019

@fr0 Could you share spec code that reproduces the slowness that changed between 3.2.0 and 3.3.2? We had some test cases where we improved test performance (like the 100x cy.visit test I posted earlier in this thread, which now runs as fast as it did in 3.2.0), but it's possible that we missed an edge case that you're still running in to.

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 20, 2019

Sure, I'll work on trying to get a reproducible example.

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 21, 2019

I was able to reproduce the problem with the most trivial example I can think of that uses Angular + Cypress.

https://github.com/fr0/cypress-slowdown

Here's the spec file. https://github.com/fr0/cypress-slowdown/blob/master/cypress/integration/sample.spec.ts

(It repeats 50 times just to get a bigger sample set.)

Note that the problem isn't in cy.visit. In testing with my app before I created this trivial example, I found that pretty much every cypress command was slower in 3.3.2 than in 3.2.0.

(Note: I created a cy.task that measures timing using process.hrtime(). If this is misguided, or if there is a better way, I'd love to be clued in.)

Thanks!

@flotwig

This comment has been minimized.

Copy link
Member

commented Jun 21, 2019

@fr0 Thank you for the example repo, I see exactly what you mean now. After some debugging, I think you're having the same issue as #4411, check out this comment to see what I mean. If so we can move future discussion of this non-cy.visit() slowdown over there.

@fr0

This comment has been minimized.

Copy link
Author

commented Jun 21, 2019

Excellent, thanks for the update. Glad you were able to reproduce it with my repo.

FYI, I (any many others) are not able to roll back to 3.2.0 because it isn't compatible with Angular 8 (due to #2069 + TypeStrong/ts-loader#929), so that's not an option right now.

I'll continue discussion in the issue you linked. Thanks again!

@cypress-bot

This comment has been minimized.

Copy link

commented Jun 27, 2019

Released in 3.3.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.