New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Any plans to run tests in parallel? #64

Closed
scottdavis opened this Issue Sep 16, 2015 · 22 comments

Comments

7 participants
@scottdavis

scottdavis commented Sep 16, 2015

Currently working on a very timer based heavy application and my current test suite is taking ~ 200 seconds to run fully which isn't too horrible since I don't run the full thing every single cycle. I was just curious if there were plans to run tests concurrently to help speed up the wait time.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Sep 16, 2015

While you're working in the browser - no.

When running headlessly - yes.

Cypress can run each individual file in parallel, very easily.

This will only really speed up CI / running headlessly from the command line though.

If you are not mocking routes, your backend / database would also have to support parallelization else there would be data collisions.

So more thought / configuration needs to be put into this, but it's technically very feasible / do-able.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Sep 16, 2015

Also now with the new assertion syntax, you can try removing all of your explicit cy.wait calls, and that should speed things up. There's no reason to have to use those anymore.

@scottdavis

This comment has been minimized.

scottdavis commented Sep 16, 2015

Oh awesome is that the case now? or do i need to pass in the file name to the run command?

So the wait calls were for setting an expectation that n amount of time has passed dealing with a custom configurable timer (media player) and asserting that something happens after n seconds. So I'm not sure if removing them will provide me with a true test. I can probably remove the waits around the animations!

@brian-mann

This comment has been minimized.

Member

brian-mann commented Sep 16, 2015

No - running tests in parallel isn't supported just yet. It complicates the reporter, and there needs to be configuration around this specific behavior, but it's very easy to do, as in 1-2 days of work. I did a proof of concept for this several months ago and it worked great.

If you can paste your code around why you're using cy.wait I can tell you if they're unnecessary or not. As long as you "guard" your commands with assertions, Cypress will automatically wait until those assertions become true.

@scottdavis

This comment has been minimized.

scottdavis commented Sep 16, 2015

   it 'should auto advance when modal is closed if autoplay is true', ->
      cy
        .visit '/'
        # Assert we are on a clean state
        .get('#gallery .display-pane:nth-child(1)').should('not.have.class', 'ng-hide')
        # Open modal
        .click force: true
        .get('.fancybox-overlay .display-pane')
        # Close modal
        .closeFancybox()
        # Wait the timers duration
        .wait(3100)
        # Assert the media player changed media
        .get('#gallery .display-pane:nth-child(1)').should('have.class', 'ng-hide')
        .get('#gallery .display-pane:nth-child(2)').should('not.have.class', 'ng-hide')
@brian-mann

This comment has been minimized.

Member

brian-mann commented Sep 16, 2015

Cypress will retry: .get('#gallery .display-pane:nth-child(1)').should('have.class', 'ng-hide') up to 4 seconds by default.

You can pass a timeout option to further increase this:

.get('#gallery .display-pane:nth-child(1)', {timeout: 8000}).should('have.class', 'ng-hide')

As long as the class ng-hide acts as your "guard" - and by guard I mean --- "if that is THE thing that changes after fancyBox closes", then you should be able to remove the wait.

@scottdavis

This comment has been minimized.

scottdavis commented Sep 16, 2015

yea i think it's apples and oranges ill switch to timeouts but the test is still going to wait at least 3000 ms for the state to change :) Thanks for your time!

@scottdavis scottdavis closed this Sep 16, 2015

@brian-mann

This comment has been minimized.

Member

brian-mann commented Sep 16, 2015

That's true, if it takes 3 seconds before your class is added, there's no getting around that - BUT not using cy.wait is vastly superior because you're not coupling your tests to your implementation.

What I mean is - you could change your app's code, for instance by reducing the timer from 3 seconds to 1 second, without touching any of your test code - the test would literally just pass much faster.

Passing a {timeout: N} option is different than cy.wait because it's saying to retry UP TO a certain amount of time, verses waiting for an explicit amount of time, every time.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Sep 16, 2015

Reopening until this is done for CI and the CLI.

@cristopher-rodrigues

This comment has been minimized.

cristopher-rodrigues commented Jan 13, 2017

@brian-mann Any prediction?

@cristopher-rodrigues

This comment has been minimized.

cristopher-rodrigues commented Jan 31, 2017

@brian-mann how can I do this: This will only really speed up CI / running headlessly from the command line though. (?) by correctly way?

@lukemadera

This comment has been minimized.

lukemadera commented Feb 7, 2017

@brian-mann I've been loving using Cypress - thanks! I saw the option to specify ONE test file, but how do you specify more than one (e.g. an array)? So we could break up our tests to run in parallel (e.g. 2 groups - each of half the spec files, then as our number of tests grows, break up into more groups, so each group runs in parallel with no more than X tests each)?

Obviously full parallel support would be ideal (any ETA on that?) but if we can specify a set of files from the CLI (or even better, point to a config that holds the array of files for that run), that should be all we need to get a minimal parallel set of cypress tests running.

@brian-mann

This comment has been minimized.

Member

brian-mann commented Feb 7, 2017

You can only parallelize browser tests at the operating system level. Users currently achieve this right now using docker. They spin up multiple docker containers and run different spec files.

Even if you were to spawn multiple browsers simultaneously - this would change their behavior. Only one application can have focus and browsers behave differently when they are not focused.

Currently there isn't a way to specify an array of spec files.

One of our users has an example of parallelizing tests here: https://docs.cypress.io/docs/userland-extensions

@brian-mann

This comment has been minimized.

Member

brian-mann commented Feb 7, 2017

After re-reading my comments above from 2015 - I wanted to clarify my position:

The way Cypress works has changed a lot since then, and there would have to be considerable changes to the underlying architecture to support parallelized browsers in a single operating system.

Today the only way to get consistent behavior is by running separate OS instances each with their own instance of Cypress running.

It still is technically possible to do this, but we have no plans to support this or even attempt to solve this problem any time soon.

As to your point - it would be possible for us to update the command line to accept an array of spec files. There would all run serially but that's a fairly small amount of work.

@lukemadera

This comment has been minimized.

lukemadera commented Feb 8, 2017

Thanks @brian-mann for the prompt reply. And yes, what I'm really asking about is #416 that you just created - even if we run on multiple machines, we need a way to specify which test files to run.

@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Oct 10, 2017

See #681 to read our proposal for supporting parralelization.

@bsell93

This comment has been minimized.

bsell93 commented Apr 4, 2018

Just as a "quick" solution you could do the following command
find ./cypress/integration -name "*.spec.js" | xargs -n1 -P2 yarn cypress run --spec
You can change the "P" value to change the number of spec files running at once.

If you use a reporter I also found a way to get that to output correctly. In my case I am conforming to Bitbucket pipelines test result output outlined here https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html.
find ./cypress/integration -name "*.spec.js" | xargs -n1 -P2 sh -c 'yarn cypress run --reporter junit --reporter-options "mochaFile=test-reports/$RANDOM.xml" --spec "$@"' --

@chit786

This comment has been minimized.

chit786 commented Apr 9, 2018

@bsell93 do you mean in same OS (with multiple terminal/ command windows) your solution can work ?

Its interesting to see how cypress reporter can behave if this is the case.

@bsell93

This comment has been minimized.

bsell93 commented Apr 9, 2018

@chit786 I'm not entirely sure what you mean, but I ran the command in macos 10.12.6 and in the same terminal window. I imagine you should be able to run it in multiple terminal windows, but I'm not sure what advantage that would bring as compared to simply changing the "P" value.

@chit786

This comment has been minimized.

chit786 commented Apr 9, 2018

@bsell93 we have separate folders inside cypress/integration folder in our project referring to separate modules/ functionalities so my question refers to that usecase.. :) anyways I will try this out to see how it looks like in action.. thanks!

@bsell93

This comment has been minimized.

bsell93 commented Apr 9, 2018

@chit786 ah! Yeah we do as well. That command will work recursively to find all that match the glob, so sub-directories tests will be ran as well.

@jennifer-shehane jennifer-shehane added this to the 3.1.0 milestone Jul 18, 2018

@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Jul 18, 2018

Closing in favor of more robust proposal here: #1690

Please follow that issue, it is close to being released. 👍

@cypress-io cypress-io locked as resolved and limited conversation to collaborators Dec 6, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.