Any plans to run tests in parallel?

[scottdavis]

Currently working on a very timer based heavy application and my current test suite is taking ~ 200 seconds to run fully which isn't too horrible since I don't run the full thing every single cycle. I was just curious if there were plans to run tests concurrently to help speed up the wait time.

[brian-mann]

While you're working in the browser - no.

When running headlessly - yes.

Cypress can run each individual file in parallel, very easily.

This will only really speed up CI / running headlessly from the command line though.

If you are not mocking routes, your backend / database would also have to support parallelization else there would be data collisions.

So more thought / configuration needs to be put into this, but it's technically very feasible / do-able.

[brian-mann]

Also now with the new assertion syntax, you can try removing all of your explicit cy.wait calls, and that should speed things up. There's no reason to have to use those anymore.

[scottdavis]

Oh awesome is that the case now? or do i need to pass in the file name to the run command?

So the wait calls were for setting an expectation that n amount of time has passed dealing with a custom configurable timer (media player) and asserting that something happens after n seconds. So I'm not sure if removing them will provide me with a true test. I can probably remove the waits around the animations!

[brian-mann]

No - running tests in parallel isn't supported just yet. It complicates the reporter, and there needs to be configuration around this specific behavior, but it's very easy to do, as in 1-2 days of work. I did a proof of concept for this several months ago and it worked great.

If you can paste your code around why you're using cy.wait I can tell you if they're unnecessary or not. As long as you "guard" your commands with assertions, Cypress will automatically wait until those assertions become true.

[scottdavis]

   it 'should auto advance when modal is closed if autoplay is true', ->
      cy
        .visit '/'
        # Assert we are on a clean state
        .get('#gallery .display-pane:nth-child(1)').should('not.have.class', 'ng-hide')
        # Open modal
        .click force: true
        .get('.fancybox-overlay .display-pane')
        # Close modal
        .closeFancybox()
        # Wait the timers duration
        .wait(3100)
        # Assert the media player changed media
        .get('#gallery .display-pane:nth-child(1)').should('have.class', 'ng-hide')
        .get('#gallery .display-pane:nth-child(2)').should('not.have.class', 'ng-hide')

[brian-mann]

Cypress will retry: .get('#gallery .display-pane:nth-child(1)').should('have.class', 'ng-hide') up to 4 seconds by default.

You can pass a timeout option to further increase this:

.get('#gallery .display-pane:nth-child(1)', {timeout: 8000}).should('have.class', 'ng-hide')

As long as the class ng-hide acts as your "guard" - and by guard I mean --- "if that is THE thing that changes after fancyBox closes", then you should be able to remove the wait.

[scottdavis]

yea i think it's apples and oranges ill switch to timeouts but the test is still going to wait at least 3000 ms for the state to change :) Thanks for your time!

[brian-mann]

That's true, if it takes 3 seconds before your class is added, there's no getting around that - BUT not using cy.wait is vastly superior because you're not coupling your tests to your implementation.

What I mean is - you could change your app's code, for instance by reducing the timer from 3 seconds to 1 second, without touching any of your test code - the test would literally just pass much faster.

Passing a {timeout: N} option is different than cy.wait because it's saying to retry UP TO a certain amount of time, verses waiting for an explicit amount of time, every time.

[brian-mann]

Reopening until this is done for CI and the CLI.

[cristopher-rodrigues]

@brian-mann Any prediction?

[cristopher-rodrigues]

@brian-mann how can I do this: This will only really speed up CI / running headlessly from the command line though. (?) by correctly way?

[lukemadera]

@brian-mann I've been loving using Cypress - thanks! I saw the option to specify ONE test file, but how do you specify more than one (e.g. an array)? So we could break up our tests to run in parallel (e.g. 2 groups - each of half the spec files, then as our number of tests grows, break up into more groups, so each group runs in parallel with no more than X tests each)?

Obviously full parallel support would be ideal (any ETA on that?) but if we can specify a set of files from the CLI (or even better, point to a config that holds the array of files for that run), that should be all we need to get a minimal parallel set of cypress tests running.

[brian-mann]

You can only parallelize browser tests at the operating system level. Users currently achieve this right now using docker. They spin up multiple docker containers and run different spec files.

Even if you were to spawn multiple browsers simultaneously - this would change their behavior. Only one application can have focus and browsers behave differently when they are not focused.

Currently there isn't a way to specify an array of spec files.

One of our users has an example of parallelizing tests here: https://docs.cypress.io/docs/userland-extensions

[brian-mann]

After re-reading my comments above from 2015 - I wanted to clarify my position:

The way Cypress works has changed a lot since then, and there would have to be considerable changes to the underlying architecture to support parallelized browsers in a single operating system.

Today the only way to get consistent behavior is by running separate OS instances each with their own instance of Cypress running.

It still is technically possible to do this, but we have no plans to support this or even attempt to solve this problem any time soon.

As to your point - it would be possible for us to update the command line to accept an array of spec files. There would all run serially but that's a fairly small amount of work.

[lukemadera]

Thanks @brian-mann for the prompt reply. And yes, what I'm really asking about is #416 that you just created - even if we run on multiple machines, we need a way to specify which test files to run.

[jennifer-shehane]

See #681 to read our proposal for supporting parralelization.

[bsell93]

Just as a "quick" solution you could do the following command
find ./cypress/integration -name "*.spec.js" | xargs -n1 -P2 yarn cypress run --spec
You can change the "P" value to change the number of spec files running at once.

If you use a reporter I also found a way to get that to output correctly. In my case I am conforming to Bitbucket pipelines test result output outlined here https://confluence.atlassian.com/bitbucket/test-reporting-in-pipelines-939708543.html.
find ./cypress/integration -name "*.spec.js" | xargs -n1 -P2 sh -c 'yarn cypress run --reporter junit --reporter-options "mochaFile=test-reports/$RANDOM.xml" --spec "$@"' --

[chit786]

@bsell93 do you mean in same OS (with multiple terminal/ command windows) your solution can work ?

Its interesting to see how cypress reporter can behave if this is the case.

[bsell93]

@chit786 I'm not entirely sure what you mean, but I ran the command in macos 10.12.6 and in the same terminal window. I imagine you should be able to run it in multiple terminal windows, but I'm not sure what advantage that would bring as compared to simply changing the "P" value.

[chit786]

@bsell93 we have separate folders inside cypress/integration folder in our project referring to separate modules/ functionalities so my question refers to that usecase.. :) anyways I will try this out to see how it looks like in action.. thanks!

[bsell93]

@chit786 ah! Yeah we do as well. That command will work recursively to find all that match the glob, so sub-directories tests will be ran as well.

[jennifer-shehane]

Closing in favor of more robust proposal here: #1690

Please follow that issue, it is close to being released. 👍