Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel tests without recording on dashboard #2520

Closed
NatashaKramarenko opened this issue Sep 25, 2018 · 11 comments

Comments

9 participants
@NatashaKramarenko
Copy link

commented Sep 25, 2018

Current behavior:

Parallelisation is a very powerful feature, but in order to use it users are forced to use recording feature as well:
cypress run --record --key=abc123 --parallel

https://docs.cypress.io/guides/guides/parallelization.html

Desired behavior:

It would be great to be able to use parallelisation feature without recording.
I am sure most of the users already have some sort of recording organised onsite.

Versions

Cypress 3.1.0

@drewbrend

This comment has been minimized.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Nov 2, 2018

Allowing for automatic parallelization without recording to the Dashboard is not part of our Roadmap. Recording is a paid service that we offer and helps support the work we do on the open source test runner.

Edit by @jennifer-shehane on 4-22-19: Parallelization is not a paid feature anymore. Please check our pricing - every plan, including the free plan can run their tests in parallel.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Nov 2, 2018

@NatashaKramarenko also please check out our Guide on this here: https://docs.cypress.io/guides/guides/parallelization.html#Overview

You'll see from some of the technical implementation details that this is not possible to do (in the way we're doing it) without involving some sort of service.

It's not that we are choosing not to make this possible, it's just that it is simply not possible to do otherwise.

@traviscrist

This comment has been minimized.

Copy link

commented Nov 14, 2018

@brian-mann

You'll see from some of the technical implementation details that this is not possible to do (in the way we're doing it) without involving some sort of service.

It seems that the limitation is the Cypress Parallelization Orchestrator but I don't see why this could not be done by just evenly splitting the tests based on the # of parallel machines and then running them independently. It would not be quite as efficient but it would also not require us to rely on cypress.io being up for our tests to run. Every additional 3rd party added to a CI pipeline is another potential point of failure.

#2525 is an example of why relying on cypress.io being up for parallelization is not the best solution.

@brian-mann

This comment has been minimized.

Copy link
Member

commented Nov 14, 2018

@traviscrist you can already do this manually yourself - it would just be specific to your CI provider, configuring the available number of machines, and then telling Cypress what exactly to run. However, nothing is going to automagically aggregate all of the data for you in that manner.

Your argument for not using a saas provider based on whether its ever gone done for a period of time is fairly erroneous considering that we're talking on Github (which has gone down multiple times in recent memory), you're downloading a package from NPM (which also has outages), and unless you're using Jenkins and hosting it yourself, then you're likely also using a CI service that has likely also had outages. I'm afraid that's just the nature of the beast, you trade off a little bit of downtime for saving development hours, and increasing productivity overall across your organization. Different teams have different thresholds, but nothing is 100%.

@traviscrist

This comment has been minimized.

Copy link

commented Nov 14, 2018

it would just be specific to your CI provider, configuring the available number of machines, and then telling Cypress what exactly to run. However, nothing is going to automagically aggregate all of the data for you in that manner.

Thanks for the idea that's my current plan to try out.

Your argument for not using a saas provider based on whether its ever gone done for a period of time is fairly erroneous considering that we're talking on Github (which has gone down multiple times in recent memory), you're downloading a package from NPM (which also has outages), and unless you're using Jenkins and hosting it yourself, then you're likely also using a CI service that has likely also had outages. I'm afraid that's just the nature of the beast, you trade off a little bit of downtime for saving development hours, and increasing productivity overall across your organization. Different teams have different thresholds, but nothing is 100%.

Your point is totally valid, but with all the outages on major services lately I've started to consider every new saas as another failure point in the ci pipeline. It seems to me that if we are to consider using cypress as a saas then it needs to have a fallback mechanism to run tests sequentially if cypress.io is not available. If a fix to #2525 is implemented then there is a fallback in place which makes using cypress.io as a saas much more compelling since our ci pipeline will not fail completely if there is an outage on cypress.io.

@mjhea0

This comment has been minimized.

Copy link

commented Feb 19, 2019

@traviscrist here's an example of how I went about setting it up https://testdriven.io/blog/running-cypress-tests-in-parallel/

@charlex

This comment has been minimized.

Copy link

commented Apr 22, 2019

I think it should be made more explicit throughout the documentation that parallel builds are a paid feature so that people don't spend time researching it if they aren't interested in paying money.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

@charlex Parallelization is not a paid feature anymore. Please check our pricing - every plan, including the free plan which allows for 500 test recordings, can run their tests in parallel.

@coding-yogi

This comment has been minimized.

Copy link

commented Jun 19, 2019

Its more of a "try out" feature than free feature as per your pricing model, a cap of 500 on number of tests

@germyjen

This comment has been minimized.

Copy link

commented Jun 26, 2019

it's really more of a trial at only 500 tests a month... we use that in a little over a week with a dozen developers and one project. I've seen a few solutions online to run parallelization independent of the dashboard, and will be spending some time this week configuring it, after some trouble with dashboard today made me realize it's not enough value added to be worth it for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.