Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Support for Cross Browser Testing #310

Open
brian-mann opened this Issue Nov 24, 2016 · 95 comments

Comments

@brian-mann
Copy link
Member

brian-mann commented Nov 24, 2016

For more detailed issue tracking, check the progress on this issue: #3207

UPDATES: Oct 20th, 2018

I gave more updates on the work we've been doing for cross browser support and other features on a webinar we gave.

https://youtu.be/FfqD1ExUGlw?t=2079

UPDATES: Sept 8th, 2018

Cross browser testing is now a front and center major focus here at Cypress. We've been experimenting over the last few months and have a working implementation to add native event support for Chrome, Firefox, Edge, and IE11. We also believe adding Mobile Safari and Android should also be straightforward - with Safari actually being the most difficult to get.

The architecture by which we are doing cross browser testing is a best-of-breed design where we tap into the native API's for Chrome and Firefox (avoiding the use of webdriver) and for Edge and IE11 we only use a minimal set of Webdriver API's - none of which are the traditional sendKeys or value endpoints, which avoids the network latency problem entirely. We plan to eventually publish this architecture so people understand how Cypress is different than Webdriver. None of the other core components of the Cypress architecture are affected by this change, it is simply how to enable native events in a consistent manner across all browsers without introducing the latency / flaky / non-determinism of Webdriver.

In additional to all these things, we are also adding Sauce Labs support (at a minimum) and will also have a proof of concept prepared for that.

As we add support for native events we are subsequently adding support for the browsers at the same time to ensure all of our algorithms are normalized correctly. However, you'll likely first see us release native event support in Chrome + Firefox, followed by IE11 sometime after that.

...below is the original post from 2016...

Does Cypress support cross browser testing?

Currently the answer is: No

Right now, we only support Chrome* variants such as Chrome, Chromium, and Canary.

Will you ever support other browsers?

Yes.

We will be able to support other Desktop Browsers and Mobile Browsers (using Webviews).

We have built Cypress from day 1 with cross browser testing in mind, and are very conscientious to ensure the API's we write today will work in the future.

The Technical Details

Why do you only support Chrome today?

  • Cypress taps into the underlying browser Automation API's of Chrome to do things like modify cookies, take screenshots, focusing the browser window, etc.
  • We also launch the browser using very specific command line flags to turn off things that get in the way of automated testing.
  • We have synchronized these Automation API's when Cypress runs headlessly (in Electron). This means running headlessly behaves identically to a real headed Chrome browser.

What do you mean by "Automation API's"?

  • Automation API's describe the functions that a browser exposes to do things outside of the Javascript Sandbox.
  • Currently Cypress comes with its own universal Driver which implements most of the cy commands using Javascript - in the same run loop as your application. This is superior in the sense that Cypress can understand, know, wait for and modify its behavior based on how your application reacts to its commands in real time.
  • The downside is its impossible to fully automate a browser entirely in Javascript. Javascript is a security sandbox which restricts what you can do.
  • To fully automate a browser you must extend beyond the sandbox and tap into the underlying Automation API's which do not have these restrictions.

What will it take to support cross browsers?

  • We have to port commands which need to extend outside of the sandbox to each other browser we are attempting to support.

How will you port these Automation API's?

  • Either manually by hand
  • Or by using the existing browser drivers

What do you mean by "existing browser drivers"?

  • Each browser has built its own driver to the Webdriver Spec.
  • Here are some examples of drivers:
  • Google Chrome Driver (ChromeDriver)
  • Micosoft Edge Driver
  • SafariDriver
  • Mozilla GeckoDriver (Firefox)
  • Internet Explorer Driver
  • Appium

Why would you port these by hand?

  • Because using the existing drivers comes with a host of problems.

What's wrong with the existing browser drivers?

  • They are built for and thus limited to the Webdriver spec.

What is the Webdriver spec?

  • It's a HTTP API spec which defines a common interface to drive all browsers consistently.
  • It enables developers to abstract away the details of how each different browser implements its Automation API's

What is the problem with Webdriver?

  • It's slow
  • It's async
  • It's stateless
  • It's exposed over an HTTP server
  • It's a lowest common denominator
  • It's limited in scope and features
  • There are small inconsistencies
  • There are various driver bugs

What do you mean by "lowest common denominator"?

  • The spec is essentially limited to only including "all the things that every browser can do".
  • This leaves a tremendous amount of power and capability off the table that more modern browsers provide.

Can you give me some examples?

  • In Webdriver you cannot clear all cookies for all domains - you can only clear cookies for the current domain context under test.
  • There is no way to tell the browser to clear its cache.
  • Chromedriver cannot run tests with Dev Tools open.
  • Oftentimes the drivers may miss elements you're trying to interact with.
  • You cannot easily control the application under test.
  • You don't have native access to window, you're limited to stringifying objects over the wire.

Does Cypress currently use Webdriver?

  • No. We wrote our own universal driver.

What would be the benefit of using Webdriver?

  • We could possibly use the existing drivers to port over various commands without having to write the manual integrations by hand.
  • This would allow us to support cross browsers much faster.

If you use Webdriver won't Cypress also inherit these limitations?

  • In most cases - no.
  • In some cases - possibly.

How can you avoid these limitations?

  • By using our universal driver to do as much automation as possible and only go outside of the browser for a limited set of commands.
  • The whole point of Cypress is to exceed the boundaries of the current testing ecosystem.
  • Because we automate the browser with a completely different architecture we are in a position to extend beyond the Webdriver Spec and also normalize its behavior consistently across all browsers.

Can you give me an example of this?

  • Cypress can clear all cookies of all domains instead of just the current domain under test.
  • We perform work outside of the browser but then synchronize it seamlessly like with cy.request.
  • We have many commands which work outside of the browser which don't need to be ported.

But won't you still be limited by Webdrivers flakiness?

  • No. Even if we were to issue native events through Webdriver, because your application is always tested through Cypress, that means Cypress can respond synchronously to events that take place, and modify its behavior accordingly.
  • The vast majority of commands do not ever have to leave the browser anyway.

Can you give me an example of what you're talking about?

  • Imagine this scenario: you have a modal opening which animates in, and you want to click the "Submit" button.
  • In Webdriver, because its an HTTP API which often involves multiple network hops, there is usually a significant delay.
  • Let's assume you're attempting to issue a native "click" to the submit button.
  • When you issue the "click" command, the browser driver calculates the center coordinates of the button at say: {clientX: 50, clientY: 100}.
  • It then issues the native click event with those coordinates.
  • By the time your application "receives" this event, it could possibly "miss" the button altogether.
  • Your test fails because your button was never clicked, and the modal was never dismissed.
  • What happened?
  • Because the modal was animating it, its coordinates were changing rapidly. Webdriver has no idea or concept of this, it only takes a coordinate snapshot at the moment in time and because it asynchronously dispatches the native event, it misses.

Why isn't this a problem with Cypress?

  • Because Cypress is in the same run loop as your application, its impossible for it to "miss".
  • It can synchrously run calculations, and it already automatically figures out if an element is animating beyond a certain threshold - and therefore wait until it stops.
  • If it did issue a native event, it could detect that this event has yet to happen, and if the element moved from the coordinate, immediately attempt to recover by either canceling the click event or issuing another click at new coordinates.

When do you think you'll support other browsers?

  • We really don't know, its not a priority for us right now.

Why isn't it a priority? This is super important!

  • We disagree, cross browser testing is not as important as you think it is.

...

The reasoning and examples behind this are a WIP. We'll be updating this doc when they are ready.

@ls-guillaume-lambert

This comment has been minimized.

Copy link

ls-guillaume-lambert commented May 25, 2017

Hey @brian-mann!

First, amazing job on Cypress. I've been researching the best tool out there for JavaScript e2e testing and Cypress is up there with TestCafé for me. The only thing really that prevents me to go full on with Cypress is the cross-browser testing capabilities.

Looking at the above comment, I understand that Cypress does not consider cross-browser testing to be important. Is there anything anywhere that can convince me that I should be thinking the same way? Various OS/Browser combinations can yield different results, and I still think the value of automated cross-browser e2e testing is valuable.

In any case, this is really the only thing that prevents me from choosing Cypress as my weapon of choice for e2e testing, and I wanted to have you updated opinion on this since this issue was opened about 6 months ago.

Is there a way we can talk about this through DM? I'd really love to discuss this with you or any Cypress team member.

Thanks in advance and keep up the amazing work!

@ls-guillaume-lambert

This comment has been minimized.

Copy link

ls-guillaume-lambert commented May 30, 2017

For those interested in knowing how the Cypress team/devs feel about this, head over to there Gitter chat and search for "cross-browser". @brian-mann took the time to give me a lot of details on this, and he also told me he would update this issue to provide more context / examples / more compelling argument.

@shcallaway

This comment has been minimized.

Copy link

shcallaway commented Sep 6, 2017

@brian-mann

anyway we will be adding other browsers
firefox and edge

Could you provide any updates on support for these browsers? Is it in development? Also, a few people mentioned support in CI for other browsers via Sauce Labs and/or Browserstack. What's the status there?

Love Cypress! 😃

@brian-mann brian-mann changed the title Support for Cross Browser Testing Proposal: Support for Cross Browser Testing Sep 23, 2017

@tonypee

This comment has been minimized.

Copy link

tonypee commented Oct 6, 2017

Seconded, an update on this would be great! thanks

@brian-mann

This comment has been minimized.

Copy link
Member Author

brian-mann commented Oct 6, 2017

It's the next project one of our team members is going to be working on. We have the implementation plan ready to go.

@bahmutov

This comment has been minimized.

Copy link
Collaborator

bahmutov commented Oct 6, 2017

Who is that team member @brian-mann ? #PleasePickMe

@brian-mann

This comment has been minimized.

Copy link
Member Author

brian-mann commented Oct 6, 2017

Nope sorry its gonna be @chrisbreiding

You gotta finish parallelization and load balancing ;-)

@joegaffey

This comment has been minimized.

Copy link

joegaffey commented Oct 12, 2017

Cypress looks fantastic but without cross browser support I'll have to stick with Selenium.

Firefox then Edge would be my preferred priority.

Any possibility for iOS Safari support?

@vvo

This comment has been minimized.

Copy link

vvo commented Oct 17, 2017

We disagree, cross browser testing is not as important as you think it is.

I am also interested in having more details about this. I don't think unit testing for cross browsers is important. But for e2e I always thought that yes this was important.

@brian-mann

This comment has been minimized.

Copy link
Member Author

brian-mann commented Oct 17, 2017

We will be adding cross browser support likely by EOY. @chrisbreiding next project is that.

The support will be:

  • Firefox
  • Safari
  • Edge

No IE support is planned. It is substantially more difficult.

@brian-mann

This comment has been minimized.

Copy link
Member Author

brian-mann commented Oct 17, 2017

We are planning on writing up a full blog post on what we think e2e testing should look like, and how cross browser fits into that. It's going to require some pretty graphics ;-)

@danieldram

This comment has been minimized.

Copy link

danieldram commented Nov 24, 2017

This is an amazing solution, but without IE, FF and Safari support, I can't get buy in from my office to use this :(

@joelgrimberg

This comment has been minimized.

Copy link

joelgrimberg commented Dec 9, 2017

great stuff you guys are doing with Cypress!
is connectivity with SauceLabs/CrossBrowserTesting/Browserstack also considered?

@nrxsmith

This comment has been minimized.

Copy link

nrxsmith commented Dec 11, 2017

Do you guys have an update on your progress with this? Any ETA?

Thanks!

@brian-mann

This comment has been minimized.

Copy link
Member Author

brian-mann commented Dec 11, 2017

@chrisbreiding next project is this - starting this month. He's finishing up the thing he's been working on now.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

jennifer-shehane commented Dec 19, 2017

Support for Firefox browser proposal: #1096

@makenova

This comment has been minimized.

Copy link

makenova commented Dec 20, 2017

@brian-mann you mentioned a blog post in this thread.

We are planning on writing up a full blog post on what we think e2e testing should look like, and how cross browser fits into that. It's going to require some pretty graphics ;-)

May I have a link to it?

@green-arrow

This comment has been minimized.

Copy link

green-arrow commented Jan 3, 2018

@brian-mann - I’m also very interested in a blog post about the rationale behind cross-browser testing not being as important. I read a few comments in the gitter channel about how modern browsers largely resolve a lot of cross-browser bugs, but I just don’t buy it. There are still huge inconsistencies between browsers with modern features like flex box. Chrome, Firefox, and Safari each have their own quirks with just this one feature that our current cross-browser test suite catch. Even more seemingly benign issues like z-index creating stacking contexts differently in every browser can cause the application to break for a user.

A blog post that addresses some of this would definitely be welcome.

@brian-mann

This comment has been minimized.

Copy link
Member Author

brian-mann commented Jan 3, 2018

Yeah unfortunately, we never wrote it. I haven't necessarily changed my position, which is that repeating the same identical test suite across a variety of browsers is not the way to go - but regardless we are adding cross browser support right now. For users wanting utter and complete coverage, you'll have that ability.

There is a law of diminishing returns here where each successive browser only marginally increases your confidence - and requires substantially more infrastructure and cost. It may be valuable in some cases to some users, but overall we run millions of tests, and only a small fraction of those failures occur in other browsers which would not have been caught by automated functional tests in chrome alone. I believe that's the key here - functional tests will only ever catch a specific class of bug that is mostly logic related, and oftentimes not necessarily visual. We have actually seen situations where most, if not all tests pass even when all of the stylesheets fail to load. One one hand, that's the power and flexibility of Cypress, on the other hand, functional tests give you limited confidence in visual artifacts.

The bigger solution and really the only one that really solves cross browser issues for good (while providing a good balance of adding confidence without diminishing returns) is visual regression testing (screenshot diffing). We are also adding support for that as well. With that feature, I would then recommend full testing in chrome, but only marginal happy path testing in the rest of the browsers with screenshot diffing. This balances what is oftentimes the primary root cause of cross browser testing (visual issues), while still covering the functional aspects, but needing primarily only in one browser.

It might go something like this assuming you have 300 tests.

Browser # Tests Screenshot Diffing Responsive Viewport Testing
Chrome 300 Yes Yes
Firefox 30 Yes No
Edge 30 Yes No
Safari 30 Yes No

Then when regressions or edge cases arise (meaning bugs found in production) in other various that the 30 happy path tests didn't fix - you analyze those and expand the test suite in those other browsers that may be affected. In fact - even without support for cross browsers today you can still do this by recreating the environment by which it happened. This is also what we see our users do and it works quite well. Once you understand the root cause of a problem you simply generate a contrived test to ensure it works.

@NoNameProvided

This comment has been minimized.

Copy link

NoNameProvided commented Jan 8, 2018

Hi @brian-mann!

No rush or anything, but do you have maybe a date to share? We are starting a new project soon and would love to use Cypress for testing, but support for Firefox/Safari is a requirement for us.

@mike-nelson

This comment has been minimized.

Copy link

mike-nelson commented Sep 22, 2018

Happy to see IE11 support being a priority. It’s still huge in the corporate world and harbours more differences to chrome that make it worthwhile testing on.

I am also thinkiing it would be cool if you could effectively test older versions of IE by using IE11’s ability to emulate old browsers. This would be amazing for testing old browser compatibility. Although supporting old browsers is not generally considered cool by developers it’s actually pretty snobbish towards the other half of the digital divide - older and poorer people who still have an old computer. We can’t justify manually testing on every version of IE but automating it could help make this coverage more realistic.

@willklein willklein referenced this issue Oct 18, 2018

Open

Support Firefox browser #1096

6 of 10 tasks complete
@toshovski

This comment has been minimized.

Copy link

toshovski commented Oct 19, 2018

Currently we are evaluating cypress and it is really a great framework. We had a discussion today about cross browser functionality. Do you guys plan to add new browsers by the end of the year? Is there a more detailed road map? We have mostly corporate customers. We need to make sure, that Firefox, IE and Edge are also working.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

jennifer-shehane commented Oct 19, 2018

Brian Mann (@brian-mann) gave some updates on the work we've been doing for cross browser support as well as other future features on our recent webinar - so check it out if you have your eye on this feature: https://youtu.be/FfqD1ExUGlw?t=2079

@toshovski

This comment has been minimized.

Copy link

toshovski commented Oct 19, 2018

Thank you @jennifer-shehane for sharing the link. It kind of answered my questions. Unfortunately there is no timeline about releasing the multi browser support, but it looks like you have a lot to do before the release.

@serkanyersen

This comment has been minimized.

Copy link

serkanyersen commented Oct 22, 2018

@jennifer-shehane Are you going to wait for all browsers to be ready or will you release as soon as one browser is ready and keep adding browsers on the way?

@jennifer-shehane

This comment has been minimized.

Copy link
Member

jennifer-shehane commented Oct 23, 2018

@serkanyersen There is some minimal viable work that has to be done to account for all browsers first, but no, when we feel another browser is ready, we will release that.

@adamchenwei

This comment has been minimized.

Copy link

adamchenwei commented Oct 23, 2018

We disagree, cross browser testing is not as important as you think it is.

I am also interested in having more details about this. I don't think unit testing for cross browsers is important. But for e2e I always thought that yes this was important.

I am super curious about this justification as well, good job so far though, Cypress! Looking forward for the final hockey stick on adoption once this ticket is closed! And ... before that, I am just like anyone else, eye balling on what ppl think about this topic out there...
https://www.quora.com/unanswered/Why-cross-browser-testing-is-not-important

@willischu

This comment has been minimized.

Copy link

willischu commented Oct 23, 2018

We disagree, cross browser testing is not as important as you think it is.

I am also interested in having more details about this. I don't think unit testing for cross browsers is important. But for e2e I always thought that yes this was important.

Agreed that unit testing for cross browsers is unnecessary! Having a small smoke test suite specifically for testing across different browsers would be ideal.

@irfancharania

This comment has been minimized.

Copy link

irfancharania commented Oct 25, 2018

@brian-mann: wrt the comment around "WebDriver API lowest common denominator".

The spec is essentially limited to only including "all the things that every browser can do".

How does Cypress get around this as you add support for other browsers?
If a browser doesn't have a specific feature, how do you address that?

Would you end up in a scenario like iMacros where each command lists browsers it will work with?

@jpike88

This comment has been minimized.

Copy link
Contributor

jpike88 commented Oct 30, 2018

IE11 differs hugely from other browsers and occasionally breaks in weird ways that we can't forsee. It would be definitely worth supporting it.

@irfancharania

This comment has been minimized.

Copy link

irfancharania commented Oct 30, 2018

IE11 differs hugely from other browsers and occasionally breaks in weird ways that we can't forsee. It would be definitely worth supporting it.

They demo'd a working proof of concept here

@kunwardeep

This comment has been minimized.

Copy link

kunwardeep commented Dec 4, 2018

@UncleGus

This comment has been minimized.

Copy link

UncleGus commented Dec 6, 2018

YAY! Microsoft are abandoning Edge!

But BOO, they're building YET ANOTHER BROWSER.

Is there any way we can make them just stop?

@stramel

This comment has been minimized.

Copy link

stramel commented Dec 18, 2018

Is there anyway we could leverage the Puppeteer API (https://github.com/GoogleChrome/puppeteer) to handle the cross-browser portion?

They currently support Chrome and Firefox. With Edge moving to a Chromium base, I imagine when it lands it will work, nearly out of the box.

@divyaarjnani

This comment has been minimized.

Copy link

divyaarjnani commented Dec 19, 2018

Does Cypress supports firefox and IE, got confused by reading the comments.

@highvoltag3

This comment has been minimized.

Copy link

highvoltag3 commented Dec 19, 2018

@divyaarjnani it does not currently, it is planned for the near-ish future (there's no set date or ETA). This issue has all the updates at the top the comments are made by anyone and has opinions, questions, suggestions and whatever so refer to cypress docs for supported functionality or follow this if you'd like to keep an eye on how things develop/evolve.

@nguyenj

This comment has been minimized.

Copy link

nguyenj commented Jan 16, 2019

Any plans on integrating it with saucelabs/browserstack/or a similar service? I know this has been asked before; but I can't seem to have any responses that directly answers this question.

Or is it considered that the cypress service is a direct competitor to those services (saucelabs/browserstack)–if so then I understand why the question has been avoided.

@kellyprankin

This comment has been minimized.

Copy link

kellyprankin commented Jan 23, 2019

Can anyone elaborate on this statement:

Why isn't it a priority? This is super important!

We disagree, cross browser testing is not as important as you think it is.

If I want to convince people to adopt this tool, I feel like they are going to scoff at that statement. Can you give any ammo?

@sbasinger

This comment has been minimized.

Copy link

sbasinger commented Jan 23, 2019

I'm in the same boat as @kellyprankin

We have a large IE user base, and that's where we have the most rendering issues.

@jennifer-shehane

This comment has been minimized.

Copy link
Member

jennifer-shehane commented Jan 24, 2019

@kellyprankin @sbasinger The comment on the importance of cross browser testing is an older comment. Our new stance:

Cross browser testing is now a front and center major focus here at Cypress.

We're working on support. It is a major undertaking and dependent on implementing Native Events first, which is making progress every day.

Also, see the list of tasks required for cross browser support here including PRs: #3207

@Bkucera Bkucera referenced this issue Jan 24, 2019

Open

Cross Browser issue tracker #3207

4 of 13 tasks complete
@adamchenwei

This comment has been minimized.

Copy link

adamchenwei commented Jan 31, 2019

Still having a reallllly hard time to sell Cypress to my team and company. The giantic piece of ugly hole is missing support of other browsers, I mean, if its IE, we can work with it, but not even firefox and safari is a bit tough... really looking forward to a fully browser supported version ... and when that did come, would be nice to come with a selenium test to cypress test convertor ha.

@Gpx

This comment has been minimized.

Copy link

Gpx commented Feb 1, 2019

Hi everyone, sorry for spamming. I follow this issue because I want to know when there are updates on cross-browser support. I believe many others do the same.

Instead, I receive notifications about people saying that they need Cypress to work on this or that browser or asking when will support land.

Can I please ask the maintainers to lock comments for outside members so that only relevant information gets shared? If that's not possible it's ok; I'll find another way to keep up-to-date with Cypress.

Sorry again for the noise and thanks to the maintainers for their hard work on Cypress!

@cypress-io cypress-io locked and limited conversation to collaborators Feb 1, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.