Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Try: Testing: Experiment with Puppeteer for E2E testing #5618
This pull request seeks to experiment to see how Google Puppeteer might fair in serving as our E2E testing solution, and whether it fills any of the gaps and frustrations with Cypress, our current solution.
What's wrong with Cypress?
I've wanted to like Cypress, and it does have a nice interface, but writing tests has been a constant source of headaches for me. I've attributed this to a few main causes:
Google's Puppeteer, and the Chrome headless effort it is built upon, currently maintains quite a bit of momentum behind it, and supports nearly all of the same functionality of Cypress without the sugar, incompatibilities, and inconsistencies.
With Puppeteer, utility functions can be defined as plain functions. We can expose Puppeteer's
The download size is a fair bit smaller, closer to 75MB on Mac, instead of Cypress' 115MB.
Another interesting consideration is that Puppeteer / Chrome headless may eventually replace the need for JSDOM. There are many instances of test code which is made uglier by the need to accommodate JSDOM behaviors.
Working with Puppeteer, there feels like less sugar. This is obvious in that:
One frustration I found, mostly in trying to port JSDOM usage, is that the testing environment and the Chrome window are insulated from one another, and communicate via a bridge. This can be seen in its various evaluate functions (e.g.
In all, I have good confidence that Puppeteer has a strong future ahead of it. Cypress gets a lot of things right (e.g. nice UX), but in many ways it tries too hard and suffers for it.
Thanks for the detailed write up and reference links
No objections from me, but I'd like to pint out what has been raised previously:
p.s. I liked seeing shiny new thing Jest Puppeteer linked which I've been watching, in 12 days it's garnered 500+ stars, which is not any significant metric other than a lot of people are expressing interest in Jest and Puppeteer in a very short time frame...
Ultimately a main consideration for me is the developer experience in authoring test cases. I want to write more E2E tests, but each time I have put effort toward it, it's consistently resulted in nothing but headaches. If a tool like Puppeteer enables this to be a painless and consistent experience, but means we're restricted to testing Chrome, I'm fine to make that compromise.
I really don't have much experience with Selenium, so I'll not speak to its role in this conversation, and how its experience aligns with my observations.
Agreed that Cypress is not great for handling selection and text interactions in general and that's very important for our project. I think the other parts where Cypress falls short are less important though.
The alternative proposed until now was Selenium and my experience with it until now was not great for several reasons::
So, yeah we need an alternative here and ideally, we would have the best of both worlds. So:
Do you know how Puppeteer handles the timeouts and the async behaviors? Do you have to tweak those separately and think about those in each test or are there generic ways to handle those without questioning the efficiency of these timeouts on every run?
The loss of the Cypress UI is unfortunate but it's not a blocker for me if Puppeteer tests prove to be stable enought and easy to write without the need to think about async timeouts...
These questions are hard to answer without actually trying it for some time, so I'd be fine keeping both for a release or two and trying to add more tests and see how it goes.
Thanks for the feedback @youknowriad . I think it's generally agreed that Cypress has some nice conveniences which smooth over some of the rough points which would be surfaced by a "closer to the metal" approach like that of Puppeteer.
To your specific points:
To Selenium vs. Puppeteer: Again, I'm not familiar with the former enough to be able to comment beyond statements which may be attributed as vanity ("new hotness"). Again noted in the original comment, it does appear Google is quite invested in their headless browser and the underlying Chrome DevTools Protocol used by both Puppeteer and Node.js' native inspector, which may speak to its long-term stability. In doing some brief research to gauge sentiments, the general consensus is they seek to achieve the same goal, with Selenium being the more mature and broadly-supporting option, yet a higher-level abstraction and suffering in heaviness/slowness/"quirks" for it.
I'll reiterate that my main objective is to make authoring E2E tests less painful and unpredictable, and am personally okay to make some sacrifices (browser compatibility, more difficult testing for less-common behaviors) in the pursuit of this goal.
For moving forward, I'd be okay to run in parallel, though 200MB of dependencies between the two is a pretty steep penalty to hold for very long.
Alternatively, I had thought to consider rewriting existing tests with Puppeteer to see if it surfaced any potential pain points along the way. The single test implemented here was a simple test to demonstrate one which has proven challenging to implement with Cypress.
referenced this pull request
Mar 19, 2018
Right now you can launch them using
Some issues were observed with Puppeteer's
I think we'll either need to:
gziolo left a comment
I didn't look that close on the logic of the tests but this looks like a good start. We need to look into Jest docs and improve the integration with puppeteer as explained in here: https://facebook.github.io/jest/docs/en/puppeteer.html and in the example repository linked there.
I tried a few different setups to move
I often see an error like this, which might be an existing issue but hidden somehow: