Alternatives to Protractor - request for comments

[jan-molak]

While the Protractor project is still maintained by the Angular team (see angular/protractor#5390) and the integration with it will continue to be supported by the Serenity/JS team as long as our community finds it useful, the project is no longer as active as it used to be.

At present, there are other modern browser integration frameworks with an active community. Ones that provide functionality similar to Protractor and that Serenity/JS could support in addition to Protractor to provide an alternative for browser-based testing.

So far, the Serenity/JS community proposed several alternatives. I raise this ticket to gather your feedback and suggestions and help us make a call regarding if and what alternative to Protractor Serenity/JS should offer in the coming months and years to best address your needs. See #6 #100 #218 #247 #326 #487

Based on my current understanding, there are several candidates that are worth considering:

• Cypress
• Playwright
• WebDriver.io
• Puppeteer
• Karma

As you might have expected, supporting another browser integration framework is a major undertaking and will require a significant investment of time and effort to understand all the features, quirks and bugs of the selected tool, find stable workarounds, design appropriate abstractions, document the APIs, and write tutorials. For this reason, rather than jumping straight into implementation, I'd like us to first focus on understanding the opportunities each framework provides to the Serenity/JS community.

That's why I'm asking for your help :-)

If you'd like to see Serenity/JS support one (or more?) of the above frameworks, please comment on this ticket and mention:

1. your framework of choice
2. the reason why you think it would make a good addition to Serenity/JS modules
3. your use case, i.e. web testing, mobile testing, cross-browser testing, performance testing, etc.
4. if you or your company would like to support the effort of integrating Serenity/JS with your framework of choice either by working on PRs or sponsoring the project

I'm looking forward to hearing your thoughts!
Jan

[AbhineetSharmax]

@jan-molak , there is no doubt that Serenity-JS is helping a lot of people understanding the framework in a quick and easy way.
Would also love to see serenityJS in action with webdriver.io with programming language Python.(Python-Selenium BDD). Thank you for every effort you do to make things easier.

Best Regards,
Abhineet Sharma

[abhinaba-ghosh]

@jan-molak

Last couple of days, I was thinking about the same and have taken some time to analyse, research and study before delivering a thought on this.

Reasons of thinking about an alternative:

1. protractor repository is facing lower commit frequency for a long time now. This does not necessarily mean that protractor is getting abandoned/archived. The new hope is here Release 5.4.3 angular/protractor#5388

2. Protractor depends on Selenium.

3. Protractor does not support well Non-Angular pages. One can always turn off the synchronisation flag at the cost of loosing the http tracking of your page. That leaves one to become dependent on the ExpectedConditions heavily which enhances the chances of hard coded timeouts.

Is migration is really necessary?

I would say yes, but not right at this moment. Protractor is still best for multi-browser testing and parallel testing. With the latest async/await support, its very easy to code in a synchronous way who comes from selenium background. Protractor is already in use since long in numerous sophisticated companies. And last but not the least, protractor has a huge support in open-source world, which most of the tools does not. They are still in infancy.

But, in longer run , serenity should support all major tools in parallel. I see serenity-js more from reporting/logging perspective. Handling low level browser interactions can be done via any tool.

How should the migration happen? (at-least my wish)

Serenity-js 2.0 is already having a modular architecture. So, why not leverage that and create modules in parallel, like:

@serenity-js/cypress
@serenity-js/karma ....

list can go on...

But there should be a tool which can easily convert existing module to different module. I mean, today i can use karma, may be tomorrow cypress. My high level screenplay/traditional calls will be same, i should easily migrate to different browser handler just by changing some configuration files. Then, we no need to think about different tools. I can run same set of BDD tests just using different handler.

If we provide that support of modularisation, then the selection of automation tool wont be a deciding factor. Serenity-js will be acting as the execution manager and provide the interface to directly develop meaningful tests.

What are the good alternatives?

I would definitively like to have Cypress at the top. On the other hand, the promises made by Playwright is phenomenal. If Playwright releases a stable version soon, we can bump up the priority for Playwright. We can consider Testcafe also in the list.

Webdriver.io is a nice tool but it mainly leverage web-drivers to execute low lever browser interactions where Playwright/cypress connect to browsers directly via proxy. So, I have a slight preference for the successors over the WDIO.

Some thoughts:

With the increasing popularity of BDD and JS/TS, i think serenity-js is having a better future than the java version of it. (OMG! did i just say that). Serenity-js is lightweight and fast. Basically it manages your test execution in run time.

So, its time to take a decision and bump up the support for serenity-js.

[christian-bromann]

I would like to chime into this:

I dont recommend webdriver.io because again you will take the path of webdrivers to connect you to the browser. With Playwright/cypress, you can connect to browsers directly via proxy.

This is not true. With v6 (which we will release soon) you don't need a driver anymore if you run tests locally. WebdriverIO by default will use Puppeteer to run its automation. This will also allow user to make use of the extended DevTools capabilities to e.g. stub network requests or emulate other parts of the browser. Read on more on this here: https://github.com/webdriverio/webdriverio/blob/master/website/blog/2020-02-22-webdriverio-v6-released.md#devtools-automation-protocol-is-now-default. That said, the wdio testrunner comes with plugins that manages drivers for you as well.

Some other advantages choosing WebdriverIO:

• active and growing support community
• part of an open source foundation: OpenJS Foundation and therefor not owned by a company
• a lot of already existing integrations into 3rd party vendors like Sauce Labs or Applitools
• tons of free available learning courses

I would be happy to support the migration efforts if you end up choosing WebdriverIO. Cheers!

[iranicus]

For us we are currently migrating a lot of our existing angular 4 projects that are heavily Jasmine karma unit test based to Angular 8 with Jest for unit tests and Cypress that works with Jest for assertions too. Therefore for us in the near future Cypress is the one that I would put forward especially due to its popularity (although sucks that they are Chrome/Chromium only). We are going to be using it for our integration tests with mocking the backend (very easily done with Cypress + Jest) to test the UI for both accessibility, layout and behavior. The idea later down the road then for us is to have Cypress running both while we are developing our screens and in our Jenkins pipeline on each commit.
Our client however is very IE11 focused for some reason so we also have a separate Java selenium-cucumber project that we run separately to test against IE11 which at the moment runs all our integration tests (the addition of Cypress in the same repo as the Angular 8 project will take care of a lot of UI tests that the Java project handles)

[abhinaba-ghosh]

I would like to chime into this:

I dont recommend webdriver.io because again you will take the path of webdrivers to connect you to the browser. With Playwright/cypress, you can connect to browsers directly via proxy.

This is not true. With v6 (which we will release soon) you don't need a driver anymore if you run tests locally. WebdriverIO by default will use Puppeteer to run its automation. This will also allow user to make use of the extended DevTools capabilities to e.g. stub network requests or emulate other parts of the browser. Read on more on this here: https://github.com/webdriverio/webdriverio/blob/master/website/blog/2020-02-22-webdriverio-v6-released.md#devtools-automation-protocol-is-now-default. That said, the wdio testrunner comes with plugins that manages drivers for you as well.

Some other advantages choosing WebdriverIO:

• active and growing support community
• part of an open source foundation: OpenJS Foundation and therefor not owned by a company
• a lot of already existing integrations into 3rd party vendors like Sauce Labs or Applitools
• tons of free available learning courses

I would be happy to support the migration efforts if you end up choosing WebdriverIO. Cheers!

Yes. I just like to withdraw my opinion on WebDriverIO. I have gone through the version 6 release. Looks really promising.

[abhinaba-ghosh]

An update for the concerned fellows angular/protractor#5442 (comment)

[sachin-s-joshi]

@jan-molak I would agree with either Cypress or WebdriverIO v6 but more preferably the later one.

Reasons:
I echo the points mentioned by @christian-bromann. Also, from Cypress point of view there is nothing wrong either but when we are thinking to develop fully baked framework then it ideally should work :

• Across browsers (not just chromium based)
• Flexibility to work on different types of test framework like jasmine,mocha/cucumber.
• General purpose Automation tool which means support for mobile, performance, scraping,etc

Currently Cypress has some trade offs which is worth to look before we choose.

My Usecases:
I have used both and loved both but I had some straightforward scenarios for which I did not really care for cypress tradeoffs.

Lastly, I will be happy to contribute and look forward to this feature in future ;) Cheers!

[jan-molak]

Hi folks, looks like the Angular team's been busy and they're looking for our feedback regarding the E2E experience.

[viper3400]

Hi @jan-molak

So you could add https://nightwatchjs.org/ to the list of candidates of browser automation tools.

As you wrote in angular/protractor#5442 (comment): "Serenity/JS, [is] an acceptance testing framework that uses Protractor to interact with web interface".

So for me the requirements of an acceptance testing framework may match in many places with the ones for a browser automation framework. But they may not be completely the same anyway. Therefore I agree with @abhinaba-ghosh in #493 (comment).

In my (short) experiences with Serenity/JS it's very Protractor bound (actually, this is why I get in contact with Serenity/JS at all).

In my opinion Serenity/JS should not be bound to this or that browser automation tool but should provide a kind of higher abstraction layer.

I think most automation tools implement chapters 10 to 17 of https://www.w3.org/TR/webdriver/, maybe not in the way, this standard describes, but in a way at all, because this is the way we interact with browsers.

For a tester it's relevant, how to find the elements in a web page, how to interact with them on a high level (click, scroll, input), how the ensure the test result and how to report this results.

How the automation tool interacts with the browser "under the hood" should not be in scope of writing acceptance test cases. I have to rely on this.

For sure, a nowadays an automation tool have to automate multiple browsers, on multiple OS and of course with support of CI ...

I'd like to advocate for an acceptance testing framework, which supports a high level "standard" like set of features of how to interact with the browser and which creates descriptive tests and reports. Under the hood it will work with Protractor and maybe in future with other automation tools. I think, Serenity/JS has gone half of the way, already.

If an automation tool has a special killer feature, it should be possible, to use this in a way as well. If I'd like to use an automation tool, which supports only Chrome in example, I should be able to configure my test set to use it with another automation tool to interact with Firefox without having to reinvent the whole test code.

PS: As the angular team wrote they will support Protractor in future, but this will come with breaking changes. So one will maybe have to rework it's automation and tests either way in order to use newer Protractor versions. We will see ..

[cipivanov]

I hate being that guy, but I have to show Playwright some love.

It has become our tooling of choice in our org when migrating our test library away from Cypress. We mostly opted for it because we wanted to speed up our test runs, through parallelization, a better development experience for our SE/SDETs when writing/debugging tests and wider cross-browser testing capabilities.

Pretty sure none of the above can be really be considered a unique Playwright selling point.

Next we are looking for a better way to structure our existing code and potentially get better reporting and mocking capabilities, hence serenity-js.

I've basically copy-pasted the Puppeteer prototype from @jan-molak, with trivial adjustments to integrate it with Playwright, in the hopes of putting it on the map and get some more traction on it (jan-molak/authenticator-browser-extension#372).

@abhinaba-ghosh you seemed particularly excited about Playwright some time ago. What are your thoughts on it now? Is it a good integration candidate?

[abhinaba-ghosh]

@cipivanov Definitely yes! I would love to see Playwright on the list. When I integrated the puppeteer prototype into the authenticator, Playwright was always in the mind.

[jan-molak]

@cipivanov thanks for adding to the conversation!

I think Playwright is really promising; I wonder if Serenity/JS should support it directly, though? Perhaps it would be better for @christian-bromann to support it in WebdriverIO, which Serenity/JS could then integrate with? This way WebdriverIO would enable integration with Selenium, Puppeteer and Playwright and provide a solid foundation for S/JS to build on.

[Playwright] has become our tooling of choice in our org when migrating our test library away from Cypress

@cipivanov could you please share more details on the factors that convinced you to migrate away from Cypress to Playwright? Perhaps we could avoid the issues you've already learnt from?

[cipivanov]

@jan-molak,

I wonder if Serenity/JS should support it directly, though?

I do not have a good understanding what WebDriverIO current or future capabilities. It doesn't look like it currently supports Playwright, although, looks like the idea was entertained but ultimately put on hold for very valid reasons.

I completely understand where you are coming from an initial effort and also maintenance perspective when suggestion WebDriverIO though.

[gerhardboer]

@jan-molak
I just got notified on this document on the Angular github where they are outlining the plan to deprecate protractor:
angular/protractor#5502

Since this will be the case in the end of 2022, i hope we can find a solution to integrate SerenityJS with before that time.

[jan-molak]

@gerhardboer - absolutely! I'm planning to support Protractor as long as people need it, so at least until the end of 2022.

My goal with Serenity/JS is for it to act as an abstraction layer and integrate with all the popular test frameworks. Ideally, I'd like for the Screenplay code to be re-usable across the different test frameworks too, so the code you wrote for Protractor could be used with say WebdriverIO or Playwright, etc.

I've launched a quick poll to see what test framework people would like Serenity/JS to integrate with next, so please feel free to cast your vote - https://twitter.com/SerenityJS/status/1379368915017592835

Also, if you'd like to help me secure more time to work on Serenity/JS, please feel free to sponsor the project, every little helps! https://github.com/sponsors/serenity-js

[jan-molak]

Closing in favour of #805; watch this space 😄

[jfstephe]

@jan-molak - Sorry to come late to the party, but I want to make the case for serenity and karma, and really that it's more important than protractor and UI element based testing tools.

Modern application frameworks allow testing components in isolation from the application. An application is basically just a large complex component, made of lots of other components. Given the requirements from the business, it falls to the development team to decide the level at which it is appropriate to implement the tests that exercise those requirements (End-to-end, integration, unit). 'End-to-end' UI element (View) based tests are slow, and even with the best design patterns take more effort than 'unit' tests to maintain. Often the same goal can be achieved from testing using ViewModels, rather than Views.

Karma, used with a good application framework allows for this as, if you've designed your application appropriately, you can write your BDD-based tests at any of these levels. With protractor etc you are forced to use the View level. With Karma you can use the same unit testing tools to test at a higher level, to a greater effect.

Is there any change that Karma and Serenity integration will happen soon/ever?

[jan-molak]

@jfstephe - The goal of Serenity/JS 3.0 is to enable a consistent DSL no matter the underlying test integration tool (Protractor, Puppeteer, Playwright, WebdriverIO, etc.).

As long as Karma, or any other tool for that matter, provide an API offering capabilities similar to what you can accomplish with Selenium or Playwright, it should be possible for Serenity/JS to support it too. What I mean by a similar capabilities is that the tool should allow for Serenity/JS to perform interactions from the new @serenity-js/web package.

Note that those APIs don't have to be exactly compatible. For example, Playwright or Puppeteer API is quite different to one of Selenium.

So, to summarise - sure, I think it would be great for Serenity/JS to support component testing better, and believe that version 3.0 should lay the foundation here.

Is adding support for Karma something you'd be interested in looking into yourself perhaps? Happy to walk you through it if you're interested.

[jfstephe]

@jan-molak - good to know thanks. I'll consider reaching out when our team has more bandwidth. It'd be good if you could give some idea of how much effort you may think this would be (days/weeks/months), assuming the karma api fits in with what is expected. Also where do you have an eta on v3 landing?

[jan-molak]

@jfstephe - sounds good! V3 APIs are all complete and stable, so the last outstanding thing to do is for the docs and tutorials to get updated to reflect the new changes, which I've already started doing.

In terms of support for Karma - have a look at the new modules: @serenity-js/playwright, @serenity-js/webdriverio, and @serenity-js/protractor.

The new architecture in V3 makes it actually quite straightforward to write extensions like that and there's a standard set of tests such extensions need to pass, so it's reasonably easy to know how far you got with the integration and how much is left.

From what I found, a new integration takes a couple of weeks to implement, but most of the time I spent is investigating the new tool's quirks, any undocumented behaviour it might have, and working around or submitting PRs to fix its bugs.

If integration with Karma is something you'd like to pick up, I'd be happy to guide you through it?

Also, Playwright has recently introduced experimental support for component testing, which I think is worth exploring, and shouldn't be too hard for Serenity/JS to support: