New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v5 wish list #2403

Closed
christian-bromann opened this Issue Nov 6, 2017 · 58 comments

Comments

Projects
None yet
@christian-bromann
Copy link
Member

christian-bromann commented Nov 6, 2017

Do you miss a feature in WebdriverIO? Do you have any feedback how we can make it better? Let us know and help use prioritize features for a new major release!

@KenzoM

This comment has been minimized.

Copy link

KenzoM commented Nov 6, 2017

@christian-bromann What are some of the proposed features so far?

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Nov 6, 2017

@KenzoM I haven't been through all tickets with my triage but I made a v5 milestone

@TuHuynhVan

This comment has been minimized.

Copy link
Member

TuHuynhVan commented Nov 6, 2017

@christian-bromann
I would like to see this request comes true:
#1695

@ORESoftware

This comment has been minimized.

Copy link

ORESoftware commented Nov 6, 2017

+1

@mkj28

This comment has been minimized.

Copy link

mkj28 commented Nov 6, 2017

I would love to see test retry for Jasmine....

@christian-bromann

This comment has been minimized.

@Efefefef

This comment has been minimized.

Copy link

Efefefef commented Nov 7, 2017

I would love to have
#2185
#1890

@ORESoftware

This comment has been minimized.

Copy link

ORESoftware commented Nov 7, 2017

@mkj28 what do you mean by retry exactly? what API would you like to see? Or link to an existing issue

@dguzmanSM

This comment has been minimized.

Copy link

dguzmanSM commented Nov 7, 2017

Hey Cris, I tried this awhile ago, and was still an issue for me: webdriverio/wdio-cucumber-framework#36

@KenzoM

This comment has been minimized.

Copy link

KenzoM commented Nov 8, 2017

Can we also integrate axe-core for Web Accessibility?
@klamping was awesome to show us the basic setup. It would be neat if V5 comes with it.

https://www.youtube.com/watch?v=6gwtqYCxNwQ&t=26s

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Nov 8, 2017

@KenzoM well axe-core already works with WebdriverIO it is just not simplified to one command. After all this would be a plugin and nothing that would be integrated into WebdriverIO core.

@klamping

This comment has been minimized.

Copy link
Contributor

klamping commented Nov 8, 2017

@KenzoM fyi, someone has taken a stab at putting together an Axe service here: https://github.com/cerner/terra-toolkit/blob/master/src/wdio/services/AxeService.js

@klamping

This comment has been minimized.

Copy link
Contributor

klamping commented Nov 8, 2017

One idea I have is to prompt for Sauce/BS/TB credentials if they aren't found (but asked for in the config). Right now, when you first initialize the config, it prompts you for your credentials. But once that's set it, it assumes it's already there.

The problem is, if someone new joins the project, they have to manually set up those Env variables. It's not a big task, but can be confusing to newcomers and those less familiar with env variables.

Instead, have WDIO do a detection if SL/BS/TB is being used (maybe if the service is set?), and prompt for (and set) those env variables for the user.

@sherlock1982

This comment has been minimized.

Copy link
Contributor

sherlock1982 commented Nov 8, 2017

There are lots of commands which have variants. You can find them with pattern /did not match a known command/. I don't know why but for some reason priority is given to commands which are not W3C compatible.

For example browser.execute and browser.windowHandlePosition always takes two requests on standalone geckodriver.

Maybe it makes sense to try W3C commands first? Or make a parameter maybe.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Nov 8, 2017

Maybe it makes sense to try W3C commands first?

When we release v5 most of the driver probably already speak the latest W3C spec and we can switch the order. Good point! Right now I think it only affects FF tests.

@KenzoM

This comment has been minimized.

Copy link

KenzoM commented Nov 8, 2017

What about having Jest Support?
#2052

@erwinheitzman

This comment has been minimized.

Copy link
Contributor

erwinheitzman commented Nov 10, 2017

I would like to see even better support for page objects / chaining.
Currently some methods do not work as expected when coupled with page objects / chains such as the scroll command which is made clear in the following testcases that I created:

    it('should scroll to specific element while chained', async function () {
        await this.client.windowHandleSize({width: 100, height: 100})
        await this.client.element('.box').scroll()

        const scrollPos = await this.client.execute(() => {
            return {
                x: $(document).scrollLeft(),
                y: $(document).scrollTop()
            }
        })

        scrollPos.value.x.should.be.approximately(15, 15)
        scrollPos.value.y.should.be.approximately(262, 40)
    })
    it('should scroll to specific element with offset while chained', async function () {
        await this.client.windowHandleSize({width: 100, height: 100})
        await this.client.element('.box').scroll(-10, -22)

        const scrollPos = await this.client.execute(() => {
            return {
                x: $(document).scrollLeft(),
                y: $(document).scrollTop()
            }
        })

        scrollPos.value.x.should.be.approximately(5, 15)
        scrollPos.value.y.should.be.approximately(240, 40)
    })
@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Nov 10, 2017

@erwinheitzman not sure why scrolling is important in your test, it usually isn't. Can you explain in more detail what the difference between:

await this.client.element('.box').scroll()

and

await this.client.element('.box').scroll(-10, -22)

is and what lack of support you see here?

@erwinheitzman

This comment has been minimized.

Copy link
Contributor

erwinheitzman commented Nov 10, 2017

@christian-bromann I agree that it usually isn't needed but when I was having trouble clicking a specific element a few days ago I tried scrolling before clicking.
When I tried PageObject.myGetter.scroll() I was suprised it didn't work the same as browser.scroll('my-selector').

The difference between the given examples is that one does not use any offset and the other does, I was trying to see if I could fix the behaviour for the scroll method so I added those tests locally (never got it working though as one test kept failing).
EDIT: I did notice you changed that test so maybe I had fixed it
a799a84

The support I was missing is to be able to use MyPageObject.myGetter.scroll() with and without offsets and with and without a selector.

It's not a absolute must have but I think it would be a nice thing if we could support it. Perhaps it's a nice addition for a version further down the line.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Nov 10, 2017

Perhaps it's a nice addition for a version further down the line.

Probably. I think it is important to provide consistent behavior. The issue is known and similar to #1879. I guess this will get definitely fixed within the next iterations.

However issues with clicks are common. The reason is usually because there is another element (e.g. with position:fixed located on top of the target element. I would rather remove that element instead of using scoll. Ultimately the scrolling command will be removed.

@erwinheitzman

This comment has been minimized.

Copy link
Contributor

erwinheitzman commented Nov 14, 2017

I'd love to see #2357 get implemented :)

@mkj28

This comment has been minimized.

Copy link

mkj28 commented Nov 23, 2017

what do you mean by retry exactly

@ORESoftware currently you can only rerun test block or hook, not entire test or test suite. So what I mean is having a global (say in wdio.conf.js) number defined of retries for entire test. Lower priority (but still better than nothing) would be rerunning entire suite.

Currently the only way to achieve this with Jasmine I can see is writing my own reporter or parsing test results myself and doing another webdriver.io pass for the specific test suites.

@kevinmcdonnell

This comment has been minimized.

Copy link
Contributor

kevinmcdonnell commented Nov 24, 2017

I think some performance profiling around running tests in parallel would be interesting. And if there could be any gains there

#1643

@kevinmcdonnell

This comment has been minimized.

Copy link
Contributor

kevinmcdonnell commented Nov 24, 2017

Also if I'm trying to fix a flaky test I like to rerun it x number of times to see if it passes consistently. Currently I just wrap the test in a for loop, not sure what a nice syntax would look like though, maybe similar to wdio implementation of retry flaky tests

christian-bromann added a commit that referenced this issue Nov 27, 2017

@kevinmcdonnell

This comment has been minimized.

Copy link
Contributor

kevinmcdonnell commented Nov 28, 2017

@christian-bromann have you thought about how wdio could use lambda functions to execute tests?

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Nov 29, 2017

lambda functions to execute tests?

@kevinmcdonnell not sure what you mean by lambda functions. Are you talking about AWS lambda? If so I guess it is an interesting approach to run even more tests in parallel.

@kevinmcdonnell

This comment has been minimized.

Copy link
Contributor

kevinmcdonnell commented Nov 29, 2017

Yeah there's been a load of projects popping up around running chrome (https://www.npmjs.com/package/@serverless-chrome/lambda) and tests in lambda.

This looks like a very interesting project for example https://github.com/graphcool/chromeless#api-documentation

Clay Smith here has gotten node tests with webdriver working in a lambda https://medium.com/clog/running-selenium-and-headless-chrome-on-aws-lambda-fb350458e4df

running tests on lambda is a game changer for running tests quickly. A 20 min suite could simply be as slow as the slowest test

@dguzmanSM

This comment has been minimized.

Copy link

dguzmanSM commented Nov 29, 2017

Hey guys, what is an ETA for v5 release?

@kevinmcdonnell

This comment has been minimized.

Copy link
Contributor

kevinmcdonnell commented Dec 7, 2017

just an update on my lambda idea. It looks like this will be more easily achieved using AWS Fargate (https://aws.amazon.com/fargate/ is a new product still in beta).

The idea is our jenkins workflow will create a container which has all the test dependencies and a checkout of the code. If our tests suite has 100 spec files we use fargate to run 100 containers which will each run 1 spec file. We would need some way to orchestrate communication with fargate and the division of spec files, I guess this could be achieved by a wdio plugin (need to look up how to create one)

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Dec 7, 2017

@kevinmcdonnell this is a great idea and will probably get shipped within the new test runner of v5. I guess the simplest integration is to check for an AWS or GCE key and run them automatically as lambda function if they exist in the config.

@kevinmcdonnell

This comment has been minimized.

Copy link
Contributor

kevinmcdonnell commented Dec 7, 2017

Note this is a new product AWS fargate not AWS lambda. Lambda has limitation on the size of what you can pass to it (which could possibly be made to work but may be harder and less versatile) whereas fargate just executes a container that you've built so it can easily have dependencies like chrome/firefox, choice of OS, a checkout of your code etc

I imagine we would use the node AWS SDK to talk to AWS fargate. This might be through keys or IAM role. We would want an option for either, as if we ran this on our Jenkins instance we would want to use IAM for security reasons.

I think there would be a build step related to building the container. Given we have many projects we would have a default container defined and then the build step would copy the code for a given project and push the generated container to a registry.

So wdio config would also need config for the container registry and the container. Then fargate would execute the command in the container which has the dependencies and a checkout of the code.

obviously wdio would want as generic an implementation as possible while supporting common use cases. hopefully in the next couple weeks i'll get to try fargate and spec what this would look like

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Dec 7, 2017

Lambda has limitation on the size of what you can pass to it (which could possibly be made to work but may be harder and less versatile)

I am not sure if we would hit that limitation. I would rather not include any browser driver to it as it increases complexity. I thought that this option would only be used combined with an external grid (e.g. Sauce Labs or home grown grid).

The use case of running the whole test including browser driver within a container should be exported into a service that requires a more complex configuration.

@dguzmanSM

This comment has been minimized.

Copy link

dguzmanSM commented Dec 7, 2017

Another issue is that Lambda has a 5 minute limit, our team was also thinking Fargate would be a better solution untill AWS increase Lambda execution time.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Dec 7, 2017

Lambda has a 5 minute limit

Oh right, well this pretty much rejects Lambda as an option for running e2e tests

@emilbader

This comment has been minimized.

Copy link

emilbader commented Dec 12, 2017

I would like to see #2406 in v5.

@erwinheitzman

This comment has been minimized.

Copy link
Contributor

erwinheitzman commented Dec 14, 2017

I think it would be great to add a list of requirements that are needed to be able run all the unit tests.
For example currently I am missing information about you needing to globally install bower, you can't run the mobile tests out of the box either.

It would also be nice if a simple npm install would install all the requirements for running the tests if possible.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Dec 14, 2017

@erwinheitzman as you might have seen we rebuild the complete project infrastructure which will make stuff easier

@mhemesath

This comment has been minimized.

Copy link
Contributor

mhemesath commented Dec 14, 2017

I have a feature request idea. Run script injection through babel matching the env the script will be injected to. This will allow me to do this with a hypothetical example:

 var result = browser.execute((...numbers) =>  {
   const result = numbers.reduce((total, value) => total + value);
   return result;
 }, 1, 2, 3, 4);

instead of

var result = browser.execute(function(a,b,c,d) {
  var result =  a + b + c + d;
  return result;
}, 1, 2, 3, 4);

Currently we test older versions of IE and use old ES syntax just because its more predictable that it will work.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Dec 14, 2017

@mhemesath sounds like a good idea .. in general we should automatically compile test files and injected script with babel if it is setup .. similar to what Jest is doing

@mhemesath

This comment has been minimized.

Copy link
Contributor

mhemesath commented Dec 16, 2017

Idea: Throw a warning if webdriver.io is configured with a maxInstances higher than the hubs maxSession or slotCounts.total. It seems exceeding either of these can lead to errors or timeouts.

This information should be possible to get through the hub API:

curl http://localhost:4444/grid/api/hub
{  
   "success":true,
   "capabilityMatcher":"org.openqa.grid.internal.utils.DefaultCapabilityMatcher",
   "newSessionWaitTimeout":300000,
   "throwOnCapabilityNotPresent":false,
   "registry":"org.openqa.grid.internal.DefaultGridRegistry",
   "cleanUpCycle":5000,
   "custom":{  

   },
   "host":"10.0.0.3",
   "maxSession":10,
   "servlets":[  

   ],
   "withoutServlets":[  

   ],
   "browserTimeout":100000,
   "debug":false,
   "jettyMaxThreads":-1,
   "port":4444,
   "role":"hub",
   "timeout":90000,
   "enablePassThrough":true,
   "newSessionRequestCount":0,
   "slotCounts":{  
      "free":0,
      "total":5
   }
}

Being services seem to be the standard way of integrating a hub configuration into wdio; part of me thinks that the maxInstances configuration should actually come from the service providing the selenium/browser integration. Standalone Selenium would probably only do 1-5 sessions, whereas a Hub service would execute the API above to configure this dynamically. I'm also sure saucelabs has a way to give back the number of session that can be executed.

@tavares111

This comment has been minimized.

Copy link

tavares111 commented Dec 22, 2017

I would love to have
#2208

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Dec 23, 2017

@tavares111 please no bugs, only features that don't exist yet

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Jan 4, 2018

@dguzmanSM @kevinmcdonnell getting back to the Lambda vs Fargate idea. I think a 5min limit is fine. By running more stuff in parallel the execution time of the actual spec file decreases and should not exceed 5min.
The fact that it requires to build a container for Fargate is problematic. It should be as easiest as possible to run something on a remote machine. Having Docker as a dependencies is not really "easy".

@patthiel

This comment has been minimized.

Copy link
Contributor

patthiel commented Jan 9, 2018

@christian-bromann @kevinmcdonnell since it looks like we're going the lambda function route, i figured it'd be good to point out that Google has a similar (read: competing) product to Lambda called Google Cloud Functions.

It'd be great if we could have service plugins to support both. Perhaps something to investigate after the initial lambda service is complete.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Jan 9, 2018

@patthiel yes, it will! I will build it on top of serverless which will support a variety of cloud provider, including Google Cloud Functions

@elkemper

This comment has been minimized.

Copy link

elkemper commented Feb 15, 2018

guys what about make scroll() returning web-element?
it'll be useful in "pipelines" like this

$('div.some.classes').scroll().click();
@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Feb 20, 2018

Just fyi: I just came across to implement this and since I've seen a lot of people doing this I just figured to embed it natively. So in a lot of cases you have this behavior:

const elem = $("selector")
elem.waitForVisible()
elem.click()

In v5 we will do the waiting for you. It checks if the initial query did found an element and if not, it would wait for a specific time (configurable in options). So all you need to do is:

const elem = $("selector")
elem.click()

However in case the element can't be found even after waiting for it we will fail the command where you wanted to use it (e.g. here when you call click).

@klamping

This comment has been minimized.

Copy link
Contributor

klamping commented Feb 20, 2018

@christian-bromann that's a really great idea, as I see this pattern all over the place. 👍

@MarianoSalerno

This comment has been minimized.

Copy link

MarianoSalerno commented Feb 20, 2018

Capybara does something like that with the find method, it waits for a configurable time period for an element to appear and, if it isn't found, throws an exception. It also fails if it finds more than one element that matches the selector, here is the doc

@kevinmcdonnell

This comment has been minimized.

Copy link
Contributor

kevinmcdonnell commented Mar 19, 2018

I think it would be good to have some more logging by default. I'm thinking for each run include:

browser and version (I just lost some time to this as my ci env was running different version to my local dev env)
path to logs (selenium/chromedriver etc) might aid discoverability of these logs for newcomers

there's probably more that is sufficiently succinct that they could be displayed to save time instead of having to rerun tests with verbose

@aamorozov

This comment has been minimized.

Copy link

aamorozov commented Mar 23, 2018

I would love to see the support for Jest as a test runner out of the box. If this will support separation for unit/webdriver tests that'd be awesome.

Also, better support for docker/docker-compose with selenium grid would be nice since it seems like more and more projects are getting containerized.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Mar 23, 2018

@aamorozov what does the jest runner provide for e2e testing? It is an excellent runner for unit tests but not really useful for e2e test. Where do you define the capabilities and other options? I recommend to use wdio with jasmine that gives you almost the same assertions as with jest.

@aamorozov

This comment has been minimized.

Copy link

aamorozov commented Mar 25, 2018

@christian-bromann
The only reason I've had in mind for that is that our dev team is using jest for unit tests, and it would be very convenient to find a way(if it can be implemented) to use jest for e2e tests as well. I agree though that there are certain complications, but i saw you've already made some work here https://github.com/christian-bromann/jest-environment-wdio, and i wonder if it is something you would like to include to v5.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Mar 26, 2018

@aamorozov I was just investigating how an integration could possibly look like but as I mentioned above there is no value in adding jest as test runner to run e2e test. It will be more likely that I will implement a jest environment to run jest tests in the browser like karma does. If you want to have jest like assertions, use the wdio with the jasmine framework adapter.

@christian-bromann

This comment has been minimized.

Copy link
Member

christian-bromann commented Sep 18, 2018

Closing this. Thank you very much all for your feedback and input. If you have any other feature request, just create a normal issue on GitHub. Thank you!

@pradeepvish1213

This comment has been minimized.

Copy link

pradeepvish1213 commented Oct 11, 2018

How to open multiremote with multiple urls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment