Support browser using proxy for UI automation

[ptrthomas]

totally missed this ! propose adding a proxy key to the configure driver

good issue for potential hacktoberfest contributors !

you will need to support this for both the native Chrome and the WebDriver paths. really don't know about mobile, but assume it is not needed

[ptrthomas]

assigned to @t-gok as per #902 (comment)

[ptrthomas]

attention all watchers - I tried to fix this, but it just doesn't seem to work - so would appreciate some help. even the native chrome doesn't work but I suspect is because my chrome install is corp-managed

this commit has the details: 0aeefb6

also we have switched from desiredCapabilities to capabilities for WebDriver W3C

references for anyone willing to help / contribute:

https://w3c.github.io/webdriver/#proxy
https://www.linuxbabe.com/desktop-linux/configure-proxy-chromium-google-chrome-command-line

[paaco]

Hi @ptrthomas, I can confirm that for Firefox the proxy setting only works with my 'headless' fix, because of "alwaysMatch" in the W3C capabilities. If you leave that block out, the proxy object is not picked up. Here is a short log when it works:

1 > User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_241)
{"capabilities":{"alwaysMatch":{"browserName":"firefox","proxy":{"proxyType":"manual","sslProxy":"127.0.0.1"},"moz:firefoxOptions":{"args":["-headless"]}}}}

09:15:26.934 [geckodriver_1579594524005] DEBUG com.intuit.karate.driver.geckodriver_1579594524005 - 1579594526931 mozrunner::runner INFO Running command: "C:\Program Files\Mozilla Firefox\firefox.exe" "-marionette" "-headless" "-foreground" "-no-remote" "-profile" "C:\Temp\rust_mozprofiledrvpB0"
09:15:27.042 [geckodriver_1579594524005] DEBUG com.intuit.karate.driver.geckodriver_1579594524005 - *** You are running in headless mode.
...
09:15:31.265 [geckodriver_1579594524005] DEBUG com.intuit.karate.driver.geckodriver_1579594524005 - 1579594531265 Marionette INFO Proxy settings initialised: {"proxyType":"manual","sslProxy":"127.0.0.1:80"}

I guess a possible fix is to always add "alwaysMatch" when setting up the "capabilities" object.

[ptrthomas]

@paaco this is great, thanks. can I request you for another PR ? you have mentioned alwaysMatch a couple times and looks like that is the way to go. I think with this out of the way I'd look for a 0.9.5 final release. by any chance have you tested the chrome docker container, wanted to see if that was ok after recent fixes to develop

[paaco]

@ptrthomas I only used the chrome docker container for very limited testing, but it seems to work fine with a very recent pull of develop when running one test at a time just now (latest image).

Using a Scenario Outline will only run the first test and fail the others immediately with "driver config / start failed" while Chrome is restarting in the container. Not sure if that is intended or not.
I just copy/paste the code from the documentation here: https://github.com/intuit/karate/tree/develop/karate-core#karate-chrome

The proxy setting is not honored, as you already said in #924 (comment). I will test adding the alwaysMatch to all driver code paths and issue a PR if that fixes it.

[ptrthomas]

@paaco thanks, yes do try for a PR. regarding Docker if you have the time, take a look at instructions to build the Docker image locally: https://github.com/intuit/karate/wiki/Developer-Guide#docker-images

I'll be testing it in a bit - but always helps to have a second opinion. I'll also look into Scenario Outline problem you mentioned

[paaco]

Proxy setting ONLY tested and verified with Firefox (geckodriver) and Chrome 78 (chromedriver). 'chrome' devtools protocol with docker-image doesn't respect the proxy settings. Probably that is set differently. I can check that later.
But I have no ability to verify MsEdge or Safari.

[ptrthomas]

Proxy setting ONLY tested and verified with Firefox (geckodriver) and Chrome 78 (chromedriver).

more than enough ! thank you :)

[ptrthomas]

@paaco would you happen to know the answer to this stack overflow question: https://stackoverflow.com/q/59875791/143475

[paaco]

@ptrthomas acceptInsecureCerts is a W3C Webdriver capability as specified here: https://www.w3.org/TR/webdriver/#capabilities
The capabilities object is created in DriverOptions with only browserName and proxy so at the moment Karate doesn't support it. But it's pretty easy to add support for acceptInsecureCerts there just like the proxy or headless setting. I can make a PR in a few days.

[ptrthomas]

@paaco that would be great, thanks !

[ptrthomas]

@paaco thanks a lot for the PR-s and with this I think a 0.9.5. final release can be cut

I have one idea and wanted to know what you think, especially for people wanting to experiment with RemoteWebDriver and SaaS providers (e.g. AWS just entered the game) - I was thinking if we add a webdriverCapabilities configure driver key which takes a JSON that will be passed as-is.

AWS capabilities: https://docs.aws.amazon.com/devicefarm/latest/testgrid/techref-support.html

does that make sense or would you suggest anything else.

[paaco]

@ptrthomas Yeah that could make sense and is easy enough to change.

acceptInsecureCerts is then automatically supported, just as all other currently uncoded features like AWS or exotic Chrome settings.

Drop the explicit headless support too and have users write moz:firefoxOptions/args/-headless inside webdriverCapabities themselves if they want. Would be more cumbersome but it takes away the invisible magic that occurs and sounds more like the Karate-way of doing.

The proxy setting is probably also better off inside webdriverCapabilities.

[ptrthomas]

@paaco thanks, I'll go ahead and make this change ! this i just an optional thing and does not change what you already implemented (default kind of behavior)

[paaco]

@ptrthomas I prefer you remove it, it's not necessary anymore when you implement webdriverCapabilities. That way you have less code to maintain.

[ptrthomas]

@paaco oh ok, I will make the change and hope you are ok with one more round of testing

[paaco]

@ptrthomas You should have kept the "alwaysMatch" line here otherwise it won't work. db6b06b#r36998158

[paaco]

@ptrthomas Hmm. On second thought, just put 'browserName' inside 'alwaysMatch' by default.

You want to be able to add a "firstMatch" especially with RemoteWebDriver instances. That works how it is implemented right now.

[ptrthomas]

@paaco ok take a look now

[paaco]

@ptrthomas It should do, but I think I prefer something like this:

1. start with webDriverCapabilities
2. if there's an alwaysMatch, putIfAbsent(browserName) in there
3. otherwise add alwaysMatch and putIfAbsent(browserName) in there

This way the browserName always gets added to alwaysMatch and not to the capabilities root. Does that make sense?

[ptrthomas]

@paaco ok how does it look now

[paaco]

@ptrthomas that looks pretty good, but I have to test it further tomorrow.

[paaco]

@ptrthomas Ok, I tested with both Firefox and Chrome and works as described above. I like it this way, it's very flexible.

[ptrthomas]

@paaco thanks for all your help on this !

[ptrthomas]

@paaco and all watching, I just made tweaks so that Karate can call a "remote" webdriver instance, here are the docs: https://github.com/intuit/karate/tree/develop/karate-core#webdriverurl

this should be of interest for those who want to use AWS and even for mobile testing. I tested this on Zalenium running locally

[ptrthomas]

@paaco after some investigation of how selenium grid and aws works I decided it is better to allow the user to specify the whole payload of the JSON of which capabilities is one key.

please refer docs. I also had to tweak the logic again for alwaysMatch but I think it is ok for the default case, so take a look

docs: https://github.com/intuit/karate/tree/develop/karate-core#webdriversession

[paaco]

@ptrthomas Nice! I love it how you make things more generic. That is what Karate makes the real Swiss army knife of API testing.

[forlixdev]

Hi. I tried the solutions proposed but it does not work for me:

  Scenario: try to use a proxy
    * def webUrlBase = "http://httpstat.us/"
    * def session = { capabilities: { browserName: 'firefox', proxy: { proxyType: 'manual', httpProxy: '127.0.0.1:8080' } } }
    * configure driver = { type: 'geckodriver', webDriverSession: '#(session)' }

    Given driver webUrlBase

It seems that the browser is not using the capabilities that were "suggested", from the log file:

09:28:02.872 [ForkJoinPool-1-worker-1] DEBUG com.intuit.karate - response time in milliseconds: 3049,83
1 < 200
1 < cache-control: no-cache
1 < content-length: 712
1 < content-type: application/json; charset=utf-8
1 < date: Sat, 23 May 2020 07:27:59 GMT
{"value":{"sessionId":"a04f2436-00ea-4430-8b71-a0e31bdb8327","capabilities":{"acceptInsecureCerts":false,"browserName":"firefox","browserVersion":"76.0.1","moz:accessibilityChecks":false,"moz:buildID":"20200508083352","moz:geckodriverVersion":"0.26.0","moz:headless":false,"moz:processID":27562,"moz:profile":"/tmp/rust_mozprofileIeK83P","moz:shutdownTimeout":60000,"moz:useNonSpecCompliantPointerOrigin":false,"moz:webdriverClick":true,"pageLoadStrategy":"normal","platformName":"linux","platformVersion":"5.6.13-300.fc32.x86_64","rotatable":false,"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify"}}}

I tried with geckodrive 0.26.0 + Firefox 76 and chromedriver 83.0.4103.39 +Chrome 83

[paaco]

@forlixdev Firefox is very picky about alwaysMatch, unlike Chrome, see #924 (comment)

Could you try:
* def session = { capabilities: { alwaysMatch: { browserName: 'firefox', proxy: { proxyType: 'manual', httpProxy: '127.0.0.1:8080' } } } }

[forlixdev]

Thank @paaco. I tried also before with Chrome but nothing, I'll retry with your suggestions with Firefox.

[forlixdev]

Ok, tested. It works with alwaysMatch. Thanks

[DineshTejaj]

@paaco : When am trying to open firefox with the below
Could you try:

• def session = { capabilities: { alwaysMatch: { browserName: 'firefox', proxy: { proxyType: 'manual', httpProxy: '127.0.0.1:8080' } } } }

Chrome browser it getting opened up, Please help me in launcing Firefox and edge drivers am good with chrome,

Please help me in opening Firefox and Edge browsers? please share the sample code here

cc: @ptrthomas

[DineshTejaj]

Getting the below error
14:35:02.571 [geckodriver_1654765502546] ERROR com.intuit.karate.shell.Command - command error: [geckodriver, --port=4444] - Cannot run program "geckodriver" (in directory "target\geckodriver_1654765502546"): CreateProcess error=2, The system cannot find the file specified
14:35:08.080 [main] ERROR com.intuit.karate - driver config / start failed: start failed, options: {type=geckodriver, webDriverSession={capabilities={alwaysMatch={moz:firefoxOptions={args=["-headless"]}}}}, target=null}
com.intuit.karate.KarateException: start failed

[ptrthomas]

@DineshTejaj I haven't tried with firefox and edge at all, do investigate if you can

[DineshTejaj]

@ptrthomas : When am using this command

• configure driver = { type: 'geckodriver', showDriverLog: true }
  AM getting the below error, [geckodriver_1654765502546] ERROR com.intuit.karate.shell.Command - command error: [geckodriver, --port=4444] - Cannot run program "geckodriver" (in directory "target\geckodriver_1654765502546"): CreateProcess error=2, The system cannot find the file specified
  14:35:08.080 [main] ERROR com.intuit.karate - driver config / start failed: start failed, options: {type=geckodriver, webDriverSession={capabilities={alwaysMatch={moz:firefoxOptions={args=["-headless"]}}}}, target=null}
  com.intuit.karate.KarateException: start failed

Karate is not supporting firefox?

[DineshTejaj]

Thank @paaco. I tried also before with Chrome but nothing, I'll retry with your suggestions with Firefox.

@forlixdev @paaco for me when am to launch firfox driver, the browser is not getting opened up, Can u please help me here, Can u please give me the feature on how to launch firefox browser

[ptrthomas]

Karate is not supporting firefox?

to the best of my knowledge, yes ;)

[paaco]

@DineshTejaj Please make sure geckodriver(.exe) from https://github.com/mozilla/geckodriver/releases is in your path because it says literally:
"Cannot run program "geckodriver" (in directory "target\geckodriver_1654765502546"): CreateProcess error=2, The system cannot find the file specified"

Firefox requires a driver to be placed in between Karate and the browser. (maybe still Edge, not sure about the chromium Edge nowadays)

[DineshTejaj]

@paaco : I tried with the gekodriver.exe now the firefox is opening perfectly but on entering a text in a text box am getting an error, only if am using firefox, if I use same lines of code for chrome it is working , please help me if am missing something or I need to add any configs

code:
Scenario: to set the config and driver

• configure driver = {type:'geckodriver' , executable:'C:\Users\dinesh\Downloads\geckodriver-v0.31.0-win64\geckodriver.exe'}
• driver 'https://courses.ultimateqa.com/users/sign_in'
• screenshot()
• driver.maximize()
  • input("//*[@id='user[email]']","abc@gmail.com")

Note: Browser is getting launched, url is getting navigated, the email id "abc@gmail.com" is also entered but after that getting below error please help

• input("//*[@id='user[email]']","abc@gmail.com")
  js failed:

01: input("//*[@id='user[email]']","abc@gmail.com")
<<<<
org.graalvm.polyglot.PolyglotException: Expected to find an object with property ['message'] in path $['value'] but found 'null'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

• com.jayway.jsonpath.internal.path.PropertyPathToken.evaluate(PropertyPathToken.java:71)
• com.jayway.jsonpath.internal.path.PathToken.handleObjectProperty(PathToken.java:81)
• com.jayway.jsonpath.internal.path.PropertyPathToken.evaluate(PropertyPathToken.java:79)
• com.jayway.jsonpath.internal.path.RootPathToken.evaluate(RootPathToken.java:62)
• com.jayway.jsonpath.internal.path.CompiledPath.evaluate(CompiledPath.java:99)
• com.jayway.jsonpath.internal.path.CompiledPath.evaluate(CompiledPath.java:107)
• com.jayway.jsonpath.JsonPath.read(JsonPath.java:185)

@ptrthomas please advice and help

[ptrthomas]

I'm locking this thread. please continue the discussion on stack overflow where you have already opened an issue: https://stackoverflow.com/q/72564273/143475

we need the community to step up and contribute any gaps. I will also say that if this process if followed, the chances of the project team providing a fix increase: https://github.com/karatelabs/karate/tree/develop/examples/ui-test