[SauceLabs] Empty custom-data key causes tests not to run

[dantleech]

If an empty custom-data key is sent to the server it prevents the tests from running.

i,e,

sauce_labs:
    extensions:
        Behat\MinkExtension:
            base_url: 'http://localhost:8000/'
            sessions:
                default:
                    sauce_labs:
                        username: sulu-cmf
                        access-key: xxx-xxx-xxx

Does not work.

The following actions made it work:

• Removing the custom-data parameter from SauceLabsFactory
• Adding values to the field (e.g. release: 1.0)

[stof]

hmm, so SauceLabs fails when receiving an empty array for custom-data ? This is a pain.

I will ork on a fix this evening unless someone else does it first.

[dantleech]

If you point me in the right direction I can make a PR.

[gggeek]

Bitten by the same thing just today.

To me the pain point is that by default the saucelabs (selenium2) driver

1. does apparently not throw an error when it gets back a 404 or other http error from its curl calls (are 404 part of the selenium2 spec/implementation?). This means that it is up to my code to detected what is wrong (nb: these are not 404 on the target site, but 404 on selenium).
2. does apparently not have a debug switch to enable display/logging of all http requests and responses sent from behat to the selenium server - but maybe I just missed it

[aik099]

does apparently not throw an error when it gets back a 404 or other http error from its curl calls (are 404 part of the selenium2 spec/implementation?). This means that it is up to my code to detected what is wrong (nb: these are not 404 on the target site, but 404 on selenium).

This is default behavior for all Mink drivers to allow assertion of these 404 and 500 pages to contain correct HTML. If an exception would be thrown on sight, then you could not have step in Behat scenario to validate that after visiting a page you've got proper HTTP response code.

does apparently not have a debug switch to enable display/logging of all http requests and responses sent from behat to the selenium server - but maybe I just missed it

How each Mink driver does it's job isn't up to Mink to debug. For example if you click a link on a page, then driver internally will catch that and make an HTTP request, but this won't pass via Session::visit method at all.

[stof]

does apparently not have a debug switch to enable display/logging of all http requests and responses sent from behat to the selenium server - but maybe I just missed it

Actually, the Selenium server already logs all these calls to STDERR when running it, so you could get them from this place to debug things

[gggeek]

@stof for BrowserStack and SauceLabs the selenium server is outside my control.
I might be able to connect to their web-based admin consoles and check if there's any logs there, but it is definitely easier to have all debug info at my fingertips on the client side (think debugging ajax calls in the browser before firebug)

[gggeek]

@aik099 I think you might be misunderstanding what I meant. There are 2 classes of errors at play here:

case a):
request: mink => selenium => browser => site
response: site gives a 40x/50x => browser => selenium => mink

case b)
request: mink => selenium
response: selenium gives a 40x/50x => mink

The bug discussed here is about case B.

As long as the rest protocol in use allows to distinguish between the 404A and the 404B, then, imho, the mink driver should throw an exception on 404B. It should of course allow the 404A to reach the testing application.

And it seems that this is the case. By testing a page which does not exist on my website, this is the communication between mink and selenium I got. Notice it has a 200, not a 404

    │ post to: string "user:token@hub.browserstack.com/wd/hub/session/5c336d50205c952b394d3a3ebaaad0b44157e593/url"
    │ payload: array(1) {
    │   ["url"]=>
    │   string(75) "http://www-fr.ez.merck.staging.mtp.infrawan.net/This_is_a_very_unlikely_url"
    │ }
    │ response: string(161) "{"state":"success","sessionId":"5c336d50205c952b394d3a3ebaaad0b44157e593","hCode":16763737,"value":null,"class":"org.openqa.selenium.remote.Response","status":0}"

[stof]

@gggeek the calls are also out of Mink controls, and the Selenium library does not have logging in place AFAIK. but anyway, this is off-topic in this issue

[stof]

@gggeek AFAIK, the Selenium library is expected to throw an exception in the second case. Otherwise, it is a bug in this library (which is not Mink, nor MinkExtension).
And anyway, for Selenium2-specific issues, please discuss them in the Selenium2Driver repo, not in this totally unrelated discussion (which is about the config in the MinkExtension).

[gggeek]

@aik099 about 'How each Mink driver does it's job isn't up to Mink to debug': I have to disagree.

Since the code here (MinkExtension) supports remote Selenium servers, making it easy to troubleshoot any connection/configuration problem to those servers is a feature of great value to any developer.

I can post this as a separate request to another project though. Would behat/mink-selenium2-driver be the appropriate one?

[stof]

@gggeek please open separate issues to discuss this stuff, because it is not related at all to the bug reported here (and this issue will be closed as soon as the bug reported here gets fixed)

[dantleech]

There is also an issue with the selenium-version, the current version is 2.40.0, but the default as defined by the Factory is 2.31.0. The factory version limits the versions of the browsers you can use afaict.

I think that the "options" should default to null and should be stripped from the configuration before sending.

[stof]

@dantleech for the selenium-version, see #186