Selenium Question(s)

[kevinquillen]

I followed the docs on installing Selenium for DrupalVM, my config.yml for Behat looks correct. Upon executing (from PHPStorm, which is executing it remotely), it times out after 60 seconds and says it could not connect to port 7055 (not sure where it is getting that from). The config.yml is set to 4444.

If I were to get it running, is there a way to make it take screenshots on error?

I want a way where I can run Behat (which I can do with the goutte driver, without using Selenium) - but have the option to view what is happening from the browser so I can investigate when a test fails.

It's been a very long time since I have ran Selenium interactively, and want to get the option to do that again. With Behat it is possible to setup multiple profiles now, granting much more flexibility, and I have found that the best way to test AJAX functionality is with Selenium while watching it actually do it in a browser.

[geerlingguy]

@kevinquillen - Yes, you can get it working inside Drupal VM with the ability to do screenshots through the phantomjs driver... but I usually recommend people either set up CI with that level of detail/automation on a separate server (e.g. Jenkins) or on their host machine, since it will use an actual browser instance, like FireFox. Drupal VM is much better imo for building the tests with a headless browser, though you can get it working either way (if you want screenshots).

I'll leave this ticket open in case anyone else wants to chime in with particular tips/tricks.

[kevinquillen]

I'll try again. I ran a selenium server on my host machine, told Behat that the selenium2 wd_host was 127.0.0.1:4444/wd/host. It just seemed to hang there.

Otherwise, everything else works great, I just want to be able to watch it execute in the browser on occasion.

[kevinquillen]

Here is an error I get when using @javascript to run scenarios, on the remote machine:

Could not open connection: Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms. Firefox console output:
r GMPProvider
1452182489157   addons.manager  DEBUG   Provider finished startup: GMPProvider
1452182489157   addons.manager  DEBUG   Starting provider: PluginProvider
1452182489157   addons.manager  DEBUG   Registering shutdown blocker for PluginProvider
1452182489158   addons.manager  DEBUG   Provider finished startup: PluginProvider
1452182489158   addons.manager  DEBUG   Completed startup sequence
1452182489269   DeferredSave.extensions.json    DEBUG   Save changes
1452182489269   addons.xpi  DEBUG   Updating XPIState for {"id":"fxdriver@googlecode.com","syncGUID":"i6-vQYnZ7XYe","location":"app-profile","version":"2.46.0","type":"extension","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"Firefox WebDriver","description":"WebDriver implementation for Firefox","creator":"Simon Stewart","homepageURL":null},"visible":true,"active":false,"userDisabled":false,"appDisabled":false,"descriptor":"/tmp/anonymous2640324584280819823webdriver-profile/extensions/fxdriver@googlecode.com","installDate":1452182488000,"updateDate":1452182488000,"applyBackgroundUpdates":1,"bootstrap":false,"size":8445045,"sourceURI":null,"releaseNotesURI":null,"softDisabled":false,"foreignInstall":true,"hasBinaryComponents":true,"strictCompatibility":false,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"3.0","maxVersion":"66.*"}],"targetPlatforms":[{"os":"Darwin","abi":null},{"os":"SunOS","abi":null},{"os":"FreeBSD","abi":null},{"os":"OpenBSD","abi":null},{"os":"WINNT","abi":"x86-msvc"},{"os":"Linux","abi":null}],"multiprocessCompatible":false,"signedState":0}
1452182489269   addons.xpi  DEBUG   getModTime: Recursive scan of fxdriver@googlecode.com
1452182489275   DeferredSave.extensions.json    DEBUG   Save changes
1452182489275   addons.xpi  DEBUG   Updating XPIState for {"id":"fxdriver@googlecode.com","syncGUID":"i6-vQYnZ7XYe","location":"app-profile","version":"2.46.0","type":"extension","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"Firefox WebDriver","description":"WebDriver implementation for Firefox","creator":"Simon Stewart","homepageURL":null},"visible":true,"active":false,"userDisabled":false,"appDisabled":true,"descriptor":"/tmp/anonymous2640324584280819823webdriver-profile/extensions/fxdriver@googlecode.com","installDate":1452182488000,"updateDate":1452182488000,"applyBackgroundUpdates":1,"bootstrap":false,"size":8445045,"sourceURI":null,"releaseNotesURI":null,"softDisabled":false,"foreignInstall":true,"hasBinaryComponents":true,"strictCompatibility":false,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"3.0","maxVersion":"66.*"}],"targetPlatforms":[{"os":"Darwin","abi":null},{"os":"SunOS","abi":null},{"os":"FreeBSD","abi":null},{"os":"OpenBSD","abi":null},{"os":"WINNT","abi":"x86-msvc"},{"os":"Linux","abi":null}],"multiprocessCompatible":false,"signedState":0}
1452182489343   addons.manager  DEBUG   Starting provider: <unnamed-provider>
1452182489343   addons.manager  DEBUG   Registering shutdown blocker for <unnamed-provider>
1452182489343   addons.manager  DEBUG   Provider finished startup: <unnamed-provider>
1452182489403   DeferredSave.extensions.json    DEBUG   Starting write
1452182489528   addons.repository   DEBUG   No addons.json found.
1452182489529   DeferredSave.addons.json    DEBUG   Save changes
1452182489530   DeferredSave.addons.json    DEBUG   Starting timer
1452182489539   addons.manager  DEBUG   Starting provider: PreviousExperimentProvider
1452182489539   addons.manager  DEBUG   Registering shutdown blocker for PreviousExperimentProvider
1452182489539   addons.manager  DEBUG   Provider finished startup: PreviousExperimentProvider
1452182489543   DeferredSave.extensions.json    DEBUG   Write succeeded
1452182489543   addons.xpi-utils    DEBUG   XPI Database saved, setting schema version preference to 17
1452182489584   DeferredSave.addons.json    DEBUG   Starting write
1452182489599   DeferredSave.addons.json    DEBUG   Write succeeded

behat.yml:

default:
  suites:
    default:
      contexts:
        - FeatureContext
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MarkupContext
    testmodule_features:
      paths:    [ %paths.base%/modules/testmodule/features ]
      contexts:
        - FeatureContext
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MarkupContext
  extensions:
    Behat\MinkExtension:
      goutte: ~
      javascript_session: selenium2
      selenium2:
        wd_host: http://local.d8contrib:4444/wd/hub
      base_url: http://local.d8contrib
    Drupal\DrupalExtension:
      blackbox: ~
      api_driver: 'drupal'
      drupal:
        drupal_root: '/var/www/docroot'
      region_map:
        content: "#content"
        highlighted: '.highlighted'

The default api driver works fine.

[kevinquillen]

This is odd... So, going off of that, I added this to drupalvm config.yml for firewall allowed ports:

  - "7055"

I also changed the selenium version to 2.48.2, which seems to be the latest stable release. Re provisioned, and now it works. I can run scenarios with selenium, or mix and match with the default driver or selenium.

This was necessary for what I was testing with BigPipe and all caching enabled in D8.

[geerlingguy]

Ah, I should've thought of that. The default set of allowed ports blocks the ports used for Selenium, so that makes sense. Let's add that to the docs (and/or open up that port in the default example config).

[kevinquillen]

So, back to the original issue now that that works... I added a new profile in my behat.yml:

hostmachine:
  suites:
    default:
      contexts:
        - FeatureContext
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MarkupContext
    testmodule_features:
      paths:    [ %paths.base%/modules/testmodule/features ]
      contexts:
        - FeatureContext
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MarkupContext
  extensions:
    Behat\MinkExtension:
      goutte: ~
      javascript_session: selenium2
      selenium2:
        wd_host: http://127.0.0.1:4444/wd/hub
      base_url: http://local.d8contrib
    Drupal\DrupalExtension:
      blackbox: ~
      api_driver: 'drupal'
      drupal:
        drupal_root: '/Users/kevinquillen/Servers/d8contrib/docroot'
      region_map:
        content: "#content"
        highlighted: '.highlighted'

I run this command in terminal, on my machine:

$ vendor/bin/behat -c behat.yml --profile hostmachine

I also started a selenium server on the host.

What I expect to happen is that Behat connects to the local selenium, opens a browser and I can watch it run through the scenarios.

Instead, this error pops up:

[PDOException]                              
SQLSTATE[HY000] [2002] Operation timed out 

The settings.php file has the right MYSQL connection, so I am not sure where it is failing at the moment. No activity occurs in the terminal window running selenium either.

[kevinquillen]

Just adding on to notes here... here is how you can submit parameters to your context(s):

contexts:
        - DesktopContext:
            browserWidth: 1440
            browserHeight: 900
            screenshot_path: '/var/www/docroot/reports/behat/screenshots'
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MarkupContext

DesktopContext:

<?php

use Drupal\DrupalExtension\Context\RawDrupalContext;
use Behat\Behat\Context\SnippetAcceptingContext;
use Behat\Gherkin\Node\PyStringNode;
use Behat\Gherkin\Node\TableNode;

/**
 * Defines application features from the specific context.
 */
class DesktopContext extends RawDrupalContext implements SnippetAcceptingContext {

  /**
   * The assigned width from our Behat config for this context.
   * @var $browserWidth
   */
  protected $browserWidth;

  /**
   * The assigned height from our Behat config for this context.
   * @var $browserWidth
   */
  protected $browserHeight;

  /**
   * The file path where screenshots are to be saved.
   * @var $screenshot_path
   */
  protected $screenshot_path;

  /**
   * Initializes context.
   *
   * Every scenario gets its own context instance.
   * You can also pass arbitrary arguments to the
   * context constructor through behat.yml.
   *
   * @param int $browserWidth
   * @param int $browserHeight
   * @param string $screenshot_path
   */
  public function __construct($browserWidth, $browserHeight, $screenshot_path) {
    $this->browserWidth = $browserWidth;
    $this->browserHeight = $browserHeight;
    $this->screenshot_path = $screenshot_path;
  }

  /**
   * @BeforeScenario
   * @param \Behat\Behat\Hook\Scope\BeforeScenarioScope $scope
   */
  public function beforeScenario(Behat\Behat\Hook\Scope\BeforeScenarioScope $scope) {
    $this->getSession()->getDriver()->resizeWindow($this->browserWidth, $this->browserHeight);
  }

  /**
   * @AfterScenario
   * @param \Behat\Behat\Hook\Scope\AfterScenarioScope $event
   */
  public function after(\Behat\Behat\Hook\Scope\AfterScenarioScope $event) {

    // if the test failed, take a screenshot so we can see what Selenium saw.
    if (!$event->getTestResult()->isPassed()) {
      $directory = $this->screenshot_path . '/' . $event->getFeature()->getTitle() . '/' . $event->getScenario()->getTitle();

      // if the directory does not exist, create it
      if (!is_dir($directory)) {
        mkdir($directory, 0777, TRUE);
      }

      $this->saveScreenshot(NULL, $directory);
    }
  }
}

This sets the browser at the assigned dimension (otherwise, Selenium seems to start the browser in a small mobile resolution), and takes a screenshot on a failed scenario for review.

The reports directory is created at the root (per the parameter in behat.yml) and any necessary folders underneath are created recursively. Just an example, my implementation saves screenshots in FeatureName/ScenarioName directory so its clear what we are looking at.

I suppose this is good enough and we don't necessarily need to see it happening in real time, now that I figured out how to screenshot. Something seems funky with running it locally and executing on the VM anyway. A good next step would be implementing different drivers, so you could use say, Chrome, other than Firefox.

This can allow for creating different contexts, like TableContext or MobileContext, with specific dimensions. It was a little hard to discern this from the docs (Behat v3 is not the default navigation), but there is a lot of flexibility in regards to testing in this manner.

I should also note that I am using the Drupal Extension - which provides a lot of steps for you OOTB.

[kevinquillen]

While this works in DrupalVM, if you really want to see BDD tests run in real time or use Chrome over Firefox, you will want to run this alongside DrupalVM:

https://github.com/SeleniumHQ/docker-selenium

Fire up the hub, then one of the browser images (the Debug containers have a VNC viewer) open the VNC viewer and voila.

It is just easier to do that way than try to install additional dependencies into the VM when Docker can be dedicated for that (and just throw away when done).

[kevinquillen]

I restarted my VM, now I get this when running within the server:

Could not open connection: Curl error thrown for http POST to http://127.0.0.1:4444/wd/hub/session with params: {"desiredCapabilities":{"tags":["local","PHP 5.6.16-2+deb.sury.org~trusty+1"],"browser":"firefox","ignoreZoomSetting":false,"name":"Behat feature suite","browserName":"firefox"}}

Failed to connect to 127.0.0.1 port 4444: Connection refused

Neither 4444/7055 seem to be listed as ports with netstat -tulpn, even though they are listed as allowed in the config.yml. How can I open them up? nc -l 4444 and 7055 seems to be ineffective.

[geerlingguy]

@kevinquillen - Did you run vagrant provision to make sure the firewall is happy? You can also just check the rules within the vm with sudo iptables -L and make sure both ports are open.

[kevinquillen]

I did update the base box today. Let me provision again. sudo iptables -L does show both ports as open and listening.

[kevinquillen]

Booya. Provisioning fixed it. Was this because I updated the base box, or I halt/up without provisioning?

[geerlingguy]

Not entirely sure... when you provisioned, which tasked showed a [changed]? Could be a few different things :/

[kevinquillen]

Updated box this week and running into this issue again... previous fixes are no longer working. Get the old Could not open connection: Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms. Firefox console output: r GMPProvider error. Ports are open.

[kevinquillen]

Another new error:

Could not open connection: Unable to bind to locking port 7054 within 45000 ms

I am going to destroy and provision again. The ports are open, but something is not clicking.

[kevinquillen]

Looks like the solution is to destroy the Vagrant instance, remove the entries from known_hosts (using auto_network plugin can cause some weird issues around ssh keys), add the 7055 port in config.yml, ssh in and open port 7055 in the VM, and then vagrant provision.

I think I will try and offload my workflow for this to the dedicated docker from Selenium, unless I am doing something wrong here.

[geerlingguy]

Regarding the Firefox error... see: SeleniumHQ/selenium#2110

[geerlingguy]

For this and #421 , I've decided to add links to the issues in the docs for Behat, and leave it at that... I've found that many projects set up Selenium, PhantomJS, Node, Behat, and the Drupal plugins slightly differently, and there's no way I want to try to account for all the flavors in Drupal VM's docs.

Rather, as long as the base components are able to work, I point to resources like the official docs, implementations like BLT, and issues like these, and I leave it up to the reader to decide how to implement automated tests into their own projects.

[welly]

I'm struggling here with this a little bit. I've followed all the suggestions in the two issues linked from the docs but I'm getting a similar error when using @javascript:

Could not open connection: Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms. Firefox console output:
d.","homepageURL":null},{"locales":["ro"],"name":"Ubuntu Modifications","description":"Pachetul Ubuntu Firefox.","creator":"Canonical Ltd.","homepageURL":null},{"locales":["ru"],"name":"Ubuntu Modifications","description":"Ра��и�ени� Firefox дл� Ubuntu","creator":"Canonical Ltd.","homepageURL":null},{"locales":["sl"],"name":"Ubuntu Modifications","description":"Ubuntu razširitve za Firefox.","creator":"Canonical Ltd.","homepageURL":null},{"locales":["sv-SE"],"name":"Ubuntu Modifications","description":"Ubuntu-paket för Firefox.","creator":"Canonical Ltd.","homepageURL":null},{"locales":["uk"],"name":"Ubuntu Modifications","description":"Уб�н��в��к� доповненн� до Firefox.","creator":"Canonical Ltd.","homepageURL":null},{"locales":["zh-CN"],"name":"Ubuntu Modifications","description":"Ubuntu �����.","creator":"Canonical Ltd.","homepageURL":null},{"locales":["zh-TW"],"name":"Ubuntu Modifications","description":"Ubuntu Firefox ����","creator":"Canonical Ltd.","homepageURL":null}],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"9.0","maxVersion":"37.0a1"}],"targetPlatforms":[],"multiprocessCompatible":false,"seen":true}
1477607438434 DeferredSave.extensions.json DEBUG Save changes
1477607438436 addons.xpi DEBUG Updating database with changes to installed add-ons
1477607438436 addons.xpi-utils DEBUG Updating add-on states
1477607438437 addons.xpi-utils DEBUG Writing add-ons list
1477607438439 addons.xpi DEBUG Registering manifest for /usr/lib/firefox/browser/features/e10srollout@mozilla.org.xpi
1477607438445 addons.xpi DEBUG Calling bootstrap method startup on e10srollout@mozilla.org version 1.3
1477607438446 addons.xpi DEBUG Registering manifest for /usr/lib/firefox/browser/features/firefox@getpocket.com.xpi
1477607438447 addons.xpi DEBUG Calling bootstrap method startup on firefox@getpocket.com version 1.0.4
1477607438454 addons.xpi DEBUG Registering manifest for /usr/lib/firefox/browser/features/webcompat@mozilla.org.xpi
1477607438457 addons.xpi DEBUG Calling bootstrap method startup on webcompat@mozilla.org version 1.0
1477607438468 addons.manager DEBUG Registering shutdown blocker for XPIProvider
1477607438469 addons.manager DEBUG Provider finished startup: XPIProvider
1477607438470 addons.manager DEBUG Starting provider: LightweightThemeManager
1477607438470 addons.manager DEBUG Registering shutdown blocker for LightweightThemeManager
1477607438471 addons.manager DEBUG Provider finished startup: LightweightThemeManager
1477607438472 addons.manager DEBUG Starting provider: GMPProvider
1477607438502 addons.manager DEBUG Registering shutdown blocker for GMPProvider
1477607438503 addons.manager DEBUG Provider finished startup: GMPProvider
1477607438504 addons.manager DEBUG Starting provider: PluginProvider
1477607438504 addons.manager DEBUG Registering shutdown blocker for PluginProvider
1477607438505 addons.manager DEBUG Provider finished startup: PluginProvider
1477607438506 addons.manager DEBUG Completed startup sequence
1477607439212 addons.manager DEBUG Starting provider:
1477607439212 addons.manager DEBUG Registering shutdown blocker for
1477607439215 addons.manager DEBUG Provider finished startup:
1477607439219 DeferredSave.extensions.json DEBUG Starting write
1477607439532 addons.repository DEBUG No addons.json found.
1477607439538 DeferredSave.addons.json DEBUG Save changes
1477607439544 DeferredSave.addons.json DEBUG Starting timer
1477607439573 addons.manager DEBUG Starting provider: PreviousExperimentProvider
1477607439573 addons.manager DEBUG Registering shutdown blocker for PreviousExperimentProvider
1477607439573 addons.manager DEBUG Provider finished startup: PreviousExperimentProvider
1477607439578 DeferredSave.extensions.json DEBUG Write succeeded
1477607439579 addons.xpi-utils DEBUG XPI Database saved, setting schema version preference to 17
1477607439608 DeferredSave.addons.json DEBUG Starting write
1477607439678 DeferredSave.addons.json DEBUG Write succeeded
(Behat\Mink\Exception\DriverException)

I've opened up port 7055 as suggested above and also installed the earlier version of selenium server. No joy. Could anyone suggest anything else? iptables -L shows

ACCEPT tcp -- anywhere anywhere tcp dpt:7055

May have to look at the docker route but I'm pretty new to Docker.

Cheers