Migrate functional UI tests from sumo-tests to kitsune

.travis.yml

@@ -1,17 +1,22 @@
 sudo: false
 language: python
 python: 2.7
+addons:
+  firefox: "latest"
 node: '0.10'
 env:
   globals:
   - DJANGO_LIVE_TEST_SERVER_ADDRESS="localhost:9900-9999"
   - PIP_DOWNLOAD_CACHE="pip_cache"
   matrix:
   - TEST_SUITE=lint
-  - TEST_SUITE=smoke ELASTICSEARCH_VERSION=1.2.4
+  - TEST_SUITE=ui ELASTICSEARCH_VERSION=1.2.4
+  - TEST_SUITE=ui MARK_EXPRESSION=smoke ELASTICSEARCH_VERSION=1.2.4
   - TEST_SUITE=django ELASTICSEARCH_VERSION=1.2.4
 matrix:
   fast_finish: true
+  allow_failures:
+   - env: TEST_SUITE=ui ELASTICSEARCH_VERSION=1.2.4
 cache:
   directories:
   - /home/travis/virtualenv

docs/hacking_howto.rst

@@ -183,6 +183,11 @@ required, but are helpful::
 
     $ ./peep.sh install -r requirements/dev.txt
 
+If you intend to run the function UI tests, you will also need to install the
+appropriate dependencies::
+
+    $ ./peep.sh install -r requirements/test.txt
+
 If you have any issues installing via ``peep``, be sure you have the required
 header files from the packages listed in the requirements section above.
 

docs/tests.rst

@@ -167,33 +167,91 @@ Here are a few tips for writing tests:
 .. _Mocha: https://mochajs.org/
 
 
-Smoketests
-==========
+Functional UI Tests
+===================
 
-We can do more comprehensive front-end testing with the smoketests.
-They're located in the ``smoketests/`` directory. See the ``README.rst``
-file in that directory for how to set them up as well as other details.
+We can do more comprehensive front-end testing with the functional UI tests.
+They're located in the ``tests/functional`` directory.
 
+Installing dependencies
+-----------------------
 
-.. _tests-chapter-qa-test-suite:
+Followed the steps in :ref:`the installation docs <hacking-howto-chapter>`,
+including the test dependencies to make sure you have everything you need to
+run the tests. If you're running the tests against a deployed environment then
+there's no need to install anything other than the test dependencies.
 
-The QA test suite
-=================
+Create test users
+-----------------
+
+Some of the tests require logging in as a administrator, and others require
+logging in as a user. To run these tests you will need to create accounts in
+the target environment. If you're running against a local instance of the
+application you can create these users by running the following script::
+
+  $ ./manage.py shell < ./scripts/create_user_and_superuser.py
+
+If you want to run the tests that require administrator access against a
+deployed instance, then you will need to ask someone on IRC to upgrade one of
+your test accounts.
+
+The credentials associated with the test users are stored in a JSON file, which
+we then pass to the tests via the command line. If you used the above mentioned
+script, then these users are stored in ``/scripts/travis/variables.json``. The
+variable file needs to be referenced on the command line when running the
+tests.
+
+The following is an example JSON file with the values missing. You can use this
+as a template:
+
+.. code:: json
+
+   {
+     "users": {
+       "default": {
+         "username": "",
+         "password": "",
+         "email": ""},
+       "admin": {
+         "username": "",
+         "password": "",
+         "email": ""}
+     }
+   }
+
+For the purposes of the examples below, assume you named your copy of the file
+``my_variables.json``.
+
+Running the tests
+-----------------
+
+Tests are run using the command line. Below are a couple of examples of running
+the tests:
+
+To run all of the desktop tests against the default environment::
+
+  $ py.test --driver Firefox --variables my_variables.json tests/functional/desktop
+
+To run against a different environment, pass in a value for ``--base-url``,
+like so::
 
-QA has their own test suite. The code is located on github at
-`<https://github.com/mozilla/sumo-tests/>`_.
+  $ py.test --base-url https://support.allizom.org --driver Firefox --variables my_variables.json tests/functional/desktop
 
-There are three test suites. They differ in what they do and where
-they run:
+To run the mobile tests you will need to target a mobile device or emulator
+using a tool like `Appium <http://appium.io/>`_::
 
-    ============  ===========================================================
-    name          description
-    ============  ===========================================================
-    sumo.fft      runs on -dev
-    sumo.stage    runs on -stage
-    sumo.prod     runs on -prod and is read-only (it doesn't change any data)
-    ============  ===========================================================
+  $ py.test --driver Remote --port 4723 \
+  --capability platformName iOS \
+  --capability platformVersion 9.2 \
+  --capability deviceName "iPhone 6" \
+  --capability browserName Safari \
+  --variables my_variables.json \
+  tests/functional/mobile
 
-There's a qatestbot in IRC. You can ask it to run the QA tests by::
+Alternatively, if you run the mobile tests in Firefox the user agent will be
+changed to masquerade as a mobile browser.
 
-    qatestbot build <test-suite>
+The pytest plugin that we use for running tests has a number of advanced
+command line options available. To see the options available, run
+``py.test --help``. The full documentation for the plugin can be found
+`here <http://pytest-selenium.readthedocs.org/>`_.

requirements/default.txt

@@ -209,6 +209,9 @@ MySQL-python==1.2.3c1
 # sha256: sboSwcRTI_2Glb5oznIoRWuG5RI08WcCfrvEzNqFLGE
 nose==1.1.2
 
+# sha256: REZqm8tW0uVodQ-RUE0SeMdOq7JZowWwbpdbh7UWNdo
+nose-exclude==0.4.1
+
 # sha256: oFHwTujsMwUFWrNNh7NsmkSTdeB8fWoFvK-kgynKx8M
 oauthlib==0.7.2
 

requirements/test.txt

@@ -0,0 +1,31 @@
+# sha256: 9m3Up1GXJaG34UrZrn09-OCbLaiAYjhuCOlByvwO8-Y
+# sha256: 0rkJx5RYMuHBnPrNlueNpova3GVkQM_H3-WbdmdE64w
+execnet==1.4.1
+
+# sha256: Sj5PMADBI4Naw5yrXMxRBkIVO8R7wfE-K7tTA5VArmk
+# sha256: plAZY8cl_CVU2r_s6K6aj7XhScCsCkL9KwLFwcV_wRQ
+py==1.4.31
+
+# sha256: f60wfCePxbGNJ9V6zO0x6euZ90zMyBccMJhi7eu7orc
+# sha256: 1anD2zUVq5WQK6CMCdHpQaIaRA3C1l8xusAmZ1CKnvw
+pytest==2.7.3
+
+# sha256: zzI98ijiZdZ1XsOaJvKXGsxZoxsN4M_QJtNBJJ2-72k
+# sha256: 3-epy0EWdBTKe56Fr71mlOjMg_2_V9DlRcIVwyzUDNY
+pytest-html==1.8.0
+
+# sha256: 6C8KJlsOI4rEKsJ115MT0KfgvvGkUGM66z1lScwU9Rc
+# sha256: vSEhAi_zJVzoL67A7zYCRi7GvOnKYntTRimGz8mzkek
+pytest-selenium==1.2.1
+
+# sha256: naX4m9sivV9wDs8QeH8jkZC3gyqzRFvzI0PNTlFBaCU
+pytest-variables==1.3
+
+# sha256: Sl4RmRIvop4wF9jRifWczF2C6EFHS6Kh7sDolgYVNiM
+pytest-xdist==1.14
+
+# sha256: xXeBXdAPE5QgP8ROuXlySwmPiCZKnviY7kW45enPWH8
+requests==2.9.1
+
+# sha256: ggVQp0DKH3RsOZoBAZhsDm-U-_48b5duP2lNtFLL4SQ
+selenium==2.52.0

scripts/create_user_and_superuser.py

@@ -0,0 +1,22 @@
+import json
+from django.contrib.auth.models import User
+
+
+def create_superuser(username, password, email):
+    User.objects.create_superuser(username=username, password=password, email=email)
+
+
+def create_user(username, password, email):
+    User.objects.create_user(username=username, password=password, email=email)
+
+
+def enter_data(data):
+    all_users = data['users']
+    user = all_users['default']
+    admin = all_users['admin']
+    create_user(username=user['username'], password=user['password'], email=user['email'])
+    create_superuser(username=admin['username'], password=admin['password'], email=admin['email'])
+
+with open('./scripts/travis/variables.json', 'r') as f:
+    data = json.load(f)
+    enter_data(data)

scripts/travis/dispatch.sh

@@ -10,8 +10,8 @@ case $SUITE in
     lint )
         pre-commit run --all-files
         ;;
-    smoke )
-        scripts/travis/smoketests.sh
+    ui )
+        scripts/travis/uitests.sh
         ;;
     django )
         scripts/travis/test.sh

scripts/travis/install.sh

@@ -12,12 +12,9 @@ echo "Install Python dependencies"
 ./peep.sh install -r requirements/default.txt
 echo
 
-# Installing dependencies for smoke tests
-if [[ $TEST_SUITE == "smoke" ]]; then
-  virtualenv venv_smoketests
-  source venv_smoketests/bin/activate
-  pip install -r smoketests/requirements.txt
-  deactivate
+# Installing dependencies for UI tests
+if [[ $TEST_SUITE == "ui" ]]; then
+  ./peep.sh install -r requirements/test.txt
 fi
 
 # Optimization: None of the rest is needed for lint tests.

scripts/travis/uitests.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# pwd is the git repo.
+set -ex
+
+echo "Starting XVFB for UI tests"
+export DISPLAY=:99.0
+/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16
+
+echo 'Starting a server'
+./manage.py shell < ./scripts/create_user_and_superuser.py
+./manage.py generatedata
+./manage.py runserver &
+sleep 3
+
+CMD="py.test"
+CMD="${CMD} -r a"
+CMD="${CMD} --verbose"
+CMD="${CMD} --base-url http://localhost:8000"
+CMD="${CMD} --html results.html"
+CMD="${CMD} --driver Firefox"
+CMD="${CMD} --variables scripts/travis/variables.json"
+if [ -n "${MARK_EXPRESSION}" ]; then CMD="${CMD} -m \"${MARK_EXPRESSION}\""; fi
+CMD="${CMD} tests/functional"
+
+echo 'Running UI tests'
+eval ${CMD}
+
+echo 'Booyahkasha!'

scripts/travis/variables.json

@@ -0,0 +1,12 @@
+{
+  "users": {
+    "default": {
+      "username": "testuser",
+      "password": "testuserpass",
+      "email": "testuser@testeserver.org"},
+    "admin": {
+      "username": "testsuperuser",
+      "password": "superuserpass",
+      "email": "superuser@testserver.org"}
+  }
+}

setup.cfg

@@ -4,3 +4,10 @@ exclude = *migrations*,
     kitsune/sumo/db_strings.py,
     scripts/peep.py
 max-line-length = 99
+
+[nosetests]
+exclude-dir=tests/functional
+
+[pytest]
+base_url=https://support.allizom.org
+sensitive_url=mozilla\.org