Migrate functional UI tests from sumo-tests to kitsune

[davehunt]

This copies the functional UI tests from the https://github.com/mozilla/sumo-tests repository, which will be deactivated once this is merged. This means these tests will be run for every commit, and allow the tests to be maintained alongside the application code base. This is part of an effort by Web QA to improve the visibility of these automated tests, and to bring them into the developer workflows.

At this time only three of the functional UI tests are passing in Travis CI. When running a local instance of kitsune I get many more tests passing, so I suspect that the instance running in Travis is not entirely representative of a fully running application. I've marked the passing tests as 'smoke', so we'll get notifications in GitHub if any of these tests regress. The full suite is also running but is allowed to fail - this should allow someone to investigate the cause of the failures and hopefully once whatever issues are addressed, we can add the 'smoke' marker to more and more tests.

I've renamed this suite from 'smoketests' as there are likely to be more than just smoke tests here. The full suite will continue to run as appropriate against dev/stage/prod but will run from this repository once this patch lands. I've also included a documentation update, which should help developers to run these tests locally.

/cc @rbillings @stephendonner @Osmose

[rbillings]

Nit: s/b "Follow the steps"

[mythmon]

@davehunt Awesome! I'm really glad to see this.

Looking at the errors you ran into, I think I know what is going on in a couple places. @safwanrahman discovered in #2721 that running the tests with a very small (virtual) screen causes strange errors, including ones like you're seeing. In particular, the site is redirecting to SSL, and for some reason changing the screen size prevents that. Crazy, but it was pretty easy to verify in #2721.

It may also be worth running the tests entirely with SSL using runsslserver, described here. This would produce a more prod-like environment, and might fix some problems.

[safwanrahman]

Is it possible to include in the test.txt?

[mythmon]

See my comment on the next line. This is here so that we can configure normal test runs to ignore the pytest tests. It should be in dev.txt.

[davehunt]

Looking at the errors you ran into, I think I know what is going on in a couple places. @safwanrahman discovered in #2721 that running the tests with a very small (virtual) screen causes strange errors, including ones like you're seeing. In particular, the site is redirecting to SSL, and for some reason changing the screen size prevents that. Crazy, but it was pretty easy to verify in #2721.

I haven't changed the screen size, it's still set to 1280x1024x16 in https://github.com/mozilla/kitsune/pull/2795/files#diff-0f775ad83429f23374705da3dafdbd63R7

It may also be worth running the tests entirely with SSL using runsslserver, described here. This would produce a more prod-like environment, and might fix some problems.

I'd like to suggest pushing this investigation to a separate issue. This is already a reasonably large patch, and I'd like to encourage follow ups that address the failing tests and increase the size of the smoke suite.

[mythmon]

I'm ok with pushing this off to future work. Even this is an improvement over what we have now.

[davehunt]

I've pushed the two easy tweaks from the review comments. Thanks @safwanrahman and @mythmon for the feedback!

[mythmon]

Ok. Lets push the harder stuff off to the future.

@davehunt Can you make sure the remaining tasks get recorded somewhere? My preference would be bugs in the Sumo product.

[davehunt]

I've raised bug 1251698 for investigating the failures and increasing the smoke test suite.