UI test for PR #26979 #27191

individual-it · 2017-02-21T09:59:05Z

Description

This is an UI test for the PR #26979 that fixes the issue #26975

This also includes the complete setup of the selenium infrastructure that can be used as the basis of further UI tests. After the discussion in #27057 behat+mink is used instead of the facebook webdriver.

For this to work travis starts behat/mink tests. And they connect to selenium running on https://saucelabs.com

Related Issue

PR #26979
issue #26975

List of PRs/issues that potentially would benefit of UI tests:

Can't set a user's quota back to Unlimited #26975
Fix long display names #27059
files_external: Keep checkbox and its label on one line #25120
Improve quota validation including a regexp which matches expected formats #25358
Rows are not updated after sharing via web ui. #27294 nextcloud/server#3688
Fixed disappearing of share info in file view #26534 nextcloud/server#2010

Motivation and Context

See issue #27055
Some bugs are hard to catch with karma & PHPUnit tests. Primarily this are front-end bugs. This PR implements end-to-end testing environment run on real browsers. Currently only Chrome browser is used, but it can be extended to multiple browsers/platforms including mobile browsers.
Currently my own sauce-labs account is used. Of course it would be good that a oC account would be used in future.

run this tests locally

download Selenium standalone server jar from http://docs.seleniumhq.org/download/
download the chromedriver from: https://sites.google.com/a/chromium.org/chromedriver/ and place it in the same folder as the selenium server
start selenium: java -jar selenium-server-standalone-3.0.1.jar -port 4445
set the 3 environment variables: SRV_HOST_NAME, SRV_HOST_URL and SRV_HOST_PORT e.g.:

export SRV_HOST_NAME=localhost
export SRV_HOST_URL=owncloud-core
export SRV_HOST_PORT=80

if no webserver is already running start php development server with: bash tests/travis/start_php_dev_server.sh (leave SRV_HOST_URL empty if you are using the php dev. server export SRV_HOST_URL="")
it will bind to: $SRV_HOST_NAME:$SRV_HOST_PORT
run tests: bash tests/travis/start_behat_tests.sh

limitations

Only PRs from local branches are tested. PRs that come from a fork cannot be tested as they do not have access to the saucelabs username+key.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

How Has This Been Tested?

@individual-it joined travis with their own account on sauce labs and has run multiple tests. Created bugs and run tests again to see if they are failing.

Screenshots (if appropriate):

travis job:
https://travis-ci.org/individual-it/owncloud-core/jobs/203733741
saucelabs job:
https://saucelabs.com/beta/tests/840fed2401854c50ab20109ac2812fb3/watch#305

This sets up UI tests. It uses behat/mink/selenium to run the tests on saucelabs.com Included are some basic tests 1. Login as admin 2. Tests to cover the fix made in PR #26979

mention-bot · 2017-02-21T09:59:08Z

@individual-it, thanks for your PR! By analyzing the history of the files in this pull request, we identified @DeepDiver1975, @PVince81 and @phisch to be potential reviewers.

SergioBertolinSG · 2017-02-22T13:07:19Z

start selenium:java -jar selenium-server-standalone-3.0.1.jar -port 4445

What version of java is good for this step? openjdk can be used there?

individual-it · 2017-02-22T15:20:15Z

According to the selenium github page you need Java 8 https://github.com/vvo/selenium-standalone/blob/master/README.md recent OpenJDK should be fine The java version on my Debian dev. machine is: openjdk version "1.8.0_111" OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14) OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

SergioBertolinSG · 2017-02-24T10:23:42Z

tests/ui/features/users.feature

+		|Unlimited  |1 GB        |
+		|1 GB       |5 GB        |
+		|5 GB       |Unlimited   |
+		|1 GB       |Unlimited   |


This feature cannot work, seems like a snippet.

Why shouldn't it work? At least it works fine for me. It runs 5 scenarios from that table. Check the bottom travis output:
https://travis-ci.org/individual-it/owncloud-core/jobs/203733741

Ahh right. It failed in my environment for different reasons.

SergioBertolinSG · 2017-02-24T10:27:22Z

tests/travis/start_behat_tests.sh

+#
+
+
+export BEHAT_PARAMS='{"extensions" : {"Behat\\MinkExtension" : {"base_url" : "http://'$SRV_HOST_NAME:$SRV_HOST_PORT/'","selenium2":{"wd_host":"http://'$SAUCE_USERNAME:$SAUCE_ACCESS_KEY'@localhost:4445/wd/hub"}}}}'


If you have localhost as hostname and a path to access your server this sentence doesn't work as it uses:

localhost/path:80 which doesn't work it has to be localhost:80/path

yes that is true, need to have $SRV_HOST_URL here too
always tested it with "start_behat_tests.sh" and that does not have any path
Also just found out that if you use port 80, the browser does not display it and so you get:
Expected to be on "http://localhost:80/index.php/login" but found "http://localhost/index.php/login" instead (SensioLabs\Behat\PageObjectExtension\PageObject\Exception\UnexpectedPageException)

both problems should be fixed now

SergioBertolinSG · 2017-02-24T11:34:37Z

Quota validation has this issue open #25358 .

DeepDiver1975 · 2017-03-01T10:30:26Z

I don't like that all travis tests are failing now - can we isolate this to only have selenium failing?

@individual-it can you please quide me to the docs on how to integrate travis and sauce? THX

individual-it · 2017-03-01T14:16:18Z

The travis tests are failing because the saucelab add-on is enabled but the sauce credentials are not set up in the owncloud travis account. All tests pass on my fork: https://travis-ci.org/individual-it/owncloud-core/builds/204940857

If you agree with the concept you should do this to integrate it and make all tests pass:

register to saucelabs and enrol in the open source plan https://saucelabs.com/open-source
in travis-ci.org set for the core repro the environment variables SAUCE_USERNAME SAUCE_ACCESS_KEY
generate the secure string with travis encrypt "your-access-key" https://docs.travis-ci.com/user/sauce-connect
get the other changes I've made in .travis.yml into the master .travis.yml. If you like to have as little changes as possible only get those from line: 17,18,34-37 (of course you would have to change line 35+37)
then I can make a new PR with all the other files included from a local branch and then all tests should pass

more information is available here: https://docs.travis-ci.com/user/sauce-connect

DeepDiver1975 · 2017-03-06T10:59:03Z

@individual-it I have added the credentials in the travis settings as environment variables.

Mind rebasing and removing your credentials? THX

individual-it · 2017-03-06T12:43:32Z

right, my instructions were far too complicated, most of it I can make in the PR itself.
So here is the PR from a local branch #27316
This PR is not needed any-more, so closing it

lock · 2019-08-03T20:01:32Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

individual-it added 4 commits February 16, 2017 15:13

build selenium branches with travis

c33a5a3

setup of UI tests

b6387c4

This sets up UI tests. It uses behat/mink/selenium to run the tests on saucelabs.com Included are some basic tests 1. Login as admin 2. Tests to cover the fix made in PR #26979

fixing a typo

09efe4b

Use clearer name setUpScenario to be consistent with PR #27189

a807580

individual-it requested review from DeepDiver1975 and SergioBertolinSG February 21, 2017 10:01

individual-it added the 3 - To Review label Feb 21, 2017

SergioBertolinSG suggested changes Feb 24, 2017

View reviewed changes

make test work when using port 80 and/or path in URL

aab0d72

individual-it mentioned this pull request Feb 28, 2017

fix for #25358 #27275

Merged

9 tasks

DeepDiver1975 self-assigned this Mar 2, 2017

SergioBertolinSG mentioned this pull request Mar 2, 2017

Run karma test on CI against several browsers owncloud/QA#385

Open

individual-it mentioned this pull request Mar 6, 2017

UI test for PR #26979 #27316

Merged

9 tasks

individual-it closed this Mar 6, 2017

lock bot locked as resolved and limited conversation to collaborators Aug 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UI test for PR #26979 #27191

UI test for PR #26979 #27191

individual-it commented Feb 21, 2017 •

edited

Loading

mention-bot commented Feb 21, 2017

SergioBertolinSG commented Feb 22, 2017

individual-it commented Feb 22, 2017 via email •

edited

Loading

SergioBertolinSG Feb 24, 2017

individual-it Feb 24, 2017

SergioBertolinSG Feb 24, 2017

SergioBertolinSG Feb 24, 2017

individual-it Feb 24, 2017

individual-it Feb 24, 2017

SergioBertolinSG commented Feb 24, 2017

DeepDiver1975 commented Mar 1, 2017

individual-it commented Mar 1, 2017

DeepDiver1975 commented Mar 6, 2017

individual-it commented Mar 6, 2017

lock bot commented Aug 3, 2019

		#


		export BEHAT_PARAMS='{"extensions" : {"Behat\\MinkExtension" : {"base_url" : "http://'$SRV_HOST_NAME:$SRV_HOST_PORT/'","selenium2":{"wd_host":"http://'$SAUCE_USERNAME:$SAUCE_ACCESS_KEY'@localhost:4445/wd/hub"}}}}'

UI test for PR #26979 #27191

UI test for PR #26979 #27191

Conversation

individual-it commented Feb 21, 2017 • edited Loading

Description

Related Issue

List of PRs/issues that potentially would benefit of UI tests:

Motivation and Context

run this tests locally

limitations

Types of changes

Checklist:

How Has This Been Tested?

Screenshots (if appropriate):

mention-bot commented Feb 21, 2017

SergioBertolinSG commented Feb 22, 2017

individual-it commented Feb 22, 2017 via email • edited Loading

SergioBertolinSG Feb 24, 2017

Choose a reason for hiding this comment

individual-it Feb 24, 2017

Choose a reason for hiding this comment

SergioBertolinSG Feb 24, 2017

Choose a reason for hiding this comment

SergioBertolinSG Feb 24, 2017

Choose a reason for hiding this comment

individual-it Feb 24, 2017

Choose a reason for hiding this comment

individual-it Feb 24, 2017

Choose a reason for hiding this comment

SergioBertolinSG commented Feb 24, 2017

DeepDiver1975 commented Mar 1, 2017

individual-it commented Mar 1, 2017

DeepDiver1975 commented Mar 6, 2017

individual-it commented Mar 6, 2017

lock bot commented Aug 3, 2019

individual-it commented Feb 21, 2017 •

edited

Loading

individual-it commented Feb 22, 2017 via email •

edited

Loading