New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run Selenium tests on pull requests #13603

Closed
nijel opened this Issue Aug 22, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@nijel
Member

nijel commented Aug 22, 2017

As we have Selenium tests more or less working right now (the exception being f36ed72#commitcomment-23788220), I think it's necessary to run them on pull requests to ensure they don't break anything.

Running the tests against BrowserStack requires authentication token, which has to be stored securely (otherwise anybody would be able to use our account there). Because of this, the token is not exposed in the pull requests environment. This is pretty much well known problem, see travis-ci/travis-ci#1946

As a solution the Travis CI team has come with JWT based authentication which generates unique token for every build based on main token. This way the builds can access the service without exposing the main token. Unfortunately BrowserStack currently does not support this. I've asked their support today if there are any plans for supporting this.

If there are no plans on the BrowserStack side, we might consider switch to SauceLabs, where this is already implemented (and seems to work fine, I'm using it on https://github.com/WeblateOrg/weblate/).

nijel referenced this issue Aug 22, 2017

Merge pull request #13602 from mauriciofauth/tracking
Refactor tracking functions to static methods
@nijel

This comment has been minimized.

Show comment
Hide comment
@nijel

nijel Aug 22, 2017

Member

The BrowserStack JWT support depends on travis-ci/travis-scheduler#81

Member

nijel commented Aug 22, 2017

The BrowserStack JWT support depends on travis-ci/travis-scheduler#81

@ibennetch

This comment has been minimized.

Show comment
Hide comment
@ibennetch

ibennetch Aug 23, 2017

Member

I don't have an opinion about how to accomplish this; your proposed switch to SauceLabs is fine by me if BrowserStack has no plans to implement it. I don't have any better idea or have any disagreement with what you wrote, so go ahead and do what you think is best.

Member

ibennetch commented Aug 23, 2017

I don't have an opinion about how to accomplish this; your proposed switch to SauceLabs is fine by me if BrowserStack has no plans to implement it. I don't have any better idea or have any disagreement with what you wrote, so go ahead and do what you think is best.

@nijel

This comment has been minimized.

Show comment
Hide comment
@nijel

nijel Aug 23, 2017

Member

I'd prefer to stick with BrowserStack (as it's less work), but running Selenium tests on PR is something we should have rather sooner than later to avoid PRs break our tests (as has already happened with #13602).

Member

nijel commented Aug 23, 2017

I'd prefer to stick with BrowserStack (as it's less work), but running Selenium tests on PR is something we should have rather sooner than later to avoid PRs break our tests (as has already happened with #13602).

@nijel

This comment has been minimized.

Show comment
Hide comment
@nijel

nijel Aug 28, 2017

Member

Okay, I've taken the recommendation from travis-ci/travis-scheduler#81 (comment) and embedded encoded key in the .travis.yml in 16c7456.

Member

nijel commented Aug 28, 2017

Okay, I've taken the recommendation from travis-ci/travis-scheduler#81 (comment) and embedded encoded key in the .travis.yml in 16c7456.

@nijel nijel closed this Aug 28, 2017

nijel added a commit that referenced this issue Aug 28, 2017

Remove Selenium tests from the maintenance branch
They are broken anyway, so there is no point in wasting resources on
them.

See #13603

Signed-off-by: Michal Čihař <michal@cihar.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment