Join GitHub today
Run Selenium tests on pull requests #13603
As we have Selenium tests more or less working right now (the exception being f36ed72#commitcomment-23788220), I think it's necessary to run them on pull requests to ensure they don't break anything.
Running the tests against BrowserStack requires authentication token, which has to be stored securely (otherwise anybody would be able to use our account there). Because of this, the token is not exposed in the pull requests environment. This is pretty much well known problem, see travis-ci/travis-ci#1946
As a solution the Travis CI team has come with JWT based authentication which generates unique token for every build based on main token. This way the builds can access the service without exposing the main token. Unfortunately BrowserStack currently does not support this. I've asked their support today if there are any plans for supporting this.
If there are no plans on the BrowserStack side, we might consider switch to SauceLabs, where this is already implemented (and seems to work fine, I'm using it on https://github.com/WeblateOrg/weblate/).
I don't have an opinion about how to accomplish this; your proposed switch to SauceLabs is fine by me if BrowserStack has no plans to implement it. I don't have any better idea or have any disagreement with what you wrote, so go ahead and do what you think is best.