This repository has been archived by the owner. It is now read-only.

Add support for Firefox UI Tests #576

Closed
whimboo opened this Issue Mar 24, 2015 · 19 comments

Comments

Projects
None yet
1 participant
@whimboo
Copy link
Contributor

whimboo commented Mar 24, 2015

We want to get our new Marionette driven Firefox UI Test to run in Mozmill CI (renaming of the system can happen later). Given that lots of tests have to be ported we will get started with our remote and update tests first, and that only for mozilla-central jobs.

What we need here is:

  1. A new job which can download the pre-configured environments from mozqa.com - similar to the mozmill one. I will check today what's actually necessary here to get such an environment up on mozqa.com.
  2. Maybe we need another global configuration setting to let us specify the version of the firefox-ui-harness. Thing is that right now we are bundled with the tests, so it would be better to clone the whole repo and keep it up to date instead until we have real version management.
  3. A job to keep the local clone of https://github.com/mozilla/firefox-ui-tests up-2-date on the master machine. This will be similar to the get_mozmill-tests job.
  4. The jobs mozilla-central_remote and mozilla-central_update have to be updated to make use of the new tests.

CC'ing a bunch of people who might be involved here: @bobsilverberg, @sydvicious, @stephendonner, @jonallengriffin, @AutomatedTester

@whimboo whimboo added the must-have label Mar 24, 2015

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Mar 24, 2015

As discussed with @AutomatedTester on IRC we don't want to get in something in a rush. We should think about the proper solution here, and I will do it once I'm back. Meanwhile we will get our tests running via Travis CI and AppVeyor for the required CI support. Given active development we can ensure to regularly get the builds tested and to see the stability of the tests.

whimboo added a commit to whimboo/mozmill-ci that referenced this issue May 12, 2015

@whimboo whimboo self-assigned this May 12, 2015

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented May 12, 2015

Point 2) will not have to be done. As decided we will keep the harness close to the tests. Therefore we have different branches in the firefox-ui-tests repository now, each of those matching a specific Firefox version. So we simply clone and pull the remote github hosted repository.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented May 20, 2015

With PR #581 we have flipped all mozilla-central related jobs to make use of Marionette. I will publish those changes to staging now. If all is fine we can get it on production by tomorrow.

whimboo added a commit that referenced this issue May 20, 2015

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented May 20, 2015

My last PR missed the correct selection of the git tool. Looks like it got missed while operating with the XML files and the Jenkins web config at the same time. I pushed a follow-up bustage fix to get this fixed:

ad50b88

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented May 20, 2015

So in staging we have problems with jgit (java-like implementation of a git client). It is not able to clone the firefox-ui-tests repository only on Windows 8.1 systems. I'm not sure why yet, because it works with Windows 8 at home. Maybe there are configuration issues on the boxes related to proxies or something else. I will have to investigate that.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented May 20, 2015

There seem to be indeed problems with jgit and proxies. I cannot get it to work at all in the SCL3 environment. As result we will now make use of native git clients on all platforms. I was hoping that this would not be necessary, but looks lke we have to do it.

whimboo added a commit that referenced this issue May 21, 2015

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented May 22, 2015

So I observed tests with Marionette in staging the last two days and everything is working fine! I have to finish to update all nodes in production before we can enable it there. So hopefully by Tuesday next week at latest.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented May 27, 2015

I pushed the changes to production. That means we run Marionette tests officially now. I will observe the results the next few days, and if all is working well we can continue with Aurora.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 4, 2015

PR #585 for switching mozilla-aurora to Firefox UI Tests has been merged and deployed to staging. So far everything looks fine. No obvious failures! Hurray!

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 5, 2015

Aurora now also runs Marionette and for todays en-US builds very well! Lets see which failures come up over the weekend, which I will have to tackle beginning of next week.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 8, 2015

A couple of our VMs in production seem to have issues with loading remote content and especially focus related misbehavior. I'm going to request a re-deployment of all those VMs. We should be green before starting to run our tests against beta builds. For VMs in staging all seems to be fine.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 10, 2015

PR #590 for beta builds got merged to master. I will push it to staging in a moment and will observe it over the next days. If all is fine we can push it to production later this week.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 11, 2015

We are currently experiencing a lot of broken tests for localized builds on beta. Mostly due to duplicated command keys, which can really break the application. But given the amount of locales we cannot cope with them all right now, and have to wait until the treeherder integration is done.

I talked with @Pike and @flodolo about this and our short term goal is to only keep the most important locales tested on beta. That will assume that localization works. The general testing for all locales has to be moved to Aurora, which is a separate issue.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 11, 2015

I will use the following list (#270 (comment)) as base for keeping the important locales enabled on beta and release. We should make sure to have at least a unicode and rtl one included.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 22, 2015

Remaining PRs for release and project branch based jobs are up. I will not convert the ondemand jobs yet, but will have a follow-up issue which cleans up remaining unnecessary jobs.

whimboo added a commit that referenced this issue Jun 23, 2015

whimboo added a commit to whimboo/mozmill-ci that referenced this issue Jun 23, 2015

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 23, 2015

Firefox UI tests are active now for release and esr38 on staging. I did a complete run and everything is green. I will wait for later today and the landing of the other open PRs before pushing all together to the production branch.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 24, 2015

PR #597 has been landed too. I will push it to staging and observe it for the next day. If all is fine we can push everything to production.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 25, 2015

Everything fine for the 39.0 and 38.0esr release. So I will now push the changes to production. Also because production is still affected by the VERSION parameter name change, which caused all tests to fail. :( I will retrigger them all once the new code is live.

@whimboo

This comment has been minimized.

Copy link
Contributor

whimboo commented Jun 25, 2015

Everything is on production now! \o/ I will leave esr31 and ondemand tests alone for now, which should not block us from closing this issue. Instead separate issues will be filed to get them removed or updated.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.