Convert ondemand update tests from Mozmill to Marionette

[whimboo]

As we have noticed today our ondemand update tests are busted for Firefox 46.0b1 due to JS engine changes. That means we on longer can make use of them. See details in bug 1255566.

A transition to the new framework has to be made now. When I do that, I will keep a copy of the current Jenkins jobs, so that softvision can still run update tests for Firefox releases older than 38.0 like esr31.

[KaiRo-at]

I personally think we can stop testing builds older than 38.0 right now, you don't need to do extra work for that.

[whimboo]

Softvision is still running update tests for esr31.* to esr38.*. And this will still be the case for the next two major releases. If you really think that I can remove those jobs I will do it.

[KaiRo-at]

Given how long it's been, I think they should stop running those and only care about ESR 38.0 and higher. Don't do extra work to still support 31, esp. as those show intermittent errors all the time anyhow.

[whimboo]

I would only have to rename the jobs, thats all. I think that I will keep them for now. They can be removed once esr31 is deprecated.

[whimboo]

Current status is that I have kept the old mozmill-based jobs and added a suffix_mozmill to them. For marionette based update tests I can already run the ondemand_update job manually. Here a result.

Tomorrow I will have to rewrite the ondemand_trigger job which will actually create those ondemand_update jobs.

[FlorinMezei]

So it sounds like we can still run updates from ESR 31... we'll still do that then until we're done with ESR 38 (which should be after the next cycle).

If it helps not supporting them, note that I have no problem dropping them for the next ESR 38.

[whimboo]

Yes, when we remove any esr31 trace from Mozmill-CI I will also delete the ondemand_update tests for Mozmill. Until then you can still use the trigger_ondemand_mozmill job.

[whimboo]

Given that we report to Treeherder I would like to not have the problem with the wrongly specified target-build-id anymore. So I would like to see a target-version entry in the config file. The trigger script will then automatically determine the build id and revision of the build per platform.

[whimboo]

Trigger script is working. Here the output: https://gist.github.com/whimboo/79ae3e1b1d7ffd641773

[whimboo]

I already pushed the PR #766 to the master branch and will make it available on staging for a complete test. If all goes well I will publish to production afterward.

[whimboo]

This is done.