Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run master commits on Taskcluster #9226

Merged
merged 6 commits into from Apr 25, 2018

Conversation

Projects
None yet
7 participants
@jgraham
Copy link
Contributor

jgraham commented Jan 29, 2018

This PR actually has two parts:

  1. A docker image for running tests. I assume this will be relatively uncontroversial, although there are certainly ways that we could improve the details.
  2. The required configuration to run each push on master on TaskCluster. This won't do anything unless the relevant GitHub integration is enabled. If it is, however, it will typically provide the full results of the testsuite within about an hour of a merge to master. You can see an example run from my fork. Each job uploads an artifact containing the results from that chunk in wptreport.json format; these could be combined to give the full set of results for each push.

This change is Reviewable

@wpt-pr-bot wpt-pr-bot added the infra label Jan 29, 2018

@wpt-pr-bot wpt-pr-bot requested a review from gsnedders Jan 29, 2018

@w3c-bots

This comment has been minimized.

Copy link

w3c-bots commented Jan 29, 2018

Build PASSED

Started: 2018-03-16 17:35:33
Finished: 2018-03-16 17:53:33

View more information about this build on:

@jgraham jgraham force-pushed the jgraham/taskcluster branch from 5bb339f to 696c6c3 Jan 30, 2018

@jgraham jgraham requested a review from mjzffr Feb 5, 2018

@jgraham jgraham force-pushed the jgraham/taskcluster branch from 696c6c3 to ab20739 Feb 6, 2018

@foolip

This comment has been minimized.

Copy link
Contributor

foolip commented Apr 13, 2018

A docker image for running tests

Interesting! @jugglinmike, very relevant to web-platform-tests/results-collection#535 (comment)

will typically provide the full results of the testsuite within about an hour of a merge to master

Wow. That'd achieve our wpt.fyi results are available within 1 hour for at least Firefox, and way sooner than I thought was possible.

@lukebjerring @boaz @mariestaver @jugglinmike, this could change our roadmap drastically, so let's discuss at the earliest opportunity.

@foolip

This comment has been minimized.

Copy link
Contributor

foolip commented Apr 13, 2018

Each job uploads an artifact containing the results from that chunk in wptreport.json format; these could be combined to give the full set of results for each push.

@Hexcles, can you take a look at the bits around this, to see if it'll be a trivial integration with a wpt.fyi results POST endpoint? Seems like the token management and such would have to be very similar to what we've speculated about for PRs on Travis.

\ &&\n git config advice.detachedHead false &&\n git\
\ checkout {{event.head.sha}} &&\n ./tools/ci/ci_taskcluster.sh\
\ chrome reftest 1 6"]
image: harjgam/web-platform-tests:0.5

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

Should we have a "web-platform-tests" organization on Docker Hub? Is that a thing?

This comment has been minimized.

Copy link
@jgraham

jgraham Apr 13, 2018

Author Contributor

It is and we can; I just created an org with the namespace webplatformtests; neither wpt or web-platform-tests worked, but I can't tell if they were taken or if there are some character/length restrictions.

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

Where does the code live for creating the docker images?

@@ -0,0 +1,687 @@
allowPullRequests: collaborators

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

This configuration file is huge and fairly repetitive. I assume it has to be checked in to the repo like .travis.yml, but could parts of it be generated?

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

Ha, there is a script :) Suggestion is then to add a comment at the top pointing to the script.

if [ $1 == "firefox" ]; then
./wpt run firefox --log-tbpl=- --log-tbpl-level=debug --log-wptreport=../artifacts/wpt_report.json --this-chunk=$3 --total-chunks=$4 --test-type=$2 -y --install-browser --no-pause --no-restart-on-unexpected
elif [ $1 == "chrome" ]; then
./wpt run chrome --log-tbpl=- --log-tbpl-level=debug --log-wptreport=../artifacts/wpt_report.json --this-chunk=$3 --total-chunks=$4 --test-type=$2 -y --no-pause --no-restart-on-unexpected

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

Which version of chrome will this be?

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

Based on https://tools.taskcluster.net/groups/bLpNdT49RGCR7nEhELQE1A I'm guessing chrome dev, but I can't tell why.

This comment has been minimized.

Copy link
@jgraham

jgraham Apr 13, 2018

Author Contributor

It gets installed using apt in tools/docker/start.sh, pretty much like we do for travis. This probably isn't the optimal way to do this; the current setup is more of a proof of concept than anything, so I'm sure there are plenty of improvements we can make.

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

Ah, well as long as it's not just accidentally chrome dev, that's just fine!

cd web-platform-tests
git pull --depth=1

# Install Chome unstable

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

In #8838 I tried to banish "unstable" everywhere except package name, can you call it "Chrome dev"?

@@ -0,0 +1,27 @@
#!/bin/bash

sudo sh -c 'echo "

This comment has been minimized.

Copy link
@foolip

foolip Apr 13, 2018

Contributor

@gsnedders wrote something to generate this output, can you use that here?

This comment has been minimized.

Copy link
@gsnedders

gsnedders Apr 17, 2018

Contributor

./wpt make-hosts-file does this


RUN apt-get -qqy install \
firefox \
chromium-browser \

This comment has been minimized.

Copy link
@Hexcles

Hexcles Apr 17, 2018

Member

IIUC, the extra dependencies below are dependencies of Chrome (that you happen to know) but not of chromium-browser.

IMHO, since Chrome is already distributed in a deb package, we can perhaps skip this (i.e. only install firefox). See my next comment in start.sh for dependency problems.

if ! sudo dpkg --install $deb_archive; then
sudo apt-get -y install --fix-broken
sudo dpkg --install $deb_archive
fi

This comment has been minimized.

Copy link
@Hexcles

Hexcles Apr 17, 2018

Member

I think this is sufficient for solving Chrome's dependencies, so there's no need for installing chromium-browser in the Docker image.

Besides, you might want to try gdebi, which installs a local deb package along with its dependencies (from apt). I think it's available by default on Ubuntu.

This comment has been minimized.

Copy link
@gsnedders

gsnedders Apr 17, 2018

Contributor

@Hexcles This does the same, FWIW. (Also, BTW, you're not on IRC.)

This comment has been minimized.

Copy link
@Hexcles

Hexcles Apr 23, 2018

Member

Yeah I know it's the same, but it'd save a couple lines :P

@foolip

This comment has been minimized.

Copy link
Contributor

foolip commented Apr 17, 2018

I'll step away from the detailed code review now as @Hexcles will do that, makes the most sense since it needs to be integrated together with the results upload endpoint, somehow.

@foolip

This comment has been minimized.

Copy link
Contributor

foolip commented Apr 17, 2018

I have a question about browser version management. If each shard individually downloads https://dl.google.com/linux/direct/google-chrome-unstable_current_amd64.deb then once every few days it's possible they will not get the same version. At minimum it would be good if the browser version is recorded in the wptreport JSON so that we can detect this. Even better would be if the browser version is somehow managed together with the container.

The very simplest would be to have the browsers installed in the container itself, which would mean updating the container daily, but that can be automated pretty easily. https://github.com/whatwg/html-build/blob/master/ci-deploy/outside-container.sh does this.

How we then ensure that the same Docker image is used for each Taskcluster shard I don't know. But it'd be very good if it's also recorded in the wptreport somehow. (A bit backwards, I know, but a variable could be passed in.)

@Hexcles

This comment has been minimized.

Copy link
Member

Hexcles commented Apr 23, 2018

Sorry I was busy (travelling) last week and not available on IRC.

I've glanced through the PR. Apart from the nit comment I made earlier, I have one high-level question: is there no way to configure the number of chunks other than using a script to dump each chunk as a task? I'm a bit surprised if Taskcluster doesn't some sharding this natively.

@jgraham

This comment has been minimized.

Copy link
Contributor Author

jgraham commented Apr 24, 2018

So. Tthe good way to support that is to write a decision task that schedules all the other tasks dynamically. We should do that, but it's not totally necessary for the first iteration.

It's possible that there's some syntax for parameterising the yaml file; I can look for documentation. I'm not very pleased with the current setup either, but it seemed like the fastest path to something that worked.

@jgraham

This comment has been minimized.

Copy link
Contributor Author

jgraham commented Apr 24, 2018

https://bugzilla.mozilla.org/show_bug.cgi?id=1335915 seems to be the bug for enabling the fancy templating stuff in .taskcluster.yml; since that's still open I assume it's not possible yet, but we should look to switch to it once that bug is fixed.

jgraham added some commits Jan 29, 2018

Add a docker image designed for running tests.
This is a docker image based on the Ubuntu1604 base, with the runtime
deps of Firefox and Chrome installed. There is also a shallow clone of
wpt to act as a placeholder. The start.sh script will update this
clone and start Xvfb, so it's possible to run tests.
Add configuration for running every push to master in TaskCluster.
TaskCluster is Mozilla's CI system, which also provides a GitHub
integration. With this configuration enabling the TaskCluster
integration on GitHub will cause each pust to master to create a
TaskCluster run consisting of the full testsuite in latest Firefox
Nightly and latest Chrome dev. The run will be chunked into 12 pieces
for testharness.js tests and 6 for reftests, plus one for wdspec
tests. With this chunking the end to end time for completing the run
is around an hour (the task timeout is set to 1.5 hours).

The resulting tasks upload wptreport.json files that can be used to
reconstruct the results of the full run. There are also tbpl log files
that could be used with e.g. Mozilla's reftest analyzer to inpect
reftest failures.
Don't return failure for wpt run if not all tests pass.
This isn't very useful unless there is expectation data avaiable, which is
not the common case
Rename wpt generate-tasks to wpt make-tasks
Also add a comment to indicate that the tasks file is generated.

@jgraham jgraham force-pushed the jgraham/taskcluster branch from 4f87b52 to e7bae7a Apr 25, 2018

@Hexcles
Copy link
Member

Hexcles left a comment

LGTM except for a nit.

(I agree with using a script to generate .taskcluster.yml for now. We can explore simplifications later.)

# to know that chrome requires

RUN apt-get -qqy install \
firefox \

This comment has been minimized.

Copy link
@Hexcles

Hexcles Apr 25, 2018

Member

Sorry I meant everything below could be removed, i.e. just apt-get -qqy install firefox should be enough.

The other packages below are "some extra deps we happen to know that chrome requires" according to the comment above. And since gdebi chrome.deb will take care of them, we don't need to guess and install these dependencies any more.

The comment above should also be updated to reflect that we are only installing firefox now.

This comment has been minimized.

Copy link
@jgraham

jgraham Apr 25, 2018

Author Contributor

Installing more stuff at build time does have the advantage that we don't do so much work at runtime, so generally it makes more sense to move stuff into the docker build rather than the other way around.

In this case we agreed on irc that it doesn't make much difference either way and there are plenty of performance optimisations and potential cleanup left on the table, so it's OK to just merging what we have now.

deb_archive=google-chrome-unstable_current_amd64.deb
wget https://dl.google.com/linux/direct/$deb_archive

sudo gdebi -n $deb_archive

This comment has been minimized.

Copy link
@jugglinmike

jugglinmike Apr 25, 2018

Contributor

Note: Installing Google Chrome will add the Google repository so your system
will automatically keep Google Chrome up to date. If you don’t want Google's
repository, do sudo touch /etc/default/google-chrome before installing the
package.

Source: https://www.google.com/intl/en/chrome/browser/

Is this relevant given the expected lifespan of the container?

This comment has been minimized.

Copy link
@jgraham

jgraham Apr 25, 2018

Author Contributor

No; any changes only persist for the lifetime of the container, which is just for a single test run.

@jgraham jgraham merged commit 875c65e into master Apr 25, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@foolip foolip deleted the jgraham/taskcluster branch Apr 25, 2018

@foolip

This comment has been minimized.

Copy link
Contributor

foolip commented Apr 25, 2018

Awesome! Looks like the very first run on master is this one, still ongoing:
https://tools.taskcluster.net/groups/Bv2TZUgeSyGhXblM8UU0aA

@foolip

This comment has been minimized.

Copy link
Contributor

foolip commented Apr 25, 2018

And here are actual results from shards from a shard that has finished:
https://taskcluster-artifacts.net/N59MId9GQoir2KBBxEH_FA/0/public/results/wpt_report.json

This will be great :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.