Execute affected tests both with and without a PR's changes

[lukebjerring]

This will allow a more apples-to-apples diff than comparing to a recent full run of the same product (see #707).

[lukebjerring]

This (executing both with and without) is a better solution to a problem that was mentioned in #745 - namely, as an interim we are comparing a PR's affected tests againts "the latest master run".
This is done with a link that points to chrome[master], for example - a link that will change over time (as new master runs arrive).

cc @foolip

[foolip]

@lukebjerring, can you send a PR for the Taskcluster changes? Here's what I'd planned to do for Azure Pipelines:

• In a "decision task" make note of what HEAD^1 and HEAD^2 of the synthetic merge commit (refs/pull/1234/merge) are, call them SAVED_MASTER and SAVED_PR_HEAD. In particular the first parent is a commit on master, and master may change.
• In the "with changes" task, check out SAVED_MASTER and merge in SAVED_PR_HEAD. Affected tests are ./wpt tests-affected $SAVED_MASTER.
• In the "without changes" task check out SAVED_MASTER and merge in SAVED_PR_HEAD, call that commit TMP_MERGE. Then check out SAVED_MASTER and test that. Affected tests are ./wpt tests-affected $TMP_MERGE.

This is tragically convoluted, but I don't see how else to ensure differences between the two states of the trees were tested are only the changes from the PRs, and not also some random contribution of changes to master that happened as the tasks were queuing.

@jgraham may have thouhts.

[foolip]

I sent web-platform-tests/wpt#14093 for a thing I found confusing about the affected tests logic.

I also just conducted an experiment on foolip/wpt to fetch git fetch https://github.com/foolip/wpt.git refs/pull/12/merge && git rev-parse FETCH_HEAD, push commits to master, and fetch it again. FETCH_HEAD did change just seconds after. So there's nothing that GitHub does to provide stability here, and honestly I can't imagine what that could be.

[Hexcles]

I also just conducted an experiment on foolip/wpt to fetch git fetch https://github.com/foolip/wpt.git refs/pull/12/merge && git rev-parse FETCH_HEAD, push commits to master, and fetch it again. FETCH_HEAD did change just seconds after. So there's nothing that GitHub does to provide stability here, and honestly I can't imagine what that could be.

This is a well known but undocumented feature: https://discourse.drone.io/t/github-claims-that-merge-refs-are-undocumented-feature/1100

It's so widely used (especially by CI services including Travis) that I don't think it will go away. That said, we shouldn't rely too heavily on the timeliness (usually, the merge ref is updated shortly after the PR is updated). Another caveat is that when there is a merge conflict, the merge ref is invalid.

[Hexcles]

I just sent a PR to modify the Taskcluster configuration of wpt to run affected tests without patch: web-platform-tests/wpt#14382

(Note that we don't really have a "decision task" yet. Each task executes independently, so various race conditions may happen, especially if a PR is being updated quickly, e.g. different merge refs in the Firefox & Chrome tasks. But I think it's good enough for now.)

In addition, we need to make some changes to wpt.fyi to accommodate this. #865 is the first step: collecting the "without patch" results. Then we can compare them against the "with patch" results.

[foolip]

It sounds like the undocumented feature is simply refs/pull/1234/merge, which is what Azure Pipelines depends on and presumably Taskcluster as well.