RFC 65: Switching to "Py3-first" for WPT tests on CI (#65)

Co-authored-by: Stephen McGruer <stephen.mcgruer@gmail.com>

rfcs/Py3-first.md

@@ -0,0 +1,52 @@
+# RFC 65: Switching to "Py3-first" for WPT tests on CI
+
+## Summary
+
+This is to propose switching test suite runs over to Python 3 on the WPT CI (Taskcluster and Azure). At the same time unit tests and infrastructure tests will still be running in Python 2 and Python 3. This is referred to as the “Py3-first" stage in the [Dropping Python 2 support plan](https://github.com/web-platform-tests/rfcs/issues/62).
+
+The minimum python 3 version supported is 3.6 with main focus on 3.8+. For the foreseeable future it is very likely to continue to run unit tests and infrastructure tests on both 3.6 and 3.8+ versions. The WPT test suite itself would run on 3.8+ in the CI.
+
+## Changes on upstream CI
+
+**1. Switch the test suite runs to use Python 3**
+* [Switch wpt run on Taskcluster to Python 3 only](https://github.com/web-platform-tests/wpt/pull/26252) (specifically, Python 3.8)
+
+* Use Python 3.8.x in Azure Pipelines for [Safari](https://github.com/web-platform-tests/wpt/pull/25044) and [Edge](https://github.com/web-platform-tests/wpt/pull/24952) runs.
+
+
+**2. Keep the current CI setup of unit tests and infrastructure tests running on both Py2 and Py3**
+
+The unit tests and infrastructure tests are already running on both Py2 and Py3. Outcomes of unit tests on Py2 and Py3 are not fully identical yet. We have created [a Unit Tests Tracksheet](https://docs.google.com/spreadsheets/d/1__dE5_ABazhDl6ONp76tbkgAimj14gD7QeaUYtfsi_w) to track the differences and give explanations.
+
+
+
+## Recommended actions for downstream (browser vendors)
+
+Switch wpt run to use Python 3, or at least run wptserve with Python 3 as soon as the upstream switches over and stabilizes. This minimizes the risk of exporting new Py2-only handlers to upstream. Waiting until after the upstream completes the switchover reduces churn in case we need to roll back to Py2 temporarily.
+
+## Risks
+
+To assess possible risks and provide evidence for  a move to Python 3 being safe, we have produced [a tracking spreadsheet](https://docs.google.com/spreadsheets/d/1cJcSW8PHe3A4m0dtBfjV4nrYLg67LzIznFbZIknDURc/edit#gid=1054524593). It compares the test results of wpt runs on Python2 and Python 3 for all the supported configurations including:
+* Linux+Chrome Dev
+* Linux+Firefox Nightly
+* Windows + Edge Dev
+* macOS + Safari Technology Preview
+
+Most different test results are caused by the fact that the test itself is flaky. For those flaky tests, links to flaky results in [wpt.fyi insights](https://wpt.fyi/insights) or [wpt.fyi recent runs ](https://wpt.fyi/runs) are given.
+
+Other cases fall into the following categories:
+* Different test names in `webdriver`. For some webdriver tests, test names are generated by pytest. If the argument passed to `@pytest.mark.parametrize` contains escape characters (e.g. `\n`,  `\t` etc.), it produces different test names in Python2 and Python3. [Issue #24888](https://github.com/web-platform-tests/wpt/issues/24888) was raised for this and discussed. The outcome is to ignore any differences because there are only a small number of changes caused and arguably the test names created in Python 3 are more meaningful.
+* Different test names in some `webaudio` tests. We noticed  `cors-check.https.html`produced different test names for both Chrome and Firefox, also `no-cors.https.html` for Firefox. This is due to the way the tests were written, which in some way is a bit strange while creating test names.
+* Python file handler local-imports issue, aka [issue #25678](https://github.com/web-platform-tests/wpt/issues/25678. An [RFC has been filed](https://github.com/web-platform-tests/rfcs/pull/68) along with [a fix](https://github.com/web-platform-tests/wpt/pull/26328) for this.
+* Newly introduced `.py` test files or codes. It is noted that before we move to the stage of "Py3-only", there is a risk that newly introduced `.py` codes might not be Python 3 compatible. We have fixed issues found in the tracking sheets but it is very likely that this kind of issues will appear again during the transition period.
+
+Looking into every different result of the WPT test suite between Python 2 and Python 3, we believe that we have each known Python 3 issue either fixed or at least explained.
+
+## Rollback and success criteria
+
+Admins (smcgruer & Hexcles) will be on call on #testing at irc.w3.org on the flag day. In case of unexpected massive failures (e.g. a large number of tests fail to run or produce invalid results), they will revert the following PRs (overriding any checks if needed):
+* [[Taskcluster] Switch wpt run on CI to Python 3 only](https://github.com/web-platform-tests/wpt/pull/26252)
+* [[Azure Pipelines] Use Python 3.8.x for Safari test-suite runs](https://github.com/web-platform-tests/wpt/pull/25044)
+* [[Azure Pipelines] Use Python 3.8.x for Edge test-suite runs](https://github.com/web-platform-tests/wpt/pull/24952)
+
+The switchover will be considered complete and successful when we have a full set of runs from all supported configurations.