diff --git a/rfcs/Py3-first.md b/rfcs/Py3-first.md new file mode 100644 index 0000000..562b399 --- /dev/null +++ b/rfcs/Py3-first.md @@ -0,0 +1,52 @@ +# RFC 65: Switching to "Py3-first" for WPT tests on CI + +## Summary + +This is to propose switching test suite runs over to Python 3 on the WPT CI (Taskcluster and Azure). At the same time unit tests and infrastructure tests will still be running in Python 2 and Python 3. This is referred to as the “Py3-first" stage in the [Dropping Python 2 support plan](https://github.com/web-platform-tests/rfcs/issues/62). + +The minimum python 3 version supported is 3.6 with main focus on 3.8+. For the foreseeable future it is very likely to continue to run unit tests and infrastructure tests on both 3.6 and 3.8+ versions. The WPT test suite itself would run on 3.8+ in the CI. + +## Changes on upstream CI + +**1. Switch the test suite runs to use Python 3** +* [Switch wpt run on Taskcluster to Python 3 only](https://github.com/web-platform-tests/wpt/pull/26252) (specifically, Python 3.8) + +* Use Python 3.8.x in Azure Pipelines for [Safari](https://github.com/web-platform-tests/wpt/pull/25044) and [Edge](https://github.com/web-platform-tests/wpt/pull/24952) runs. + + +**2. Keep the current CI setup of unit tests and infrastructure tests running on both Py2 and Py3** + +The unit tests and infrastructure tests are already running on both Py2 and Py3. Outcomes of unit tests on Py2 and Py3 are not fully identical yet. We have created [a Unit Tests Tracksheet](https://docs.google.com/spreadsheets/d/1__dE5_ABazhDl6ONp76tbkgAimj14gD7QeaUYtfsi_w) to track the differences and give explanations. + + + +## Recommended actions for downstream (browser vendors) + +Switch wpt run to use Python 3, or at least run wptserve with Python 3 as soon as the upstream switches over and stabilizes. This minimizes the risk of exporting new Py2-only handlers to upstream. Waiting until after the upstream completes the switchover reduces churn in case we need to roll back to Py2 temporarily. + +## Risks + +To assess possible risks and provide evidence for a move to Python 3 being safe, we have produced [a tracking spreadsheet](https://docs.google.com/spreadsheets/d/1cJcSW8PHe3A4m0dtBfjV4nrYLg67LzIznFbZIknDURc/edit#gid=1054524593). It compares the test results of wpt runs on Python2 and Python 3 for all the supported configurations including: +* Linux+Chrome Dev +* Linux+Firefox Nightly +* Windows + Edge Dev +* macOS + Safari Technology Preview + +Most different test results are caused by the fact that the test itself is flaky. For those flaky tests, links to flaky results in [wpt.fyi insights](https://wpt.fyi/insights) or [wpt.fyi recent runs ](https://wpt.fyi/runs) are given. + +Other cases fall into the following categories: +* Different test names in `webdriver`. For some webdriver tests, test names are generated by pytest. If the argument passed to `@pytest.mark.parametrize` contains escape characters (e.g. `\n`, `\t` etc.), it produces different test names in Python2 and Python3. [Issue #24888](https://github.com/web-platform-tests/wpt/issues/24888) was raised for this and discussed. The outcome is to ignore any differences because there are only a small number of changes caused and arguably the test names created in Python 3 are more meaningful. +* Different test names in some `webaudio` tests. We noticed `cors-check.https.html`produced different test names for both Chrome and Firefox, also `no-cors.https.html` for Firefox. This is due to the way the tests were written, which in some way is a bit strange while creating test names. +* Python file handler local-imports issue, aka [issue #25678](https://github.com/web-platform-tests/wpt/issues/25678. An [RFC has been filed](https://github.com/web-platform-tests/rfcs/pull/68) along with [a fix](https://github.com/web-platform-tests/wpt/pull/26328) for this. +* Newly introduced `.py` test files or codes. It is noted that before we move to the stage of "Py3-only", there is a risk that newly introduced `.py` codes might not be Python 3 compatible. We have fixed issues found in the tracking sheets but it is very likely that this kind of issues will appear again during the transition period. + +Looking into every different result of the WPT test suite between Python 2 and Python 3, we believe that we have each known Python 3 issue either fixed or at least explained. + +## Rollback and success criteria + +Admins (smcgruer & Hexcles) will be on call on #testing at irc.w3.org on the flag day. In case of unexpected massive failures (e.g. a large number of tests fail to run or produce invalid results), they will revert the following PRs (overriding any checks if needed): +* [[Taskcluster] Switch wpt run on CI to Python 3 only](https://github.com/web-platform-tests/wpt/pull/26252) +* [[Azure Pipelines] Use Python 3.8.x for Safari test-suite runs](https://github.com/web-platform-tests/wpt/pull/25044) +* [[Azure Pipelines] Use Python 3.8.x for Edge test-suite runs](https://github.com/web-platform-tests/wpt/pull/24952) + +The switchover will be considered complete and successful when we have a full set of runs from all supported configurations.