Accessibility test Kitchen Sink with Playwright (#1260)

* Add core-js so dev build works Install pytest-playwright and (npm) axe-core Write test that scans admonitions with Axe How to run test: nox -s compile nox -s docs python -m http.server -d docs/_build/html/ nox -s test -- -k accessibility * add more pages, test server * add documentation * add prerequisites to nox test * update package-lock.json via stb * update package-lock.json via nox -s compile * streamline, update docs * typo * sync lockfile (remove core-js) * Update tests/utils.py Co-authored-by: Daniel McCloy <dan@mccloy.info> * Update tests/README.md Co-authored-by: Tania Allard <taniar.allard@gmail.com> * Update docs/community/topics/accessibility.md Co-authored-by: Tania Allard <taniar.allard@gmail.com> * Update tests/README.md Co-authored-by: Tania Allard <taniar.allard@gmail.com> * Update tests/utils.py Co-authored-by: Tania Allard <taniar.allard@gmail.com> * Update tests/utils.py Co-authored-by: Tania Allard <taniar.allard@gmail.com> * - nox session a11y - add pytest-base-url plugin - push more test setup into page fixture - add license header to pretty_axe_results - move pretty_axe_results and check_warnings to utils/ * update docs * uncomment commented-out tests * regenerate package-lock.json via: rm package-lock.json rm -r node_modules stb npm install --include=dev * oops finish updating tests * do not include a11y tests in main test session * remove redundant decorator * Update tests/README.md Co-authored-by: Daniel McCloy <dan@mccloy.info> * Update tests/README.md Co-authored-by: Daniel McCloy <dan@mccloy.info> * Update tests/README.md Co-authored-by: Tania Allard <taniar.allard@gmail.com> * Remove base-url Pytest plugin * Update tests/utils/pretty_axe_results.py Co-authored-by: Tania Allard <taniar.allard@gmail.com> * Apply suggestions from code review Co-authored-by: Tania Allard <taniar.allard@gmail.com> * fix pretty_axe_results * Generate each test run from parameters * remove unused marker url_path * undo multiline * update doc string * Add CI check for --with-deps suggested by Tony * oops, rm extra playwright install line * update comment * update docstring * testing the pytest-axe mark * run prettier * Revert "testing the pytest-axe mark" This reverts commit f7b92ea. * Fix D104 * skip a11y tests on CI runs * run a11y tests on one CI worker only * ✅ Register a11y mark and separate dependencies * 👷 Update CI - delegate playwright install and cache * 📝 Update test docs * ✅ Ensure a11y tests are skipped unles called * 👷 Ensure CI installs all deps * ⏪ Revert to installing playwright * 👷 Forgot to re-add a11y deps * ✏️ Fix typo * 🔧 Fix path - seems to have been messed up when moving things * Update tests/test_a11y.py --------- Co-authored-by: Daniel McCloy <dan@mccloy.info> Co-authored-by: Tania Allard <taniar.allard@gmail.com>
pydata · Apr 14, 2023 · cbebfc9 · cbebfc9
1 parent d939eda
commit cbebfc9
Show file tree

Hide file tree

Showing 14 changed files with 327 additions and 63 deletions.
diff --git a/.github/workflows/prerelease.yml b/.github/workflows/prerelease.yml
@@ -36,7 +36,7 @@ jobs:
 
       - name: Check that there are no unexpected Sphinx warnings
         if: matrix.python-version == '3.10'
-        run: python tests/check_warnings.py
+        run: python tests/utils/check_warnings.py
 
       - name: Run the tests
         run: |

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -56,6 +56,9 @@ jobs:
           # windows test
           - os: windows-latest
             python-version: "3.11"
+    # needed to cache the browsers for the accessibility tests
+    env:
+      PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/pw-browsers
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v3
@@ -90,14 +93,30 @@ jobs:
           pip install nox
           nox -s compile
       - name: Run tests
-        run: pytest --color=yes --cov pydata_sphinx_theme --cov-branch --cov-report xml:cov.xml --cov-report= --cov-fail-under ${{ env.COVERAGE_THRESHOLD }}
+        run: pytest -m "not a11y" --color=yes --cov pydata_sphinx_theme --cov-branch --cov-report xml:cov.xml --cov-report= --cov-fail-under ${{ env.COVERAGE_THRESHOLD }}
       - name: Upload to Codecov
         if: matrix.python-version == '3.11' && matrix.os == 'ubuntu-latest'
         uses: codecov/codecov-action@v3.1.2
         with:
           files: cov.xml
           fail_ci_if_error: true
 
+      # note I am setting this on top of the Python cache as I could not find
+      # how to set the hash key on the python one
+      - name: Set up browser cache (for accessibility tests)
+        uses: actions/cache@v3
+        with:
+          path: |
+            ${{ github.workspace }}/pw-browsers
+          key: ${{ runner.os }}-pw-${{ hashFiles('pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-pw-
+
+      - name: Run accessibility tests with playwright
+        if: matrix.python-version == '3.11' && matrix.os == 'ubuntu-latest'
+        run: |
+          nox -s a11y
+
   # Build our site on the 3 major OSes and check for Sphinx warnings
   build-site:
     needs: [lint]
@@ -124,7 +143,7 @@ jobs:
       - name: Build docs
         run: sphinx-build -b html docs/ docs/_build/html --keep-going -w warnings.txt
       - name: Check for unexpected Sphinx warnings
-        run: python tests/check_warnings.py
+        run: python tests/utils/check_warnings.py
 
   # Run local Lighthouse audit against built site
   audit:

diff --git a/docs/community/setup.md b/docs/community/setup.md
@@ -22,10 +22,10 @@ The sections below cover the steps to do this in more detail.
 
 ## Clone the repository
 
-First off you'll need your own copy of the `pydata-sphinx-theme` codebase.
+First off you'll need your copy of the `pydata-sphinx-theme` codebase.
 You can clone it for local development like so:
 
-1. **Fork the repository** so you have your own copy on GitHub.
+1. **Fork the repository**, so you have your own copy on GitHub.
    See [the GitHub forking guide](https://docs.github.com/en/get-started/quickstart/fork-a-repo) for more information.
 2. **Clone the repository locally** so that you have a local copy to work from:
 
@@ -36,14 +36,14 @@ You can clone it for local development like so:
 
 ## Install your tools
 
-Building a Sphinx site uses a combination of Python and Jinja to manage HTML, SCSS, and Javascript.
+Building a Sphinx site uses a combination of Python and Jinja to manage HTML, SCSS, and JavaScript.
 To simplify this process, we use a few helper tools:
 
-- [The Sphinx Theme Builder](https://sphinx-theme-builder.readthedocs.io/en/latest/) to automatically perform compilation of web assets.
+- [The Sphinx Theme Builder](https://sphinx-theme-builder.readthedocs.io/en/latest/) compiles web assets in an automated way.
 - [pre-commit](https://pre-commit.com/) for automatically enforcing code standards and quality checks before commits.
-- [nox](https://nox.thea.codes/), for automating common development tasks.
+- [nox](https://nox.thea.codes/) for automating common development tasks.
 
-In particular, `nox` can be used to automatically create isolated local development environments with all of the correct packages installed to work on the theme.
+In particular, `nox` can be used to automatically create isolated local development environments with all the correct packages installed to work on the theme.
 The rest of this guide focuses on using `nox` to start with a basic environment.
 
 ```{seealso}
@@ -58,9 +58,9 @@ To start, install `nox`:
 $ pip install nox
 ```
 
-You can call `nox` from the command line in order to perform common actions that are needed in building the theme.
+You can call `nox` from the command line to perform common actions that are needed in building the theme.
 `nox` operates with isolated environments, so each action has its own packages installed in a local directory (`.nox`).
-For common development actions, you'll simply need to use `nox` and won't need to set up any other packages.
+For common development actions, you'll only need to use `nox` and won't need to set up any other packages.
 
 ### Setup `pre-commit`
 
@@ -116,7 +116,7 @@ Now that you've built the documentation, edit one of the source files to see how
 1. **Make an edit to a page**. For example, add a word or fix a typo on any page.
 2. **Rebuild the documentation** with `nox -s docs`
 
-It should go much faster this time, because `nox` is re-using the old environment, and because Sphinx has cached the pages that you didn't change.
+It should go much faster this time because `nox` is re-using the previously created environment, and because Sphinx has cached the pages that you didn't change.
 
 ## Compile the CSS/JS assets
 
@@ -138,15 +138,15 @@ The `sphinx-theme-builder` will bundle these assets automatically when we make a
 
 ## Run a development server
 
-You can combine the above two actions and run a development server so that changes to `src/` are automatically bundled with the package, and the documentation is immediately reloaded in a live preview window.
+You can combine the above two actions (build the docs and compile JS/CSS assets) and run a development server so that changes to `src/` are automatically bundled with the package, and the documentation is immediately reloaded in a live preview window.
 
 To run the development server with `nox`, run the following command:
 
 ```console
 $ nox -s docs-live
 ```
 
-When working on the theme, saving changes to any of these directories:
+When working on the theme, making changes to any of these directories:
 
 - `src/js/index.js`
 - `src/scss/index.scss`
@@ -161,19 +161,33 @@ will cause the development server to do the following:
 
 ## Run the tests
 
-This theme uses `pytest` for its testing, with a lightweight fixture defined
-in the `test_build.py` script that makes it easy to run a Sphinx build using
-this theme and inspect the results.
+This theme uses `pytest` for its testing. There is a lightweight fixture defined
+in the `test_build.py` script that makes it straightforward to run a Sphinx build using
+this theme and inspect the results. There are also several automated accessibility checks in
+`test_a11y.py`.
 
-In addition, we use [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/)
-to ensure that the HTML generated by the theme is what we'd expect. This module
+```{warning}
+Currently, the automated accessibility tests check the Kitchen Sink page only.
+We are working on extending coverage to the rest of the theme.
+```
+
+In addition, we use
+[pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/) to
+ensure that the HTML generated by the theme is what we'd expect. This module
 provides a `file_regression` fixture that will check the contents of an object
 against a reference file on disk. If the structure of the two differs, then the
 test will fail. If we _expect_ the structure to differ, then delete the file on
-disk and run the test. A new file will be created, and subsequent tests will pass.
+disk and run the test. A new file will be created, and subsequent tests will
+pass.
 
-To run the tests with `nox`, run the following command:
+To run the build tests with `nox`, run the following command:
 
 ```console
 $ nox -s test
 ```
+
+To run the accessibility checks:
+
+```console
+$ nox -s a11y
+```
diff --git a/docs/community/topics/accessibility.md b/docs/community/topics/accessibility.md
@@ -1,11 +1,18 @@
 # Accessibility checks
 
-The accessibility checking tools can find a number of common HTML patterns which
+```{note}
+April-2023: we are currently
+[re-evaluating how we do accessibility checks](https://github.com/pydata/pydata-sphinx-theme/issues/1168)
+and reporting, so this may change soon.
+```
+
+In general, accessibility-checking tools can find a limited number of common HTML patterns which
 assistive technology can't help users understand.
-We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.
-The configuration for Lighthouse is in:
 
-- `.github/workflows/lighthouserc.json`
+## Accessibility checks as part of our development process
+
+We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.
+The configuration for Lighthouse can be found in the `.github/workflows/lighthouserc.json` file.
 
-For more information about configuring lighthouse, see [the lighthouse documentation](https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md).
+For more information about configuring Lighthouse, see [the Lighthouse documentation](https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/configuration.md).
 For more information about Accessibility in general, see [](../../user_guide/accessibility.rst).
diff --git a/noxfile.py b/noxfile.py
@@ -5,6 +5,7 @@
 
     nox -s docs -- -r
 """
+import os
 import shutil as sh
 import tempfile
 from pathlib import Path
@@ -87,7 +88,28 @@ def test(session: nox.Session) -> None:
     if _should_install(session):
         session.install("-e", ".[test]")
     session.run(*split("pybabel compile -d src/pydata_sphinx_theme/locale -D sphinx"))
-    session.run("pytest", *session.posargs)
+    session.run("pytest", "-m", "not a11y", *session.posargs)
+
+
+@nox.session()
+def a11y(session: nox.Session) -> None:
+    """Run the accessibility test suite only."""
+    if _should_install(session):
+        session.install("-e", ".[test, a11y]")
+        # Install the drivers that Playwright needs to control the browsers.
+        if os.environ.get("CI") or os.environ.get("GITPOD_WORKSPACE_ID"):
+            # CI and other cloud environments are potentially missing system
+            # dependencies, so we tell Playwright to also install the system
+            # dependencies
+            session.run("playwright", "install", "--with-deps")
+        else:
+            # But most dev environments have the needed system dependencies
+            session.run("playwright", "install")
+    # Build the docs so we can run accessibility tests against them.
+    session.run("nox", "-s", "docs")
+    # The next step would be to open a server to the docs for Playwright, but
+    # that is done in the test file, along with the accessibility checks.
+    session.run("pytest", "-m", "a11y", *session.posargs)
 
 
 @nox.session(name="test-sphinx")

diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -5,6 +5,7 @@
     "build": "webpack"
   },
   "devDependencies": {
+    "axe-core": "^4.6.3",
     "copy-webpack-plugin": "^11.0.0",
     "css-loader": "^3.4.2",
     "css-minimizer-webpack-plugin": "^4.2.2",