axe api changes with more fine grained exceptions (#26)

* axe api changes with more fine grained exceptions

* raises correct exception

* raises type or tuple of types

* update settings test

* move exceptions to their own file

* add hypothesis to gitignore

* update git for pixi info

* update locks

* add pixi

* add a workflow that uses pixi

* control ff install location

* updates tests and dependencies

* pad cell numbers

* exclude widegets from settings test

* fix settings error

* include window sin the tests

* rm raise expr

* update pixi lock

* skip ubuntu windows

* update pixi

* add docs to pixi

* fix bad raise in test

* update the artifact downloader

* add scripts for extracting axe rules

* mv axe and testing utils

* update tests

* stricter test conditions

* change w3c fixtures

* bug in test

* update tests

* fix test and deprecation warning

* remove passing test

* updating tests to satisfy ci

* drop old non pixi workflow

* rnm pixi test workflow

* squash old code

* add generate function for testing axe

* add a test for the axe test tools

.gitattributes

@@ -0,0 +1,3 @@
+# GitHub syntax highlighting
+pixi.lock linguist-language=YAML
+

.github/workflows/pixi.yml

@@ -0,0 +1,109 @@
+name: test with pixi
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: ["*"]
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  PYTHONUNBUFFERED: "1"
+  PIP_DISABLE_PIP_VERSION_CHECK: "1"
+  USE_PIXI_VERSION: 0.22.0
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.6.0
+        with:
+          pixi-version: v${{ env.USE_PIXI_VERSION }}
+          cache: true
+          environments: build
+      - run: pixi run -ve build build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: nbconvert-a11y-${{ github.run_number }}-dist
+          path: ./dist
+
+  test:
+    name: (${{ matrix.os }}) test
+    needs: [build]
+    runs-on: ${{ matrix.os }}-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: ["ubuntu"]
+    # add matrix for browsers later on
+    steps:
+      - run: git config --global core.autocrlf false
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.6.0
+        with:
+          pixi-version: v${{ env.USE_PIXI_VERSION }}
+          cache: true
+          environments: test
+      - uses: actions/download-artifact@v4
+        with:
+          name: nbconvert-a11y-${{ github.run_number }}-dist
+          path: ./dist
+      # need to make this install from the wheel
+      - run: pixi run -ve test test
+      - if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: |-
+            nbconvert-a11y-${{ github.run_number }}-${{ matrix.os }}-test
+          path: |
+            ./.pixi/task-cache-v0
+            ./build/reports
+  
+  docs:
+    name: docs
+    needs: [test]
+    runs-on: ubuntu-latest
+    steps:
+      - run: git config --global core.autocrlf false
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.6.0
+        with:
+          pixi-version: v${{ env.USE_PIXI_VERSION }}
+          cache: true
+          environments: docs
+      - run: pixi run -ve docs build
+      - if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: nbconvert-a11y-${{ github.run_number }}-site
+          path: site
+
+  publish:
+    name: publish the mkdocs build to github pages
+    needs: [docs]
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout repo
+        uses: actions/checkout@v4
+      - uses: actions/download-artifact@v4
+        with:
+          name: nbconvert-a11y-${{ github.run_number }}-site
+          path: site
+      - name: Deploy main 🚀
+        uses: JamesIves/github-pages-deploy-action@v4
+        if: ${{ github.ref_name == 'main' }}
+        with:
+          folder: site # The folder the action should deploy.
+          single-commit: true
+      - name: Deploy non-main 🚀
+        uses: JamesIves/github-pages-deploy-action@v4
+        if: ${{ github.ref_name != 'main' }}
+        with:
+          folder: site # The folder the action should deploy.
+          single-commit: true
+          target-folder: branch/${{ github.ref_name }}

.github/workflows/test.yml

@@ -1,116 +1,98 @@
-name: pytest nbconvert-a11y, axe test exports, build docs.
+name: test with pixi
 
 on:
   push:
     branches: [main]
   pull_request:
-    branches: ['*']
+    branches: ["*"]
   workflow_dispatch:
 
+concurrency:
+  group: ${{ github.ref }}
+  cancel-in-progress: true
+
 env:
-  # Increase this value to reset cache if environments have not changed
-  CACHE_NUMBER: 2
-  # squash some known warnings
-  JUPYTER_PLATFORM_DIRS: 1
+  PYTHONUNBUFFERED: "1"
+  PIP_DISABLE_PIP_VERSION_CHECK: "1"
+  USE_PIXI_VERSION: 0.22.0
 
 jobs:
-  format:
-    name: format
+  build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.6.0
         with:
-          fetch-depth: 0
-      - uses: actions/setup-python@v4
+          pixi-version: v${{ env.USE_PIXI_VERSION }}
+          cache: true
+          environments: build
+      - run: pixi run -ve build build
+      - uses: actions/upload-artifact@v4
         with:
-          python-version: "3.12"
-          cache: pip
-          cache-dependency-path: pyproject.toml
-      - name: install dev dependencies
-        run: python -m pip install --upgrade pip hatch
-      - name: run formatters
-        run: |
-          echo "~~~bash" > "${GITHUB_STEP_SUMMARY}"
-          hatch run format:code 2>&1 | tee --append "${GITHUB_STEP_SUMMARY}"
-          echo "~~~" >> "${GITHUB_STEP_SUMMARY}"
-      - name: print diff
-        run: |
-          echo "~~~diff" >> "${GITHUB_STEP_SUMMARY}"
-          git diff | tee --append "${GITHUB_STEP_SUMMARY}"
-          echo "~~~" >> "${GITHUB_STEP_SUMMARY}"
+          name: nbconvert-a11y-${{ github.run_number }}-dist
+          path: ./dist
 
   test:
-    name: test package and accessibility
-    defaults:
-      run:
-        shell: bash -el {0}
+    name: (${{ matrix.os }}) test
+    needs: [build]
+    runs-on: ${{ matrix.os }}-latest
     strategy:
+      fail-fast: false
       matrix:
-        python-version:
-          - "3.10"
-    runs-on: ubuntu-latest
+        os: ["ubuntu"]
+    # add matrix for browsers later on
     steps:
-      - name: fetch all history and tags
-        uses: actions/checkout@v3
+      - run: git config --global core.autocrlf false
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.6.0
         with:
-          fetch-depth: 0
-      - uses: mamba-org/setup-micromamba@v1
+          pixi-version: v${{ env.USE_PIXI_VERSION }}
+          cache: true
+          environments: test
+      - uses: actions/download-artifact@v4
         with:
-          environment-file: .github/test-environment.yml
-      - name: init plawright
-        run: |
-          playwright install --with-deps chromium
-      - name: init node & files
-        run: |
-          yarn
-          doit copy
-      - name: init dev module
-        run: |
-          python3 -m pip install -e . --no-deps --no-build-isolation --ignore-installed
-      - name: check pip env
-        run: |
-          python3 -m pip check
-      - name: smoke test
-        run: |
-          # the smoke generate html assets that are used in the accessibility testing.
-          # we run this script to generate assets and test the nbconvert-a11y module.
-          # failures here will stop any docs builds
-          pytest --color=yes tests/test_smoke.py
-      - name: build wheel and sdist
-        run: |
-          pyproject-build
-      - uses: actions/upload-artifact@v3
+          name: nbconvert-a11y-${{ github.run_number }}-dist
+          path: ./dist
+      # need to make this install from the wheel
+      - run: pixi run -ve test test
+      - if: always()
+        uses: actions/upload-artifact@v4
         with:
-          name: dist
-          path: dist
-      - name: mkdocs
-        run: |
-          mkdocs build -v
-      - uses: actions/upload-artifact@v3
+          name: |-
+            nbconvert-a11y-${{ github.run_number }}-${{ matrix.os }}-test
+          path: |
+            ./.pixi/task-cache-v0
+            ./build/reports
+  
+  docs:
+    name: docs
+    needs: [test]
+    runs-on: ubuntu-latest
+    steps:
+      - run: git config --global core.autocrlf false
+      - uses: actions/checkout@v4
+      - uses: prefix-dev/setup-pixi@v0.6.0
+        with:
+          pixi-version: v${{ env.USE_PIXI_VERSION }}
+          cache: true
+          environments: docs
+      - run: pixi run -ve docs build
+      - if: always()
+        uses: actions/upload-artifact@v4
         with:
-          name: site
+          name: nbconvert-a11y-${{ github.run_number }}-site
           path: site
-      - name: a11y tests
-        # always build the docs to see what the new versions look like.
-        # continue-on-error: true
-        run: |
-          pytest \
-            --color=yes \
-            -n auto \
-            --deselect tests/test_smoke.py  \
-            --self-contained-html \
-            --html=tests/exports/pytest/report.html
 
   publish:
     name: publish the mkdocs build to github pages
-    needs: [test]
+    needs: [docs]
     runs-on: ubuntu-latest
     steps:
       - name: checkout repo
-        uses: actions/checkout@v3
-      - uses: actions/download-artifact@v3
+        uses: actions/checkout@v4
+      - uses: actions/download-artifact@v4
         with:
-          name: site
+          name: nbconvert-a11y-${{ github.run_number }}-site
           path: site
       - name: Deploy main 🚀
         uses: JamesIves/github-pages-deploy-action@v4
@@ -125,45 +107,3 @@ jobs:
           folder: site # The folder the action should deploy.
           single-commit: true
           target-folder: branch/${{ github.ref_name }}
-
-  release:
-    name: draft release when tagged
-    if: startsWith(github.ref, 'refs/tags/')
-    needs: [test]
-    runs-on: ubuntu-latest
-    permissions:
-      id-token: write
-      contents: write
-    steps:
-      - name: fetch contents
-        uses: actions/checkout@v3
-      - uses: actions/download-artifact@v3
-        with:
-          name: dist
-          path: dist
-      - uses: actions/setup-python@v4
-        with:
-          python-version: "3.11"
-          cache: pip
-          cache-dependency-path: pyproject.toml
-      - name: install twine and pytest
-        run: |
-          pip install twine pytest
-      - name: Publish package distributions to TestPyPI
-        run: |
-          twine upload --repository testpypi \
-          --user __token__ --password ${{secrets.HATCH_TEST_INDEX_AUTH}} \
-          dist/*
-      - name: install nbconvert-a11y dependencies from test pip
-        run: |
-          pip install \
-          --index-url 'https://test.pypi.org/simple/' \
-          --extra-index-url 'https://pypi.org/simple/' \
-          nbconvert-a11y
-      - name: test test release
-        run: |
-          pytest tests/test_smoke.py
-      - uses: ncipollo/release-action@v1
-        with:
-          artifacts: "dist/.*"
-          draft: true # does not trigger a created event

.gitignore

@@ -1,3 +1,9 @@
+.hypothesis
+.pixi
+.local-browsers
+
+
+dist/
 __pycache__
 .#*
 .doit.*
@@ -20,4 +26,4 @@ settings.json
 site
 tests/exports
 tests/out.html
-tests/outputs/*.html
+tests/outputs/*.html

nbconvert_a11y/axe/README.md

@@ -0,0 +1,27 @@
+# testing axe in python
+
+the `test_axe` module provides utilities for running `axe-core` on html document in python. it is designed for asynchronous interactive use and synchronous testing with the `pytest_axe` extension.
+
+`axe-core` is a javascript module that must be run inside of a live browser. currently, `playwright` is the only browser automation supporter, but `selenium` could be added.
+
+<!-- can we add it to robots from jupyter after that? -->
+
+## axe exceptions in python
+
+each release contains a version of axe and a set of exceptions generated for that specific version.
+the types are useful in accounting for expected accessibility that can be fixed later.
+
+## playwright testing
+
+it turns out the best ergonomics with working with axe from playwright are to append methods to the `Page` class. it has the following methods when `test_axe` is active:
+
+`Page.axe`
+: run axe and return the test results as a python dictionary
+
+`Page.test_axe`
+: run axe and return the test results as a python exception group
+
+`Page.aom`
+: an alias to the soon to be deprecated `page.accessibility.snapshot()`
+
+we have to main synchronous and asynchronous methods for playwright

nbconvert_a11y/axe/__init__.py

@@ -0,0 +1,4 @@
+from .async_axe import *
+from ..browsers import Browser
+from .axe_exceptions import *
+from .ipy import load_ipython_extension, unload_ipython_extension