Add pa11y testing and reporting #294

bollwyvl · 2020-12-11T19:09:43Z

Hi folks!

This is an 80% solution to the up-front work of integration pa11y-ci as proposed in #292. I can split this up, if needed, but kinda felt like getting it working end-to-end (up to CI reporting) was important for demonstrating value. Note that it shouldn't fail right now, as I've whitelisted all the known error codes... that's where the 20% comes in, and each of them will likely require a design decision.

dependencies
- adds sphinx-sitemap to docs/requirements.txt
- adds pa11y-ci and pa11y-ci-reporter-html to package.json
- add .yarnrc to do a local, offline package cache
  - the pa11y stack brings in a fair number of new dependencies, this helps reduce the impact
docs/conf.py
- use sphinx-sitemap (naievly) to generate a (simple) sitemap.xml
  - this is skipped on RTD, as they generate a better one with timestamps, etc.
docs/a11y.py
- runs pa11y-ci and the report generator
- reads docs/a11y-roadmap.txt
  - the built-in stuff just hides the reports for codes... we want to see them, so we can fix them, but not fail builds while in-progress
workflows/tests.yml
- adds python 3.9, and uses it for most tasks
- adds pip/yarn caching
- adds pip check so we don't get "fun" surprises, especially with the new pip solver
- normalize the ports for the tornado server
- call docs/a11y.py
- upload report
docs/*.rst
- add metadata which ends up in the meta tags on the built HTML to maintain lighthouse score
docs/accessibility.rst
- adds some of this stuff (definitely the most WIP)
docs/contributing.rst
- update with how to run locally, etc.

bollwyvl · 2020-12-11T19:28:11Z

I realized ignoring all of the a11y roadmap items doesn't make for good review, have commented out one to show some ❌ . Also, haven't run down the lighthouse issue, but it probably good to show that both audits run, even if one fails.

The audit step was already the long pole in CI at ~3min... we could split them up, but it would be even more duplication, etc.

bollwyvl · 2020-12-11T20:24:23Z

Ok, I'm pretty happy with my self-review of the docs, and the output that shows up in both the log and the annotations. Probably no further work planned on this, gonna bop over to #293...

bollwyvl · 2020-12-18T02:23:13Z

Just updated this to master...

choldgraf

This looks really nice and I love these docs on accessibility for the user :-) - my main concern is that the CI/CD builds now feel much more complex to me (e.g. I can read and understand the old ones with a casual glance as a non-expert in github actions, while these versions look much scarier to me (and thus will look like this for other non-expert maintainers as well, I suspect). Perhaps others can tell me if I am just being too conservative here?

But in general, I think this looks really nice, thanks for putting in this work to make sure the theme is closer to meeting the a11y goals that we should have for our communities.

I know @jorisvandenbossche may have some time to think about this theme in the next week, perhaps we can plan to discuss then?

choldgraf · 2020-12-18T10:59:45Z

.github/workflows/tests.yml

+    - name: Cache pip
+      uses: actions/cache@v1
+      with:
+        path: ${{ steps.pip-cache.outputs.dir }}


why do we now need this much more complex install config? It's fine if it's really needed, but just looking at it seems much less-maintainable to an average person to me 🤔

sure, can drop the cache.

well if it's really useful then we should keep it, I just think we should be in the habit of justifying added complexity and maintenance burden, so I'm wondering what benefit it brings! (not out of skepticism, just because I do not know)

simplified on 8e122bb etc

Using the CI cache makes lighter use of community resources, e.g. pypi. It's not making a detectable speed difference, however in this case.

To really drive down CLI complexity, I can highly recommend doit... it's nice for multi-language projects like this.

choldgraf · 2020-12-18T11:02:03Z

docs/contributing.rst

+Auditing accessibility
+======================
+
+The accessibility checking tools are installed along with the rest of the `yarn`


could you add in a link to pa11y and maybe a description of what we mean by "auditing accessibility"? Many devs may have no experience with a11y or UI audits, so this is all new to them. (maybe this is just fixed by linking to the user-facing docs on a11y in this PR?)

update on 970ddd5

docs/contributing.rst

choldgraf · 2020-12-18T11:04:19Z

docs/user_guide/accessibility.rst

+----------------
+
+If not using a more robust `internationalization approach <https://www.sphinx-doc.org/en/master/usage/advanced/intl.html>`__ ,
+speciying at least the baseline natural language will assistive technology


Suggested change

speciying at least the baseline natural language will assistive technology

speciying at least the baseline natural language will help assistive technology

choldgraf · 2020-12-18T11:04:47Z

docs/user_guide/accessibility.rst

+different content appears on a website.
+
+If using a service like `ReadTheDocs <https://readthedocs.com>`__, these files
+will be created for your, but for some of the other approaches below, it's handy


Suggested change

will be created for your, but for some of the other approaches below, it's handy

will be created for you automatically, but for some of the other approaches below, it's handy

bollwyvl · 2020-12-18T14:34:32Z

I'll also go ahead and un-comment the demonstration rule in the roadmap, so that the audits start passing.

bollwyvl · 2020-12-18T15:37:28Z

Alright, back to 💚... hardly seems like i could let some of the scores slide on this particular PR!

choldgraf · 2020-12-18T15:41:30Z

Great - this looks really nice @bollwyvl, thanks :-) I'll await @jorisvandenbossche's (and maybe @hoetmaaiers if he's interested) thoughts since it's a big-ish change to the CI/CD and we should get some team buy-in

isabela-pf · 2021-07-02T17:33:31Z

Is there something still blocking this PR? I'd love to see this merged since so many projects use this theme to produce documentation and this could be a great way to start bringing accessibility into more communities.

choldgraf · 2021-07-02T20:46:58Z

Mostly I think it is waiting a 👍 from others in the team (maybe in particular @jorisvandenbossche ) since it is a big ish change to the codebase

MarsBarLee · 2021-07-05T16:21:16Z

I can't comment on the changes since I don't know much about CI, but NumPy would certainly appreciate this!

bollwyvl · 2021-08-25T19:49:32Z

All back up to running/reporting... I'd be happy to start fixing some of these things (like investigating the lighthouse regressions) once we start putting these things under test...

choldgraf

This looks good to me - I don't have a ton of experience with a11y but am fine merging this in and iterating rather than having a big ol' PR dangling here.

choldgraf · 2021-08-25T22:08:39Z

docs/conf.py

+# specifying the natural language populates some key tags
+language = "en"
+
+# ReadTheDocs has its own way of generating sitemaps, etc.


why don't we just use sphinx_sitemap anyway? Do they clash or something? I'd be fine just using the extension rather than special-casing RTD unless it's noticeably better

Yeah, the RTD one is generally better. is aware of versions/languages, etc.

choldgraf · 2021-08-25T22:10:53Z

docs/demo/demo.rst

@@ -555,3 +555,8 @@ And here we demonstrate ``xarray`` to ensure that it shows up properly.
            coords={"x": [10, 20]}, attrs={"foo": "bar"}
          )
   data
+
+
+.. meta::


Is this like a section-specific metadata description or something? I haven't seen these in Sphinx world before

this populates the for-real meta tag in the head of a document: https://docutils.sourceforge.io/docs/ref/rst/directives.html#metadata

docs/user_guide/accessibility.rst

choldgraf · 2021-08-25T22:13:46Z

docs/user_guide/accessibility.rst

+        The features and plans for addressing accessibility concerns on pydata-sphinx-theme
+
+*************
+Accessibility


Just a quick thought but doesn't need to block this PR - I wonder if the ReadTheDocs team (👋 @ericholscher @astrojuanlu) would be interested in maintaining a guide like this for the broader docs community, so it's not just in this theme.

welp, we've already seen a number of thumbs on this PR from various impactful downstreams, so there's interest. If anything, i'd rather see the baseline sphinx themes become more aware of accessible HTML5, but i haven't dipped my toe in that community, yet.

one thing we've been discussing is extracting the reporting/roadmapping capability into something standalone (e.g. local and in CI), potentially under the jupyter/acessibility banner... we've already replicated much of this over there, though it hasn't been exercised very often to keep a stack under continuous assessment. Probably out of scope for docs sites, but for complex things like web apps, there's a lot more that can be done with the keyword-based features, e.g. Click on #search-input, Type foo bar.

@bollwyvl you may be interested in @pradyunsg's effort to build a new "base sphinx template" that follows more modern best practices: https://github.com/pradyunsg/sphinx-basic-ng

We are even considering re-factoring this theme to build off of that one (potentially something that @damianavila will look into soon!)

Yeah, that would be great. Whatever it is, throwing as much possible spec validation at the lowest substantive level will yield the greatest impacts for authors and, ultimately, readers. If a subtheme comes along and hacks a span soup on top of a beautifully-organized semantic layout, then the project actually trying to publish some spec-compliant docs won't get much benefit.

As to adopting that theme: having this testing in place, at this level, starting to fix roadmap issues, and blocking merges when audits fail would make it more reasonable to do a large-scale dependency swap, and catch regressions.

yeah - I agree with all of those points.

One challenge that I foresee here is that this project already has a problem of stagnation. We have a lot of open PRs that are dangling and have gone stale (as witnessed by this PR!)...I do worry that yet another thing that raises the energy barrier to merging things will also make it less likely that anything changes.

Maybe one example is the CI/CD in this PR. It seems that something called "lighthouse" is causing the CICD to fail. I'm not really sure what that means but I think it means we are below some kind of quality standard on accessibility?

Indeed, it looks like CI hasn't run successfully for the last two months, or so.

So somewhere in the last few months, the codebase (or one or more of its dependencies that actually make DOM) regressed on performance, privacy, best practices, and/or accessibility metrics that are reflected in the lighthouse report from some "high water" mark (presumably what it was doing when I originally made the PR). We can track back to #216, when lighthouse was added, and compare against the data from when the issue arose, and start clawing value for readers back. But if nobody's looking, it can't get any better, basically the whole premise of this PR.

I see robust tests and checks as a key enabler to improving the process of accepting high-quality contributioons: there's enough important stuff reliant on this code that "move fast and break things," probably isn't the right mantra anyway.

I agree with all of this - I'm not advocating for moving fast, just trying to be realistic given the very limited developer resources this theme has on an ongoing basis.

It sounds like from an a11y perspective, and given that the tests were broken before, that the best approach would be to merge this in and then see if there are more actionable things to do to get the tests passing?

Yeah, I could unbreak some more findings (that's why that meta stuff is there, for example), but again, master isn't green right now. Once we have forensic data to start looking at, we can move forward more confidently.

Co-authored-by: Chris Holdgraf <choldgraf@gmail.com>

choldgraf · 2021-08-25T23:20:57Z

Note for others (particularly @jorisvandenbossche I suspect). I'll plan on merging this unless there are really significant changes in 48hrs. If you object please say so and I'm happy to hold off!

choldgraf · 2021-08-27T18:50:04Z

OK - as we have (mostly) passing tests (except for one we expect to be broken on master), and since a11y is important, and since this PR has been open for 8 months, and since nobody has objected in the last two days. I'm going to merge this :-)

thanks @bollwyvl for slogging through this one!

isabela-pf · 2021-08-27T18:54:50Z

Hooray! Thanks @bollwyvl for sticking with this and @choldgraf for getting it reviewed and merged! This is a big step to helping a lot of projects identify and fix inaccessibility in their documentation. 💕

melissawm · 2021-08-27T18:58:29Z

Great news! Thank you @bollwyvl, @choldgraf and all involved for your work on this!

bollwyvl added 2 commits December 11, 2020 13:46

add pa11y-ci reporting

b63a39c

update docs

5755c49

bollwyvl mentioned this pull request Dec 11, 2020

Plan to improve accessibility documentation (and testing) #292

Closed

bollwyvl added 2 commits December 11, 2020 14:14

fix warnings in new docs

ab8748e

linting, comment out a roadmap item so it will fail

cb3149d

add some developer docs, in-browser tools

1c10ad7

bollwyvl changed the title ~~[wip] Add pa11y testing and reporting~~ Add pa11y testing and reporting Dec 11, 2020

bollwyvl added 4 commits December 11, 2020 14:57

try to get audit to show up in annotation

2ca1778

yep, i sure don't know rst

dfabae5

try annotations directly

e26b2cb

clean up more docs

ecf6970

Merge remote-tracking branch 'upstream/master' into add-pa11y

c8001f2

choldgraf reviewed Dec 18, 2020

View reviewed changes

bollwyvl added 4 commits December 18, 2020 08:36

further streamline test workflow

8e122bb

rework language about audits, links to specs

970ddd5

fix reporter

8db7e24

address more comments

0997016

bollwyvl added 6 commits December 18, 2020 09:35

uncomment intentional fail

4bb257c

restore http streaming log

125d5fb

more reporting

eb84c10

get SEO score back up by adding metatags, normalizing URLs to 127.0.0.1

fac2b3d

fix url for accessibility page in lighthouse

c846cae

tweak some language settings

3376f82

bollwyvl mentioned this pull request Jan 16, 2021

January 2021 Docs Meeting pandas-dev/pandas#39183

Closed

bollwyvl added 2 commits March 25, 2021 09:52

merge upstream

88c60f3

resolve yarn.lock

d6c043d

bollwyvl mentioned this pull request Mar 27, 2021

add badges, minimal codecov, mk2 #352

Merged

bollwyvl mentioned this pull request Apr 6, 2021

Move away from pandas styling of the default (demo) theme #373

Closed

bollwyvl added 5 commits August 25, 2021 13:57

merge upstream

6aef1c6

resolve yarn.lock

c8ecd78

revert default port to 8000

df90d50

install nodejs

1f1ba7a

improve artifact uploading

11a4650

bollwyvl requested review from choldgraf and jorisvandenbossche August 25, 2021 19:48

choldgraf approved these changes Aug 25, 2021

View reviewed changes

Update docs/user_guide/accessibility.rst

809c5ab

Co-authored-by: Chris Holdgraf <choldgraf@gmail.com>

choldgraf merged commit 9232480 into pydata:master Aug 27, 2021

bollwyvl mentioned this pull request Aug 27, 2021

Get CI audits back to passing #455

Closed

djhoese mentioned this pull request Aug 28, 2021

Fix accessibility issues in website vispy/vispy#2138

Open

trallard mentioned this pull request Nov 11, 2021

Add automated a11y tests to JupyterLab Quansight-Labs/jupyter-a11y-mgmt#4

Closed

4 tasks

gabalafou mentioned this pull request Nov 18, 2021

Team update - Nov 17, 2021 Quansight-Labs/jupyter-a11y-mgmt#61

Closed

damianavila mentioned this pull request Jan 11, 2022

MAINT: Use different lighthouse action #548

Closed

isabela-pf mentioned this pull request Jun 6, 2022

[TEST] - Add automated accessibility tests to Jupyterlab's documentation theme Quansight-Labs/jupyter-a11y-mgmt#14

Closed

trallard mentioned this pull request Nov 11, 2022

[DEL]- Contribute automated tests to PyData Sphinx theme Quansight-Labs/jupyter-a11y-mgmt#26

Closed

trallard mentioned this pull request Feb 8, 2023

ENH - Improvements to automated accessibility tests #1168

Closed

	speciying at least the baseline natural language will assistive technology
	speciying at least the baseline natural language will help assistive technology

	will be created for your, but for some of the other approaches below, it's handy
	will be created for you automatically, but for some of the other approaches below, it's handy

Add pa11y testing and reporting #294

Add pa11y testing and reporting #294

Conversation

bollwyvl commented Dec 11, 2020 • edited Loading

bollwyvl commented Dec 11, 2020

bollwyvl commented Dec 11, 2020

bollwyvl commented Dec 18, 2020

choldgraf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bollwyvl Dec 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bollwyvl commented Dec 18, 2020

bollwyvl commented Dec 18, 2020

choldgraf commented Dec 18, 2020

isabela-pf commented Jul 2, 2021

choldgraf commented Jul 2, 2021

MarsBarLee commented Jul 5, 2021

bollwyvl commented Aug 25, 2021

choldgraf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

choldgraf Aug 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

choldgraf commented Aug 25, 2021

choldgraf commented Aug 27, 2021

isabela-pf commented Aug 27, 2021

melissawm commented Aug 27, 2021

bollwyvl commented Dec 11, 2020 •

edited

Loading

bollwyvl Dec 18, 2020 •

edited

Loading

choldgraf Aug 26, 2021 •

edited

Loading