Release pa11y 2.0

[rowanmanning]

Pa11y 2.0 is currently in development, and viewable on the 2.x branch for now. Issues will appear in the 2.0 milestone, and we'll be aiming for near feature parity with pa11y 1.x.

Pa11y 2.0 is currently in an alpha state, and is not ready to replace 1.x as part of your toolchain. This is mainly because not all of the accessibility rules present in 1.x have been built yet. Also, things may (and are likely to) change before we reach Beta. You can install the alpha version with:

npm install -g https://github.com/nature/pa11y/tarball/2.x

For contributors: feedback is hugely appreciated, and no change to the current 2.0 API/code is too big to consider.

The major changes in this version are as follows:

• Drop the PhantomJS dependency
• Drop HTML CodeSniffer as a tester
• Drop Section508 support (just for initial release – it can be built in later)
• Allow writing custom rules
• Allow configuring of existing rules
• Allow testing of HTML snippets as well as URLs
• Allow ignoring rules and changing error levels

[rowanmanning]

@dotcode, @whymarrh, @stephenmathieson – I thought you might be interested to see this.

[stephenmathieson]

No phantom?!? What are you using for analysis?

[rowanmanning]

Currently trying jsdom, which works well so far because everything's in a Node context rather than having to write Phantom scripts. Open to change if it doesn't work out :)

[mfairchild365]

I'm curious, why you are dropping HTML_CodeSniffer as a tester?

[rowanmanning]

@mfairchild365 one of the things we've wanted to do for quite a while is customise the rules and suites, as well as be able to write our own. This wasn't really possible (or easy) with HTML CodeSniffer.

Also because HTML CodeSniffer is a bookmarklet, it imposes lots of technical restrictions for running it on the command-line – the majority of the code in pa11y 1.x is working around issues running a bookmarklet via PhantomJS on the command line.

Does this answer your question? It's something we've been thinking about for a while, and should hopefully result in a lot more innovation from pa11y in future.

[mfairchild365]

Yep, that answers my question. :)

This is me just thinking out loud, so feel free to ignore it. I've noticed several test suites for accessibility either in development or recently released. Just a few of them:

• https://code.google.com/p/ainspector/
• http://tenon.io/
• http://quailjs.org/
• http://wave.webaim.org/
• http://squizlabs.github.io/HTML_CodeSniffer/
• http://achecker.ca/checker/index.php

And frankly, all of these have problems. Writing solid tests, implementing them, and making them understandable to less technical people is difficult. I think that rather than competing with each other and creating distinct systems, it would be better to unite under one flag. A w3c community was recently formed to address many of these problems: http://www.w3.org/community/auto-wcag/

One thing that I like about the current version of pa11y is that it tries to use an existing tester rather than implementing its own. I also like that it allows web developers to fire up the bookmarklet and pinpoint errors that were reported by pa11y, visually highlighting the problem area and making the problem easier to understand.

I understand that there are technical limitations with continuing to use a 3rd party tool, but I think realizing and addressing the benefits of using a 3rd party tool is helpful.

Just some thoughts from someone who is invested in the 'automated accessibility testing industry', wants to see it succeed, and likes pa11y and the work that you all have done. :)

[dotcode]

@rowanmanning indeed, very interested, thanks :)

[whymarrh]

Really happy to hear about v2!

I like what @mfairchild365 has pointed out, he has listed a few tools that are in the same category and it might make sense to move in the direction of a tool that integrates with others. It's not a easy task, but something to consider — it's one of many possibilities.

Regarding PhantomJS: I think moving away from it is both a great idea and one that may cause pose a new set of problems. If I understand correctly, some of the WCAG deal with things like colour contrast — I'm not sure if analyzing that given only the DOM is possible? (The good thing about moving away from PhantomJS, like @rowanmanning said, is not having to write scripts for it or having to communicate with it, which is a huge pain point in general.)

Some quick thoughts in terms of features for v2:

• This has already been listed, but the ability to run tests against both live sites, local sites, and HTML snippets (e.g. a file or stdin) would be huge.
• Run tests against a whole site recursively? (spidering support #16)

[whymarrh]

Some ideas regarding architecture (and now it's me thinking aloud), maybe building pa11y to support both a 3rd-party lib such as HTML_CodeSniffer alongside the ability to write custom tests could work?

Also, maybe running PhantomJS as a server and communicating with it via HTTP? That might help to keep things sane while still reaping the benefits of having the full web stack (DOM, CSSOM, JS VM, etc).

[rowanmanning]

Totally not ignoring your excellent responses – just rushing to get sprint work finished for the end of the week. Proper reply to follow.

[rowanmanning]

@mfairchild365 that's an excellent list! I've seen some of these, but not all, really useful. I'm with you on the benefits of using a 3rd-party tool, and it's not a decision we took lightly. The way we're mostly using pa11y is as part of the dashboard for which we really need to focus on speed and lower resource consumption – ideally anyone should be able to spin up a dashboard on cheap or free cloud hosting (e.g. Heroku).

We'll be actively maintaining the 1.x tree for the forseeable future, as a lot of people are relying on the HTML_CodeSniffer output format.

I wasn't aware of the Automated WCAG Monitoring Community Group, thanks for sharing! It's on my list of things to peruse before continuing.

@whymarrh I'm going to have to test the contrast thing actually. I assumed I'd be able to do it with jsdom, but I should check sooner rather than later for sure!

On features: yeah I was quite pleased that stdin/HTML snippet support just kind of came for free with jsdom. Also with #16, I tested this briefly with the existing pa11y (very hacked together script) when we were building dashboard. It was slow to say the least and a massive resource hog. I would really like to do this, but might consider it as a separate project down the line.

I do really like the idea of being able to plug multiple 3rd-party libs into pa11y, I'm going to investigate feasability. My only concern is the number of workarounds/command-line options we might have to squeeze in. I'd ideally like to keep the next version as lean/fast as possible.

Thanks so much for your feedback, all. I've got a bunch of stuff to research/experiment with!

[whymarrh]

Another idea for 2.x: Multiple test configurations (from #50). Run tests against multiple standards, viewports, URLs, or what have you.

[paipai]

Glad to hear about the V2.
Lots of changes, I'll test it soon. This seems to me a good direction to drop dependencies on other tools.

I'll really apreciate too :

• Run tests against a whole site recursively? (spidering support #16)
• And a way to get a backup of the source code and screenshot of the tested page (but without phantom it seems difficult to make screenshots).

Thanks for the nice job.