Suggestion to lean on existing CI services

[joecorcoran]

Firstly, let me say – awesome ideas! I'm excited about the prospect of Pa11y becoming a high-profile part of software development.

Having read through the recent announcement, it seems like the plans for Sidekick are so similar to existing continuous integration tools in a number of ways. This makes me wonder – why not lean on one of them?

Since a continuous integration service (disclosure: I work for Travis CI) can already handle the webhook –> pa11y –> post results part of the pipeline, would it make sense to focus efforts on building an awesome pa11y dashboard that can interpret those results?

This would seem to have a number of benefits.

1. Less work required from the Pa11y team
2. Lean on years of acquired knowledge in a specific part of the pipeline
3. Running a11y tests as part of the "proper" build would privilege the concept of a11y tests generally. As a separate service there is the potential for Pa11y test results to be yet another badge in the README file to be kinda ignored ("a11y tests failed but the real tests passed, so let's deploy and fix that later").

If you're interested in leaning on Travis for builds, for example, I'm happy to advise you along the way. What do you all think? I realise I'm coming at this with little background, so maybe I'm missing crucial details.

[rowanmanning]

Hi Joe, thanks for your suggestion!

I'd like to be able to do this if we can, the one barrier we have is that Pa11y requires the site to be deployed ideally before it can run tests, e.g. to a QA environment. It's not possible to fail builds in most CI systems after they've deployed, if I'm right?

We could still possibly have CI run the tests against a running instance of an application and push results to the relevant dashboard. Then CI could still decide to fail if there are too many errors in that scenario.

I've investigated the possibility of building a pa11y-ci module in the past which is more of a CLI interface that's more geared towards a CI environment. Maybe there's more overlap with Sidekick than I thought?

If we build Sidekick in a way where it can accept results as JSON from any source, then that would be ideal. We could probably come up with a minimum viable product much more quickly that way. E.g. build the database/API/display first, then work on different ways to run the tests could happen in parallel.

Does that make sense? I feel like I'm rambling.

Anyway we'll definitely keep this in mind as we're working. Thanks again! Ideas are flowing, and we'll be in touch for advice when it comes to build time :)

---

edit: I think at least half of my reply is repeating your suggestion in slightly different words 😄

[joecorcoran]

I'd like to be able to do this if we can, the one barrier we have is that Pa11y requires the site to be deployed ideally before it can run tests, e.g. to a QA environment. It's not possible to fail builds in most CI systems after they've deployed, if I'm right?

I'm wondering if you could run a web server on the CI box and then run e.g.
pa11y http://localhost:3333 instead of pa11y some-website.net – is there a conceptual reason why it needs to run against a site that was already deployed? Maybe I just don't have the context.

[rowanmanning]

Nope. I think that really was just my (incorrect) stream of consciousness. We already do exactly this for the Pa11y website itself on Travis. We could definitely build a tool that streamlines this and sends the full report to Sidekick, while still failing builds if we want it to.

[joecorcoran]

Awesome, yeah that's exactly what I had in mind. Let me know if/when you need any help on the CI side. 🚀

[lawrencec]

Great project btw!

Funnily enough, I also wanted to suggest leaning on CI services. Would be good to leave the scheduling to CI and concentrate on the dashboard. That's how a lot of other services work.

There is another use case that I don't think is covered by Pa11y yet and that is running tests on pages that are part of a journey e.g pages that need to be signed in to view. That's where the advantage of running a webservice to collate the results can come in.

Integration/E2E tests (as run by CI) can run the pa11y tests at any point within the journey and ping the result to a server and a dashboard can be built from that. I helped build a page diffing dashboard tool exactly this way and we made plugins for the various functional testing frameworks, e.g webdriver, out there to integrate with the webservice. Some additional work was required to group results by journeys, builds and browsers for the dashboard but I don't think this changes your existing road plan too much. Pa11y can still be a dashboard site which accepts urls but one that also integrates with people's existing systems.

In any case, I think that the road plan should consider this use case as not enough a11y testing is done by devs or test teams and anything that can be done to lower the barrier would be helpful.

Not entirely certain that my interpretation of the existing road plan was correct, so I hope I haven't got too much wrong and perhaps you considered this use case already?

[rowanmanning]

Hey @joecorcoran, we're definitely now leaning towards the core/first version leaning on CI for test running. There's some work that'll need doing to get there, and we have another proposal for a CI-centric test runner.

It'd be great to get your thoughts on that if you have time. Also thanks for starting this discussion :)

[joecorcoran]

Cool! I'm away at the moment, but just letting you know I've seen this. I'll reply in detail next week when I'm back. 🔜

[joecorcoran]

Just read through the Pa11y CI proposal. Nice work! 🎉 It all sounds extremely useful. Agree that it should stay separate from the main Pa11y tool, although it can probably use it under the hood, right?

The only thing that jumps out at me right now, is that pushing individual page results to Sidekick as they occur, rather than waiting for the end of the run and pushing once, will add a significant amount of development work. It makes each Sidekick build into a big state machine and if one request is dropped for whatever reason, you'll end up with difficult choices about how to display an incomplete build. I would consider sticking with collecting the results like a regular test runner and then posting them to Sidekick once at the end, at least as a first iteration.

[rowanmanning]

Thanks for the feedback @joecorcoran! Actually I have a PR in Pa11y CI with an initial build. It doesn't include Sidekick for now but it's already much more useful than Pa11y itself in a CI environment 👍 (FYI not expecting you to look over that whole monster PR – the README should give you enough of an idea if you're interested in how it works).

...posting them to Sidekick once at the end, at least as a first iteration.

Yep, I agree that the full Sidekick results should be posted once at the end. The way CI is currently working lends itself towards this.

...separate from the main Pa11y tool, although it can probably use it under the hood, right?

Also yes, it is using the main Pa11y lib :)

[stvnrlly]

Hey there, I wanted to make sure that you all know about continua11y, which I think has some similar goals here and therefore might be useful as you plan this project. The idea was to test accessibility as part of the CI suite rather than at recurring intervals, and to provide badges as a sort of encouragement to others. You can see a version of it running. It works fairly well but has some obvious problems.

It's built with a heavy reliance on pa11y, with two additional tools, and runs as a step inside of Travis or CircleCI. First, pa11y-crawl to find all of the pages on a site, run pa11y on them, and produce a single report. Second, pa11y-reporter-full-json to add the URL and such into the report to allow (hopefully) for parallel scans.

I have plans to work on it some more, but would be very happy to replace the whole setup with Sidekick.
Let me know if you have questions about why I did things a certain way, or if I can be of use here.

[syndy1989]

@stvnrlly

Hi there, I'm able to get all the pages on a site but how to run pa11y on them to produce a single report. Any advise on this would be helpful. Thanks in advance.