Feature request: establish syntax to allow users to pass all available config to service through runners

[calebtr-metro]

Currently, pa11y does not appear to pass all configuration options to runners to use with a service.

For example, Axe has configuration options such as "disableOtherRules" and "checks" that do not get passed through the axe runner. I realize I can open a ticket on the axe runner and/or fork it and fix that - but it seems better to have a solution that would work more universally.

One use case to use a custom script, as in #543. I think the solution is good as it clearly distinguishes pa11y config from runner/service config.

My initial use case a user can create a custom rule for axe and run it through pa11y. I didn't get any traction on stack overflow, so apologies if this feature exists and I am just not finding it.

A sample custom rule for axe would be:

{
        rules: [
            {
                id: 'my-cool-rule',
                enabled: true,
                all: ['auto-fail'],
            },
        ],
        checks: [
            {
                id: 'auto-fail',
                metadata: {
                    impact: 'critical',
                    messages: {
                        pass: 'Surprise! This test passed!',
                        fail: 'This test did not pass, sadly.'
                    }
                },
                evaluate: function () {
                    return false;
                }
            }
        ]
    }

Solutions:

Add config objects to the runners property, as in #543. This keeps pa11y and service configs separate.

Alternatively, pa11y could look for a file, axe.config, etc. when passing config to the runner.

We would also need to establish which set of config takes precedence when there is a conflict (ie from my reading, what pa11y calls "rules", axe calls "standards", and axe uses "rules" to mean specific rules that are actually a set of checks).

Runners would have to be updated to actually send the rules on to the services.

Thanks for your time, and again, apologies if there is a way to do this already.

[joeyciechanowicz]

This would be useful, and would allow users to configure the runners outside of the realm of what pa11y supports.

The two runners we have (HTML_CS and aXe) are both configured a bit differently.

HTML_CS

The only configuration we perform is to enable rules by pushing them on a window object. And then pass a standard at the run call. I'm not aware of any more configuration we can perform on HTML_CS.

aXe

We build the configuration during one function, getAxeOptions(https://github.com/pa11y/pa11y/blob/master/lib/runners/axe.js#L69). Which would mean it would be possible to merge in a runnerConfig object.

Potential approach

IMO we should surface a runnerConfig to the runners, but leave it up to the runner to then do anything with. We would need to name the config after the runner as we pass the same arguments to each runner.

Config would then look like:

{
   "runnerConfig":{
      // must match the name of a runner
      "axe":{
         "rules":[
            {
               "id":"my-cool-rule",
               "enabled":true,
               "all":[
                  "auto-fail"
               ]
            }
         ]
      }
   }
}

And then pass it in as a 3rd optional argument to the runner

runner.run = async (options, pa11y, runnerConfig) => {
};

Regarding the precedence of config options, I think pa11y options should overwrite any extra configuration as that should be the most common path.

@Dikshita25

[calebtr-metro]

Ah, thanks for looking at this.

What are next steps?

• update docs in pa11y
• create a pr in each runner project ?

[joeyciechanowicz]

Well, al the runners are now part of pa11y in the lib/runners folder. So that's much easier.

So steps (in my mind) would be something like:

• Add support for a runnerConfig: {axe: {}, htmlcs} section
  • Grab the runners config (if it has one) off the options and pass it as a 3rd argument in the runner.js
• Tests!
• Docs