Config options are not reflected in results

[mohanraj-r]

I am not able to get the config options working. I have tried with axe-webdriverjs and puppeteer (#1008)

Versions:
axe-core@3.1.2
axe-webdriver@2.0.1
selenium-webdriver@4.0.0-alpha.1
chromedriver@2.42.0
Google chrome: Version 69.0.3497.100 (Official Build) (64-bit)

For Tooling issues:
- Node version: v10.11.0
- Platform:  Mac v10.13.6 

Webdriver code:

const AxeBuilder = require("axe-webdriverjs")
const webDriver = require('selenium-webdriver');

// Config options for axe
//  Ref: https://github.com/dequelabs/axe-core/blob/develop/doc/API.md#options-parameter
const axeConfig = {
  runOnly: {
    type: "tag",
    values: ["wcag2a", "wcag2aa"]
  },
  resultTypes: ["violations", "incomplete"]
};

const driver = new webDriver.Builder()
  .forBrowser('chrome')
  .build();

driver.get(url)
  .then(() => {
    // @ts-ignore
    AxeBuilder(driver)
      .configure(axeConfig)
      .analyze((results) => {
        console.log(JSON.stringify(results));
      });
  });

Output (only first-level keys are shown below for brevity):

{
  "inapplicable": [ ...
],
  "incomplete": [ ...
 ],
  "passes": [ ...
],
  "timestamp": "2018-10-09T17:16:58.320Z",
  "url": "https://www.google.com/",
  "violations": [ ...
]
}

I tried the following in chrome devtools (the browser instance opened by the above selenium webdriver code):

const axeConfig = {
  runOnly: {
    type: "tag",
    values: ["wcag2a", "wcag2aa"]
  },
  resultTypes: ["violations", "incomplete"]
};
axe.run(document, axeConfig)

Promise {<pending>}
__proto__: Promise
[[PromiseStatus]]: "resolved"
[[PromiseValue]]: Object
inapplicable: Array(22)
0:
description: "Ensures <area> elements of image maps have alternate text"
help: "Active <area> elements must have alternate text"
helpUrl: "https://dequeuniversity.com/rules/axe/3.1/area-alt?application=webdriverjs"
id: "area-alt"
impact: null
nodes: []
tags: (5) ["cat.text-alternatives", "wcag2a", "wcag111", "section508", "section508.22.a"]
... // output snipped 

You can see from the above output that even though the config specifies resultTypes: ["violations", "incomplete"] the output in both webdriverjs and direct axe.run() invocation in devtools contains other result types

[mohanraj-r]

I also tried restricting the runOnly to values: ["wcag2a"] - but I find the output contains results from other tags as well

const axeConfig = {
  runOnly: {
    type: "tag",
    values: ["wcag2a"]
  },
  resultTypes: ["violations", "incomplete"]
};

...
  "tags": [
        "cat.color",
        "wcag2aa",
        "wcag143"
      ]
...
"tags": [
        "cat.sensory-and-visual-cues",
        "wcag2aa",
        "wcag144"
      ]
...
 "tags": [
        "cat.language",
        "wcag2aa",
        "wcag312"
      ]
...
"tags": [
        "cat.text-alternatives",
        "wcag2aa",
        "wcag125",
        "section508",
        "section508.22.b"
      ]

[jhausler1266]

@dylanb @marcysutton i'm not sure if you can help us with this item, but we're hoping to integrage aXe internally, and this is currently blocking us. Thanks.

[dylanb]

@jhausler1266 The resultTypes option will not eliminate the other results, it will only truncate them. Can you provide the complete result object so we can verify?

[mohanraj-r]

Thanks @jhausler1266 @dylanb

The entire output json and the test js used to produce the output is at https://gist.github.com/mohanraj-r/080d97bdbba6a59a61452cd0aeb24efc

For the config

const axeConfig = {
    runOnly: {
        type: "tag",
        values: ["wcag2aa"]
    },
    resultTypes: ["violations", "incomplete"]
}

there are 2 issues I see with the output

1. resultTypes is not respected. Output contains inapplicable and passes as well
2. runOnly is not respected. There are several Results in the output with tags that don't have wcag2aa

Both of which shouldn't occur according to the config options doc

[marcysutton]

@mohanraj-r those are definitely separate issues, as the code for those lives in different places. Can you open a separate issue regarding runOnly and tags, and we can focus this one on resultTypes?

For resultTypes, the output when specifying violations and incomplete is expected to still contain arrays for inapplicable and passes; those are required for API stability with other products. You can filter them out yourself, if you don't want them to appear.

However–@dylanb, as I mentioned internally the API docs are wrong for resultTypes; the text mentions reducing down to a single pass/inapplicable result but instead it filters the nodes within each of those, and the result arrays stay the same size. We should either change that behavior to actually reduce results or fix the docs to mention filtering nodes. IMO it's a very confusing API as-is.

[dylanb]

@marcysutton we should clarify the documentation because the behavior is as intended.

For clarity: the intent of the resultTypes option is to provide a performance improvement by restricting the nodes returned for each rule to 1 node in each of the groupings not included in resultTypes

This provides a performance improvement by avoiding the calculation of unique selectors (expensive) for every node that would otherwise appear in the list.