False negatives definition

[audreymaniez]

In the section 15.2. Accuracy Benchmarking, false positive and negative are defined :

False positives: This is the percentage of test targets, that were failed by the rule, but were not failed by an accessibility expert.

False negatives: This is the percentage of test targets, that were passed by the rule, but were failed by an accessibility expert.

To be consistent with the definition of a false positive, I would say for false negatives : "not passed" instead of "failed" because a test target can be missed by the accessibility expert which will evaluate the rule to "innaplicable" and not "failed" ?

Proposal :

False negatives: This is the percentage of test targets, that were passed by the rule, but were not passed by an accessibility expert.

[WilcoFiers]

@audreymaniez Thanks for the feedback. This is intentional. I would not consider it a false negative if something was passed by a rule and called as inapplicable by an accessibility expert. I don't think that's a noteworthy distinction.

[maryjom]

More to Wilco's point - if an accessibility expert called it inapplicable, that is essentially equivalent to "pass" - which wouldn't be a false negative.

[WilcoFiers]

Discussed during the call today. General consensus seems to be that the way the Accuracy Benchmark section is written up today doesn't quite work. None of the contributors measure accuracy in the way proposed. Instead relying on bug reports to figure out false positives, and in some cases making "beta" versions of rules to try them out before finalising them.

Proposal on the table is to replace this section with a more generic section on how to approach rule quality. This should probably not be part of normative document but be place in notes that can be referenced from the rules format.

[maryjom]

Need a generic section talking about benchmarking, false positives, false negatives. Make it a "SHOULD" and link out to the Review Process document.