core(config): Re-weight a11y scores based on severity and frequency

[robdodson]

Here's a stab at re-weighting the a11y tests, accounting for severity and frequency. This isn't a super scientific process but I'll try to explain my methodology.

For each weight I tried to consider two factors:

• How common is the issue?
• Could this completely break an experience or is it just really annoying?

Because we don't have the ability to identify non-applicable tests yet, it means that there are some items which are uncommon but should be fatal. For example, a video without captions is pretty bad and should really sting, but if we weigh it too heavily, it's like an automatic free 10 points to the 99.9% of sites which don't use <video>. In these cases I had to stick to a middle-of-the-road weight so they still hurt if they actually occur, but they also don't give too much of a free boost to everybody else. Fixing dequelabs/axe-core#473 will help a lot here.

I tested the new weights against 3 sites. I don't want to publicly shame anyone here so I'll just call them "Good site", "OK site", and "Bad site". Here's a before and after of their scores:

	Before	After
Good Site	91	89
OK Site	86	73
Bad Site	77	54

New weights are as follows:

audit	failure rate		axe severity	subjective weight 1
color-contrast	73.21%	▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅	major	6
link-name	66.23%	▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅	critical	9
image-alt	50.59%	▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅	critical	8
label	47.54%	▅▅▅▅▅▅▅▅▅▅▅▅▅▅	critical	10
html-has-lang	36.58%	▅▅▅▅▅▅▅▅▅▅	major	4
frame-title	34.07%	▅▅▅▅▅▅▅▅▅▅	critical	5
duplicate-id	32.38%	▅▅▅▅▅▅▅▅▅	critical	5
meta-viewport	31.82%	▅▅▅▅▅▅▅▅▅	critical	3
button-name	18.40%	▅▅▅▅▅	critical	10
list	10.71%	▅▅▅	critical	5
bypass	7.06%	▅▅	critical	10
listitem	4.82%	▅	major	4
tabindex	4.48%	▅	critical	4
aria-valid-attr-value	4.37%	▅	critical	5
aria-required-children	3.82%	▅	critical	5
input-image-alt	2.01%		critical	1
document-title	1.87%		major	2
aria-allowed-attr	1.69%		critical	3
definition-list	1.51%		major	1
object-alt	1.39%		critical	4
meta-refresh	0.79%		critical	1
aria-roles	0.64%		major	3
html-lang-valid	0.49%		major	1
aria-required-parent	0.42%		critical	2
dlitem	0.35%		major	1
aria-required-attr	0.28%		critical	2
accesskeys	0.17%		critical	1
aria-valid-attr	0.14%		critical	2
valid-lang	0.11%		major	1
video-description	0.01%		major	3
td-headers-attr	0.01%		critical	1
video-caption	0.01%		critical	4
layout-table	0.00%		major	1
audio-caption	0.00%		critical	4
th-has-data-cells	0.00%		major	1

[patrickhulce]

nice, love the data driven decision here! these all seem reasonable to me, seems like the best we can do in absence of applicability tests 👍

mind rebasing and double-checking we don't have any tests dependent on the current weights since its failing travis atm?

[brendankenny]

nice!

[paulirish]

wfm. your call here. :)

[robdodson]

Cool thanks for looking it over folks 😁

…

On Tue, Oct 10, 2017, 6:27 PM Paul Irish ***@***.***> wrote: Merged #3515 <#3515>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3515 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBFDb29uOUmzMw0DoR40VTdQDfQayWQks5srBl3gaJpZM4PzG34> .