How strict should our test cases be?

[WilcoFiers]

We recently added a few test cases in Handling Presentational role conflict resolution. This has caused our implementation numbers to drop significantly.

As product owner, this isn't high on my list of priorities. I've never once heard that missing support for this edge case caused a problem. My only motivation to want to implement it comes from it being a requirement for an ACT rules.

With my chair hat back on, that feels like we've overstepped our bounds. There are much more pressing issues to be resolved than check to make sure every implementor understands conflict resolution. I came across a similar issue this week, one we haven't addressed yet. I've been hesitant to even raise it because I don't think it's something we should invest time in. (The issue was that we're not taking presentational children into account anywhere.)

Another example of this is in the color-contrast rule that I wrote. I added an example where text shadow is used to up the contrast ratio just above the minimal requirements. This situation is very uncommon. Unlike the previous one I do know this occasionally causes false positives, but again it is very very rare, and by no means trivial to support.

I want us to have a conversation about how we draw the line when it comes to adding test cases.

[patrickhlauke]

is this more a concern that this may make tools look unnecessarily bad/worse due to edge cases, rather than a completeness of cases?

i think there's value for human auditors to know about edge cases and to have some (semi-official/offical) rule/interpretation of that edge case, as those are the situations that cause the most puzzlement with auditors.

[WilcoFiers]

The primary purpose of test cases is to check that implementations are sufficiently consistent. So the more prescriptive we are with test cases, the slower we'll get implementors for them. That in turn slows down the rate at which we can hand these rules off to the W3C for publication.

I think that being too prescriptive slows down the process of publishing rules. That's my main concern here. There are still a significant number of gaps in our ruleset. I think our time is better spent building that out, getting adoption at a higher level, and not focus too much on these edge cases until we're further along getting more implementors implementing more rules.

The edge cases will still be described in the rules, they just won't be covered by test cases.

[dd8]

Could be difficult to avoid dealing with presentation conflict issues unless test cases using role=none or role=presentation are completely removed - see #1308

Otherwise implementations that handle the edge case would return different results to implementations that don't. It would be undesirable to penalise implementations that handle the edge cases.

[WilcoFiers]

There's some granularity in this which I didn't think of when I opened this issue. We could have test cases that can lead to a "partially consistent" implementation if something is missing, but not an inconsistent, by only testing for these things in failed examples. This happened in the button rule: https://act-rules.github.io/rules/97a4e1#failed-example-5

Axe-core here doesn't have conflict resolution implemented yet. Because there are only failed examples of this, it is now set to partially-consistent.

[dd8]

It looks like role=none or role=presentation is used in lots of inapplicable test cases, but sometimes gets over-ridden by presentation conflict resolution, so some of these inapplicable test cases are actually failures.

IMO it would be better to leave in test cases showing conflict resolution (and fix the broken test cases) to document something that's not obvious and hard to get right (see #1308 for existence proof of this)

Reverting the PR that added conflict resolution test cases won't fix the problem due to the widespread use of role=none in pre-existing examples.

[Jym77]

Some thoughts / follow-up to yesterday's CG call…
(and you already know it is going to be one of these long messages I'm always writing…)

---

Whatever we do, one clear thing is that we should not "punish" tools that implement the specs in details.
Taking the example of Presentational roles conflict resolution, we had a bunch of Inapplicable Examples like <button role="none">Foo</button> where a tool that did implement the specs would find it applicable and thus not get a perfect "grade" on the implementation of that rule.
I think it is pretty important to avoid these situations and to make sure that the tools following the specs don't get "punished" by rule who doesn't.

On the other hand, "punishing" tools that don't follow the specs is less problematic. We can easily argue that these tools are not "perfect"… Of course, depending on the widespreadness of the corner case we talk, we might nonetheless want to avoid that.

---

As mentioned during the call, it could be a good idea to put the "corner case" examples in Failed test cases rather than in Passed/Inapplicable ones. This way, a tool not implementing these will end up being a "partial implementation" which is better than not being an implementation at all…

And I would argue "partial implementation" is the correct term in such a case. A tool that ignores shadow DOM and only look into the light DOM is not really implementing a rule with targets both in light and shadow trees. I think it is correct to say such a tool is only a partial implementation of the rule.

---

There is something blurry about the acceptance of the part of the specs we talk about.
Pushing it to the extreme, if a tool was only working for HTML4, not HTML5, we probably wouldn't discuss whether we should move all our examples to HTML4 in order to count this as an implementation. We'd probably conclude that said tool should just upgrade to HTML5.
Or if a tool was ignoring aria-labelledby we probably wouldn't be talking about whether or not we should remove them and count the tool as an implementation.

That illustrate the blurry line where some parts of the specs, we all agree that they are needed in order to be an implementation, and some we may have diverging opinions. That is fine, but everybody is going to put the line in a different place and thus without any objective criterion we'll end up in endless discussions in some cases…

---

Of course, related to this is the browser support of the feature. Something (like HTML5 or aria-labelledby) that is well supported by every single browser and AT is definitely something we should follow. Something where support is flaky or inconsistent is more open to discussion.
This gets close to the recurring question of "should we write rules according to specs or according to what browsers do?" My position on this question remains the same: we should follow the specs, we can allow for some divergence but (back to the first point) make sure that we never "punish" tools that follow the specs.

---

Leading to the main question of which position we (the CG) should hold…
I think that we should spearhead the specs and support them fully rather than stay in the back. We are an emanation of the W3C and it would feel weird to send a message of "W3C specs say to do so, but if you do it differently it's totally fine"…

We've been spearheading specs in some domains such as adapting our examples to ARIA 1.2. This was done without "breaking" ARIA 1.1 (so examples are compatible with both), but still, we moved forward ahead of the rest and are ready for ARIA 1.2 which is "only" a working draft.

---

TL; DR
So, my position is that the CG should spearhead the adoptions of the specs.
We can try and do so by putting the more controversial examples only as Failed Examples in order to have the tools ignoring them as "partial implementations". This is however going to end up in many discussions of whether such spec is concerned or not…
We should absolutely make sure that tools which do respect the specs can get "full marks" on the rule.