Define assertion verdicts in glossary

[mcking65]

The system currently records "pass" and "fail" for all assertions regardless of priority (MUST, SHOULD, or MAY).

Vispero recommended that "MAY" assertions have a verdict of "Supported" or "Unsupported".

The glossary does not currently define terminology for verdicts.

Decide whether to have different verdicts based on priority and formalize the definition of the verdictsin the glossary.

[css-meeting-bot]

The ARIA-AT Community Group just discussed Define assertion verdicts.

The full IRC log of that discussion

<jugglinmike> topic: Define assertion verdicts
<jugglinmike> github: https://github.com//issues/1050
<jugglinmike> Matt_King: Vispero believes that it sounds odd to call the result of a "MAY" assertion as a "pass" or a "fail"
<jugglinmike> Matt_King: I think I agree. We're interested in defining some terminology for how we describe when ATs do or don't "check the box" when it comes to "MAY" assertions
<jugglinmike> Matt_King: I've suggested "supported" and "not supported" instead of "passed" and "failed"
<jugglinmike> Matt_King: When it comes to "SHOULD" assertions, it seems like everyone is satisfied with the terms "pass" and "fail"
<jugglinmike> Hadi: That sounds good to me
<jugglinmike> James_Scholes: when it comes to "SHOULD", I find myself flip-flopping between different viewpoints
<jugglinmike> Joe_Humbert: That sounds good to me
<jugglinmike> Murray_Moss: Me, too
<jugglinmike> LolaOdelola: My question is more about the use of "MAY"/"SHOULD"/"MUST" in general, and I know that conversation has already happened, so I don't want to re-tread those points
<jugglinmike> Joe_Humbert: I'd like to get feedback from developers and see if the framing helps them understand how well the different patterns are supported by the different ATs
<jugglinmike> LolaOdelola: Yeah! I also wonder if there's a bit of a conflict between what developers want and what AT vendors want
<jugglinmike> LolaOdela: I also wonder how many developers would even be familiar with these more standards-facing terms like "SHOULD"
<jugglinmike> s/LolaOdela/LolaOdelola/
<jugglinmike> Matt_King: This is really for AT developers. I think web authors are unlikely to drill down into this level of granularity; they will probably be reviewing the test plan reports at the top level
<jugglinmike> Hadi: regarding whether it is geared toward AT developers or web developers: while I see occasionally that some web developers are interested in seeing this level of granularity, I think that overall, they shouldn't have to be concerned about specific features
<jugglinmike> Hadi: I think they should just worry about implementing the APG and considering that as sufficient
<jugglinmike> Joe_Humbert: I agree that would be ideal, but I think that practically speaking, web developers have to be able to prove certain things in certain situations (e.g. when it comes to accessibility compliance)
<jugglinmike> Matt_King: We are planning to write a detailed explanation about how to interpret the reports. This will certainly be covered there, too
<jugglinmike> Joe_Humbert: That eases my concerns significantly
<jugglinmike> Zakim, end the meeting

[mcking65]

For July 3, 2024 meeting, call for consensus.

Definition of assertion verdict

A judgement of whether an assistive technology exhibits the expected behavior defined by an
assertion
when a
test
that includes the assertion is performed. The term used to express the judgement depends on the
assertion priority
as follows:

Assertion Priority	Behavior is Exhibited	Behavior is Not Exhibited
Must	Passed	Failed
Should	Passed	Failed
May	Supported	Unsupported

App Changes

Update language in tables that show command results in report pages, completed tests in test runner, and results in candidate review.

[mcking65]

I added the definition of assertion verdict to the glossary.

[mcking65]

Raised issue w3c/aria-at-app#1145. That issue will track remaining implementation work. Closing this issue as complete.