Email: Siteimprove Comments on WCAG 3.0 Draft (3 - Conformance testing is difficult to carry out reliably and in a reproducible manner)

[cwadamsoforacle]

Comment from email:
Conformance testing is difficult to carry out reliably and in a reproducible manner
Gregg Vanderheiden expounded in his review on the serious problems that the current draft imposes on conformance testing and test reproducibility. We agree with Vanderheiden’s observation: Qualitative scoring cannot be used for conformance testing; a test must comprise criteria that either fail or pass if said criteria are to be employed in any legislative manner. Otherwise, two testers may produce altogether different results, which is unacceptable for WCAG in its present role in European legilastion.
There is also the problem of scope, which in the current draft is left poorly delineated. Who determines what the processes that should be subjected to testing are? How can one ensure that two similar services or apps or documents when tested by different providers account for similar scope and type of processes or use cases?
Given the taxing nature of the WCAG 3.0 testing effort, as discussed above, one may be tempted to opt for simple processes with less complex user interface components or aspects, and/or limit the number of processes to a minimum – what ever the smallest acceptable number might be – in order to make the testing effort at least somewhat financially feasible. This, then, would further compound the issue of expected lower quality of testing, which we presaged above.
WCAG 3 comments_Siteimprove_Codebusters_April7_2021.docx

[cwadamsoforacle]

Thank you for your comment. Project members are working on your comment. You may see discussion in the comment thread and we may ask for additional information as we work on it. We will mark the official response when we are finished and close the issue.

[cwadamsoforacle]

This issue needs to be split into 2, one covering "Reproducibility of tests", and one for "scope".