From e4084af72355e1de8d2eddc361ba6571bd2ac148 Mon Sep 17 00:00:00 2001 From: Wilco Fiers Date: Sun, 14 Oct 2018 15:56:30 +0200 Subject: [PATCH] CR Proposal (#283) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### Issues resolved by this PR: - Clear split between requirements for rule authors vs. rule implementers : Closes #272 - Need a definition of Test Target : Closes #271 - 16. Rule Aggregation - Add Requirement : Closes #252 - Accessibility Requirements: Updates requested (Shadi) : Closes #250 - 15.2. Accuracy Benchmarking - Update definition : Closes #236 - Outcomes of atomic tests : Closes #235 - clarifying the relationship between rule application and accessibility conformance : Closes #227 - ACT i18n checks : Closes #226 - Clarify the definition of Rule Aggregation : Closes #165 - Clarify how to do "Implementation validation" and "accuracy benchmarking" for manual or semi-automated rules : Closes #163 - Clarify the output format definition : Closes #162 - Mark all Informative sections as such : Closes #280 - "Failed" / "Passed" or "Fail"/"Pass"? : Closes #279 - Require Rule ID's in Atomic Rule List : Closes #261 - Scope sections states ACT rules should not be used for conformance : Closes #281 - Section “13. Accessibility Support” unclear : Closes #221 - Examples need to be identified accessibly : Closes #187 ### Outstanding discussions before CR: - List of features for exit criteria #224 ### Outstanding editorials before CR: - Need non-versioned biblio reference tags #216 *** Preview | Diff --- act-rules-format.bs | 390 +++++++++++++++++++++++++------------------- 1 file changed, 224 insertions(+), 166 deletions(-) diff --git a/act-rules-format.bs b/act-rules-format.bs index 9c9c1aeb..69e34ac2 100644 --- a/act-rules-format.bs +++ b/act-rules-format.bs @@ -1,11 +1,11 @@
 Title: Accessibility Conformance Testing Rules Format
-Shortname: ACT-Format
+Shortname: ACT-Rules-Format
 URL: https://w3c.github.io/wcag-act/act-rules-format.html
 Previous Version: https://w3c.github.io/wcag-act/archive_act-format/2016-12-17.html
 Level: 1.0
-Status: ED
-Group: act-framework
+Status: w3c/ED
+Group: act-rules-format
 Editor: Wilco Fiers, Deque Systems
 Editor: Maureen Kraft, IBM Corp.
 Abstract: The Accessibility Conformance Testing (ACT) Rules Format 1.0 defines a format for writing accessibility test rules. These rules can be carried out fully-automatically, semi-automatically, and manually. This common format allows any party involved in accessibility testing to document and share their testing procedures in a robust and understandable manner. This enables transparency and harmonization of testing methods, including methods implemented by accessibility test tools.
@@ -17,7 +17,7 @@ Introduction {#intro}
 
 There are currently many test procedures and tools available which aid their users in testing web content for conformance to accessibility standards such as the [Web Content Accessibility Guidelines (WCAG)](https://www.w3.org/WAI/standards-guidelines/wcag/) [[WCAG]]. As the web develops in both size and complexity, these procedures and tools are essential for managing the accessibility of resources available on the web.
 
-This format is intended to enable a consistent interpretation of how to test for [accessibility requirements](#structure-accessibility-requirements) and promote consistent results of accessibility tests. It is intended to be used to describe both manual accessibility tests as well as automated tests performed by accessibility testing tools (ATTs).
+This format is intended to enable a consistent interpretation of how to test for [accessibility requirements](#accessibility-requirements) and promote consistent results of accessibility tests. It is intended to be used to describe both manual accessibility tests as well as automated tests performed by accessibility testing tools.
 
 Describing how to test certain accessibility requirements will result in accessibility tests that are transparent, with test results that are reproducible. The Accessibility Conformance Testing (ACT) Rules Format defines the requirements for these test descriptions, known as Accessibility Conformance Testing Rules (ACT Rules).
 
@@ -25,23 +25,19 @@ Describing how to test certain accessibility requirements will result in accessi
 Scope {#scope}
 ==============
 
-The ACT Rules Format defined in this specification is focused on the description of rules that can be used in testing content created using web technologies, such as [Hypertext Markup Language](https://www.w3.org/TR/html/) [[HTML]], [Cascading Style Sheets](https://www.w3.org/TR/CSS/) [[CSS2]], [Accessible Rich Internet Applications](https://www.w3.org/WAI/standards-guidelines/aria/) [[WAI-ARIA]], [Scaleable Vector Graphics](https://www.w3.org/TR/SVG/) [[SVG2]] and more, including digital publishing. The ACT Rules Format, however, is designed to be technology agnostic, meaning that it can conceivably be used to describe test rules for other technologies.
+The ACT Rules Format defined in this specification is focused on the description of rules that can be used in testing content created using web technologies, such as [Hypertext Markup Language](https://www.w3.org/TR/html/) [[HTML]], [Cascading Style Sheets](https://www.w3.org/TR/CSS/) [[CSS2]], [Accessible Rich Internet Applications](https://www.w3.org/WAI/standards-guidelines/aria/) [[WAI-ARIA]], [Scaleable Vector Graphics](https://www.w3.org/TR/SVG/) [[SVG2]] and more, including digital publishing. The ACT Rules Format is designed to be technology agnostic, meaning that it can conceivably be used to describe test rules for other technologies.
 
-The ACT Rules Format can be used to describe ACT Rules dedicated to testing the [accessibility requirements](#structure-accessibility-requirements) defined in Web Content Accessibility Guidelines [[WCAG]], which are specifically designed for web content. Other accessibility requirements applicable to web technologies can also be testable with ACT Rules. For example, ACT Rules could be developed to test the conformance of web-based user agents to the [User Agent Accessibility Guidelines](https://www.w3.org/WAI/standards-guidelines/uaag/) [[UAAG20]]. However, the ACT Rules Format may not always be suitable to describe tests for other types of accessibility requirements.
-
-Because ACT Rules rarely test the entirety of an accessibility requirement, passing ACT rules does not necessarily mean that an accessibility requirement is met. It is important to understand that **ACT Rules test non-conformance** to accessibility requirements. In some cases conformance can be inferred from the absence of failures. Unlike WCAG sufficient techniques, ACT Rules can not be used for conformance claims unless the rule explicitly states it can be used that way. See [Rule Aggregation](#rule-aggregation) for details.
+The ACT Rules Format can be used to describe ACT Rules dedicated to testing the [accessibility requirements](#accessibility-requirements) defined in Web Content Accessibility Guidelines [[WCAG]], which are specifically designed for web content. Other accessibility requirements applicable to web technologies can also be testable with ACT Rules. For example, ACT Rules could be developed to test the conformance of web-based user agents to the [User Agent Accessibility Guidelines](https://www.w3.org/WAI/standards-guidelines/uaag/) [[UAAG20]]. The ACT Rules Format might not always be suitable to describe tests for other types of accessibility requirements.
 
 
 ACT Rule Types {#rule-types}
 ============================
 
-In accessibility, there are often different technical solutions to make the same type of content accessible. These solutions could be tested in a single rule; however, such a rule tends to be quite complex, making it difficult to understand and maintain. The ACT Rules Format solves this by providing two types of rules:
-
-- Atomic rules describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other parts of a web page are to be tested, and when those elements are considered to fail the rule. These rules should be kept small and atomic. Meaning that atomic rules test a single "failure condition", and do so without using results from other rules.
+In accessibility, there are often different technical solutions to make the same type of content accessible. Multiple solutions could be tested in a single rule; however, such a rule tends to be quite complex, making it difficult to understand and maintain. The ACT Rules Format solves this by providing two types of rules:
 
-- Composite rules describe how results from atomic rules should be used to decide if an accessibility requirement was failed. If there are multiple failures necessary for an accessibility requirement to fail, each "type" of failure should be written as separate atomic rules, and the logic for determining the ´pass´ or ´fail´ for the accessibility requirement, across the atomic rules, should be handled by a composite rule.
+- Atomic rules describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other parts of a [=test subject=] are to be tested, and when those elements are considered to fail the rule. These rules are to be kept small and *atomic*. Meaning that atomic rules test a single "failure condition", and do so without using the [=outcome=] of other rules.
 
-The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of `pass` outcomes from atomic rules is sufficient for an accessibility requirement not to `fail`. Not all atomic rules have to be part of a composite rule. Atomic rules only have to be part of a composite rule if failing that atomic rule does not directly fail [Accessibility Requirements](#structure-accessibility-requirements). An atomic rule MAY be part of multiple composite rules.
+- Composite rules describe how the [=outcome=] of multiple [=atomic rules=] can be combined into a single outcome. A composite rule can have multiple "satisfying conditions", each of these provided in a separate atomic rule. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them is used to determine a single outcome.
 
 

Composite rule: Header cells in HTML tables (WCAG 2 Success Criterion 1.3.1).

@@ -52,69 +48,117 @@ The separation between atomic rules and composite rules creates a division of re
  • Header indicated by using the `headers` attribute
  • Header indicated by using ARIA labels
  • -

    If any one of these rules passes, the test result of the composite rule for the table cell is "passed".

    +

    If any one of these rules passes, the outcome of the composite rule for the table cell is "passed".

    +The separation between atomic rules and composite rules can be seen as a division of responsibility. Atomic rules typically test if web content is correctly implemented in a particular solution. Composite rules can test if a combination of outcomes from atomic rules satisfy the accessibility requirement, in part or as a whole. + +Not all atomic rules have to be part of a composite rule. Composite rules are used when the outcome of multiple atomic rules needs to be combined in order to determine (non-)conformance of a test subject to the [Accessibility Requirements](#accessibility-requirements) + ACT Rule Structure {#act-rule-structure} =============================== -ACT Rules MUST be written in an format that conforms to the Web Content Accessibility Guidelines [[WCAG]] or a comparable accessibility guidelines recommendation. ACT test cases are allowed to contain inaccessible content. If any test case contains accessibility issues listed in [WCAG 2.1 Section 5.2.5 Non-Interference](https://www.w3.org/TR/WCAG21/#cc5), users MUST be warned of this in advance. Using an accessibility format supports people with disabilities. It also makes internationalization of ACT rules easier. - -An ACT Rule MUST consist of the following items. +An ACT Rule MUST consist of the following items: * Descriptive Title * [Rule Identifier](#rule-identifier) -* [Rule Description](#structure-description) +* [Rule Description](#rule-description) * [Rule type](#rule-types) -* [Accessibility Requirements](#structure-accessibility-requirements), if any +* [Accessibility Requirements](#accessibility-requirements), if any * [Aspects Under Test](#input-aspects) (for atomic rules) OR [Atomic Rules List](#atomic-rules-list) (for composite rules) * [Applicability](#applicability) * [Expectations](#expectations) -* [Limitations, Assumptions or Exceptions](#structure-limitations-assumptions-exceptions), if any -* [Accessibility Support](#acc-support) (optional) +* [Limitations, Assumptions or Exceptions](#limitations-assumptions-exceptions), if any +* [Accessibility Support](#accessibility-support) (optional) +* [Test Cases](#test-cases) +* ACT Rules Format [=outcome=] definition + +ACT Rules MUST be written in a format that conforms to the Web Content Accessibility Guidelines [[WCAG]] or a comperable comparable accessibility standard. ACT Rule [test cases](#test-cases) are allowed to contain inaccessible content. If any test case contains accessibility issues listed in [WCAG 2.1 Section 5.2.5 Non-Interference](https://www.w3.org/TR/WCAG21/#cc5), users MUST be warned of this in advance. Using an accessibility format supports people with disabilities. It also makes internationalisation of ACT rules easier. + Rule Identifier {#rule-identifier} ================================== -An ACT Rule MUST have a unique identifier that can be any unique text value, such as plain text, URL or a database identifier. +An ACT Rule MUST have a unique identifier that can be any unique text, such as plain text, URL or a database identifier. + +
    +

    Example: ACT Rules may use file names as identifiers. They include a technology directory, followed by a handle that includes an element name or attribute:

    +
      +
    • html+svg/video-alternative
    • +
    • html+svg/meta-no-refresh
    • +
    • html+svg/unique-id
    • +
    +
    -Rule Description {#structure-description} +Rule Description {#rule-description} ========================================= -An ACT Rule MUST have a description that is in plain language and provides a brief explanation of what the rule does. +An ACT Rule MUST have a description that is in plain language, and provides a brief explanation of what the rule does. + +
    + **Example**: This rule checks that the HTML page has a title +
    + +
    + **Example**: The rule checks that for the `html` element, there is no mismatch between the primary language in non-empty `lang` and `xml:lang` attributes, if both are used. +
    -Accessibility Requirements {#structure-accessibility-requirements} -================================================================== +Accessibility Requirements {#accessibility-requirements} +======================================================== -
    - **Editor's note:** The ACT Task Force is looking for feedback about the use of the term "pass" in relation to rules. While rules can "pass", their corresponding accessibility requirement can fail. A section has been added to make this explicit in the rule, but we would like to know if this is sufficient. +Accessibility requirements are just that: A requirement that a particular web page conforms to for it to be considered accessible. A common example of accessibility requirements are the WCAG 2.1 success criteria. Some organizations have additional requirements that come from different sources, such as laws, internal standards, etc. These too are considered accessibility requirements which can be tested using the ACT Rules Format. + + +Accessibility Requirements List {#accessibility-requirements-list} +------------------------------------------------------------------ + +An ACT Rule SHOULD list an identifier and reference for each accessibility requirement that it maps to. + +
    +

    Example: The outcome of this rule affects conformance to the following WCAG 2.1 success criteria:

    +
      +
    • [1.3.1 Info and Relationships](https://www.w3.org/TR/WCAG21/#info-and-relationships)
    • +
    • [4.1.2 Name, Role, Value](https://www.w3.org/TR/WCAG21/#name-role-value)
    • +
    -Accessibility requirements are just that: A requirement that a particular web page must conform to for it to be considered accessible. This can (and usually does) include the WCAG success criteria. Often organizations have additional requirements which may come from different sources, such as local laws, internal standards, etc. These too are considered accessibility requirements and can be tested using the ACT Rules Format. What the precise requirements are for any test is beyond the scope of the ACT Rules Format. +One reason why an [=atomic rule=] might not list accessibility requirements is because it is part of a [=composite rule=], and only a `failed` [=outcome=] in the composite rule would mean not satisfying the accessibility requirement. Another reason might be that an ACT Rule is designed to evaluate a best practice that is not part of any accessibility standard. -[=Atomic rules=] and [=composite rules=] SHOULD identify the accessibility requirements that fail when the outcome of a rule is Fail. An ACT Rule is a complete or partial test for one or more accessibility requirements. This means that most ACT rules test only part of an accessibility requirement, but it MUST NOT test more than the accessibility requirement it lists. -Because ACT Rules often have a smaller scope than the accessibility requirement they test, passing a rule does not necessarily mean that the accessibility requirement has passed. ACT Rules MUST indicate when they can not be used to determine that the accessibility requirement passed. +Accessibility Requirements Mapping {#accessibility-requirements-mapping} +------------------------------------------------------------------------ -Outcomes from an ACT Rule SHOULD be consistent with the accessibility requirement, e.g. a rule only returns the outcome Fail when the content fails the accessibility requirement. This means that the rule maps to the accessibility requirement, as opposed to it merely being related to the requirement, thematically or otherwise. Because of this, atomic rules used in composite rules often do not map to any accessibility requirement. Failing the composite rule fails the accessibility requirement, but failing any of its atomic rules may not. In such cases the atomic rules MUST NOT list the accessibility requirements. These could be provided as background information instead. +If any accessibility requirements are listed, the ACT Rule MUST indicate how each outcome impacts the conformance to the listed accessibility requirements. For WCAG 2.1 Success Criteria, this means indicating whether or not the success criterion can be satisfied, or if further testing is needed. + +
    +

    Example: The outcome of this rule maps to the accessibility requirements in the following manner:

    +
      +
    • `failed`: The success criteria are *not satisfied*
    • +
    • `passed` or `inapplicable`: Further testing is necessary to determine if the success criteria are satisfied
    • +
    +
    + +
    + **Note**: In WCAG 2.1, success criteria do not evaluate to `passed`, `failed` or `inapplicable`. Rather they can be *satisfied* (or not). (See the [WCAG 2.1 definition: satisfies a success criterion](https://www.w3.org/TR/WCAG21/#dfn-satisfies).) If a success criterion is *not satisfied* a web page can only conform if there is a conforming alternative version, as described in [WCAG 2.1 Conformance Requirement 1](https://www.w3.org/TR/WCAG21/#cc1). +
    Aspects Under Test (Atomic rules only) {#input-aspects} ======================================================= -An aspect is a distinct part of the [test subject](#output-test-subject) or its underlying implementation. For example, rendering a particular piece of content to an end user involves multiple different technologies, some or all of which may be of interest to an ACT Rule. Some rules need to operate directly on the [Hypertext Transfer Protocol](https://tools.ietf.org/html/rfc7230) [[http11]] messages exchanged between a server and a client, while others need to operate on the [Document Object Model](https://dom.spec.whatwg.org) [[DOM]] tree exposed by a web browser. Rules may need to operate on several aspects simultaneously, such as both the HTTP messages and the DOM tree. +An aspect is a distinct part of the [=test subject=]. For example, rendering a particular piece of content to an end user involves multiple different technologies, some or all of which can be of use in an [atomic rule]. Some rules need to operate directly on the [Hypertext Transfer Protocol](https://tools.ietf.org/html/rfc7230) [[http11]] messages exchanged between a server and a client, while others need to operate on the [Document Object Model](https://dom.spec.whatwg.org) [[DOM]] tree exposed by a web browser. Rules can operate on several aspects simultaneously, such as both the HTTP messages and the DOM tree. [=Atomic rules=] MUST list the aspects used in the [applicability](#applicability-atomic) and [expectations](#expectations-atomic). Some aspects are already well defined in a formal specification within the context of web content, such as HTTP messages, DOM tree, and CSS styling [[CSS2]]. These do not warrant a detailed description further than a reference to the corresponding section in this specification (see [Common Aspects under Test](https://w3c.github.io/wcag-act/NOTE-act-rules-common-aspects.html)). -For other aspects that are not well defined, an ACT Rule MUST include either a detailed description of the aspect in question or a reference to to a well defined description. +For other aspects that are not well defined, an ACT Rule MUST include either a detailed description of the aspect in question or a reference to a well defined description.
    -

    Test aspects for a rule that checks if images have an accessible name:

    +

    Example: Test aspects for a rule that checks if images have an accessible name:

    • DOM Tree
    • CSS Styling
    • @@ -122,7 +166,7 @@ For other aspects that are not well defined, an ACT Rule MUST include either a d
    -

    Test aspects for a rule that checks if a transcript is available for videos:

    +

    Example: Test aspects for a rule that checks if a transcript is available for videos:

    • DOM Tree
    • CSS Styling
    • @@ -132,8 +176,8 @@ For other aspects that are not well defined, an ACT Rule MUST include either a d
    -

    Test aspects for a rule that checks for use of (language specific) generic link texts like "click here" and "more":

    -
      +

      Example: Test aspects for a rule that checks for use of (language specific) generic link texts like "click here" and "more":

      +
      • DOM Tree
      • CSS Styling
      • Language
      • @@ -144,43 +188,43 @@ For other aspects that are not well defined, an ACT Rule MUST include either a d Atomic Rules List (Composite rules only) {#atomic-rules-list} ========================================================== -A [=composite rule=] uses results from [=atomic rules=] and applies a logic to them so that for each [test target](#output-test-target) a single [outcome](#output-outcome) can be determined. The identifiers of each atomic rules used in the [expectations](#expectations-composed) MUST be listed in the composed rule. The atomic rules list describes the input for composed rules, similar to how [aspects under test](#input-aspects) describe the input for atomic rules. +A [=composite rule=] uses [=outcomes=] from [=atomic rules=] and applies a logic to them so that for each [=test target=] a single outcome can be determined. The identifiers of each atomic rules used in the [expectations](#expectations-composite) MUST be listed in the composite rule. The atomic rules list describes the input for composite rules, similar to how [aspects under test](#input-aspects) describe the input for atomic rules. Applicability {#applicability} ============================== -The applicability describes "what" (parts of) the [test subject](#output-test-subject) should be tested (the test target). +The applicability describes what (parts of) the [=test subject=] are tested. Applicability for Atomic Rules {#applicability-atomic} ------------------------------------------------------ -The applicability section is a required part of an [=atomic rule=]. It MUST contain a precise description of the parts of the [test subject](#output-test-subject) to which the rule applies. For example, specific nodes in the DOM [[DOM]] tree, or tags that are incorrectly closed in an HTML [[HTML]] document. These are known as the [test targets](#output-test-target). The applicability MUST only use information provided through [test aspects](#input-aspects) of the same rule. No other information should be used in the applicability. +The applicability section is a required part of an [=atomic rule=]. It MUST contain a precise description of the parts of the [=test subject=] to which the rule applies. For example, specific nodes in the DOM [[DOM]] tree, or tags that are incorrectly closed in an HTML [[HTML]] document. These are known as the [=test targets=]. The applicability MUST only use information provided through [test aspects](#input-aspects) of the same rule. No other information can be used in the applicability. -Applicability MUST be described objectively, unambiguously and in plain language. When a formal syntax, such as a [CSS selector](https://www.w3.org/TR/selectors-3/) [[css3-selectors]] or [XML Path Language](https://www.w3.org/TR/xpath/) [[XPATH]], can be used, that (part of the) applicability MAY use that syntax in addition to the plain language description. While testing, if nothing within the [test subject](#output-test-subject) matches the applicability of the rule, the outcome is `inapplicable`. +Applicability MUST be described objectively, unambiguously and in plain language. When a formal syntax, such as a [CSS selector](https://www.w3.org/TR/selectors-3/) [[css3-selectors]] or [XML Path Language](https://www.w3.org/TR/xpath/) [[XPATH]], can be used, that (part of the) applicability MAY use that syntax in addition to the plain language description. -An objective description is one that can be resolved without uncertainty in a given technology. Examples of objective properties in HTML are element names, their computed role, the spacing between two elements, etc. Subjective properties on the other hand, are concepts like decorative, navigation mechanism and pre-recorded. Even concepts like headings and images can be misunderstood. For example, describing that the rule examines the tag name, the accessibility role, or the element's purpose on the web page. The latter of which is almost impossible to define objectively. When used in applicability, these concepts MUST have an objective definition. This definition can be part of a larger glossary shared between rules. +An objective description is one that can be resolved without uncertainty, in a given technology. Examples of objective properties in HTML are element names, their computed role, the spacing between two elements, etc. Subjective properties on the other hand, are concepts like decorative, navigation mechanism and pre-recorded. Even concepts like headings and images can be misunderstood. For example, describing that the rule examines the tag name, the accessibility role, or the element's purpose on the web page. The latter of which is almost impossible to define objectively. When used in applicability, potentially ambiguous concepts MUST be defined objectively. This definition can be part of a larger glossary shared between rules.
        -

        The applicability of an atomic rule testing WCAG 2.1 Audio Control:

        +

        Example: The applicability of an atomic rule testing WCAG 2.1 Audio Control:

        Any video or audio element(s) with the autoplay attribute, as well as any object element(s) that is used for automatically playing video or audio when the web page loads.

        -Applicability for Composite Rules {#applicability-composed} +Applicability for Composite Rules {#applicability-composite} ---------------------------------------------------------- -A [=composite rule=] defines how the outcomes from its atomic rules are used to determine a single outcome for each applicable test target. +A [=composite rule=] defines how the [=outcomes=] from rules in its [atomic rules list](#atomic-rules-list) are used to determine a single outcome. -The applicability of a composite rule is defined as the union of all the applicability sections of its [=atomic rules=]. Because of this, applicability of a composite rule can be inferred from the atomic rules it uses results from. Since the applicability can be inferred, rule authors MAY omit applicability for a composite rule. This can be useful if it is difficult to express the combined applicability in plain language. +The applicability of a composite rule is defined as the union of all the applicability sections of its [=atomic rules=]. This can be inferred from the [list of atomic rules](#atomic-rules-list). Rule authors MAY describe the applicability for composite rules. This can be useful if it is difficult to express the combined applicability in plain language. If the composite rule includes applicability, it MUST be the union of all the applicability sections of the atomic rules. Note that atomic rules in a composite rule MAY have different applicability. Because of this, not every element applicable within the composite rule is tested by every atomic rule.
        -

        A composite rule about `img` elements uses results from atomic rules that have the following applicability:

        +

        Example: A composite rule about `img` elements uses results from atomic rules that have the following applicability:

        • Atomic rule1: All `img` element with an `alt` attribute
        • Atomic rule1: All `img` element without an `alt` attribute
        • @@ -195,18 +239,19 @@ Note that atomic rules in a composite rule MAY have different applicability. Bec Expectations {#expectations} ============================== -The expectations describe what the requirements are for the [test targets](#output-test-target) defined in the applicability section. + +The expectations describe what the requirements are for the [=test targets=] derived from the [applicability](#applicability). Expectations for Atomic Rules {#expectations-atomic} ---------------------------------------------------- -An [=atomic rule=] MUST contain one or more expectations. An expectation is an assertion that must be true about each [test target](#output-test-target) (see [Applicability](#applicability)). Each expectation MUST be distinct, unambiguous, and be written in plain language. Unlike in applicability, a certain level of subjectivity is allowed in expectations. Meaning that the expectation has only one possible meaning, but that meaning isn't fully quantifiable. +An [=atomic rule=] MUST contain one or more expectations. An expectation is an assertion that is true about each [=test target=] (see [Applicability](#applicability)). Each expectation MUST be distinct, unambiguous, and be written in plain language. Unlike in applicability, a certain level of subjectivity is allowed in expectations. Meaning that the expectation has only one possible meaning, but that meaning MAY not fully quantifiable. -When all expectations are true for a test target, the test target `passed` the rule. If one or more expectations are false, the test target `failed` the rule. If the atomic rule is used in a [=composed rule=], the composed rule may be `passed` when the atomic rule `failed`, depending on the [applicability](#applicability-composed) and [exxpectations-composed] of the composed rule. +When all expectations are true for a test target, the test target `passed` the rule. If one or more expectations are false, the test target `failed` the rule. When atomic rules are used in composite rules, the outcome of the composite rules can be `passed` when the outcome of individual atomic rule is `failed`, depending on the [expectations](#expectations-composite) of the composite rule.
          -

          A rule for labels of HTML `input` elements may have the following expectations:

          +

          Example: A rule for labels of HTML `input` elements might have the following expectations:

          1. The test target has an accessible name (as described in [Accessible Name and Description: Computation and API Mappings 1.1](https://www.w3.org/TR/accname-aam-1.1/#mapping_additional_nd_te)). [[accname-aam-1.1]]
          2. The accessible name describes the purpose of the test target.
          3. @@ -216,12 +261,12 @@ When all expectations are true for a test target, the test target `passed` the r An atomic rule expectation MUST only use information available in the [test aspects](#input-aspects), from the applicability, and other expectations of the same rule. No other information can be used in the expectation. So for instance, an expectation could be "Expectation 1 is true and ...", but it can't be "Rule XYZ passed and ...". This ensures the rule is encapsulated. -Expectations for Composite Rules {#expectations-composed} +Expectations for Composite Rules {#expectations-composite} -------------------------------------------------------- -A [=composite rule=] MUST contain one or more expectations that describes the logic that is used to determine a single `pass` or `fail` result based on the results of its [=atomic rules=]. An expectation is an assertion, written in plain language, that must be true about the outcomes of [=atomic rules=] listed in [atomic rules list](#atomic-rules-list). A composite rule expectation MUST NOT use information from [test aspects](#input-aspects) or from the [test applicability](#applicability). +A [=composite rule=] MUST contain one or more expectations that describes the logic that is used to determine a single `passed` or `failed` [=outcome=] for each [=test target=], based on the outcomes of [=atomic rules=] listed in [atomic rules list](#atomic-rules-list). -When all expectations are true for a test target, the test target `passed` the rule. If one or more expectations is false, the test target `failed` the rule. This works the same way for atomic rules. +When all expectations are true for a test target, the test target `passed` the rule. If one or more expectations is false, the test target `failed` the rule. This works the same way for atomic rules. A composite rule expectation MUST NOT use information from [test aspects](#input-aspects).

            A composite rule for WCAG 2.1 Audio Description or Media Alternative applies a logic across results from three atomic rules. The expectation of the composite rule is as follows:

            @@ -235,158 +280,171 @@ When all expectations are true for a test target, the test target `passed` the r
            -Limitations, Assumptions or Exceptions {#structure-limitations-assumptions-exceptions} +Limitations, Assumptions or Exceptions {#limitations-assumptions-exceptions} ====================================================================================== -An ACT Rule MUST list any limitations, assumptions or any exceptions for the test, the test environment, technologies being used or the subject being tested. For example, a rule that would partially test WCAG 2.0 Success Criterion 1.4.3 Contrast (Minimum) based on the inspection of CSS properties could state that it is only applicable to HTML text content stylable with CSS, and that the rule does not support images of text. +An ACT Rule MUST list any limitations, assumptions or any exceptions for the evaluation, the test environment, technologies being used or the subject being tested. For example, a rule that would partially test WCAG 2.0 Success Criterion 1.4.3 Contrast (Minimum) based on the inspection of CSS properties could state that it is only applicable to HTML text content stylable with CSS, and that the rule does not support images of text. -Sometimes there are multiple plausible ways that an accessibility requirement can be interpreted. For instance, it is not immediately obvious if emoji characters should be considered "text" or "non-text content" under WCAG 2.0. Whatever the interpretation is, this MUST be documented in the rule. +Sometimes there are multiple plausible ways that an accessibility requirement can be interpreted. For instance, it is not immediately obvious if emoji characters are "text" or "non-text content" under WCAG 2.1. Whatever the interpretation is, this MUST be documented in the rule. -Accessibility Support {#acc-support} +Accessibility Support {#accessibility-support} ==================================== -ACT Rules are designed to test the conformance of content using web technologies to accessibility requirements. However, not every feature of a web technology is implemented in all assistive technologies or user agents that a website may need to support. The concept of [accessibility supported](https://www.w3.org/TR/WCAG20/#accessibility-supporteddef) use of a Web technology is defined in WCAG [[WCAG]]. Because of this, ACT Rules are not necessarily applicable in all test scenarios. For instance, a web page that has to work in assistive technologies that have no WAI-ARIA [[WAI-ARIA]] support, wouldn’t be tested with an ACT Rule that relies on WAI-ARIA support, since this rule could give results that are inconsistent with the accessibility experience of users of those assistive technologies. - -Even within a [=composite rule=], some [=atomic rules=] may not always be applicable. This leaves one fewer solution for passing that particular composite rule. To support users of ACT Rules in properly defining the [accessibility support baseline](https://www.w3.org/TR/WCAG-EM/#step1c) in their test scenarios, an ACT Rule SHOULD include a warning if there are significant accessibility support concerns known about a rule. +Content can be designed to rely on the support for particular accessibility features by different assistive technologies and user agents. For example, content using a particular WAI-ARIA [[WAI-ARIA]] feature relies on that feature to be supported by assistive technologies and user agents. This content would not work for assistive technologies and user agents that do not support WAI-ARIA. WCAG [[WCAG]] provides a definition for [accessibility supported](https://www.w3.org/TR/WCAG20/#accessibility-supporteddef) use of a Web technology. +An ACT Rule SHOULD include known limitations on accessibility support. For example, an atomic rule that checks for a particular accessibility feature that is known to be not widely supported by assistive technologies and web browsers, or a composite rule that includes atomic rules with known accessibility support limitations. -ACT Data Format (Output Data) {#output} -======================================= - -With ACT Rules, it is important that data coming from different sources can be compared. By having a shared vocabulary, accessibility data that is produced by different auditors can be compared and, where necessary, aggregated. Therefore, every ACT Rule MUST express the output in a format that has all of the features described in the ACT Data Format. - -Rules are tested in two steps. Firstly, the applicability is used to find a list of [Test Targets](#output-test-target) (elements, tags or other "components") within the web page or other [test subject](#output-test-subject). Then each test target is tested to see if all of the [expectations](#expectations) are true. This will give the outcome for each test target. For contextual information, the output data must also include [test subject](#output-test-subject) and the [rule identifier](#rule-identifier). - -This will mean that every time a rule is executed on a page, it will return a set with zero or more results, each of which MUST have at least the following properties: - -- [Rule Identifier](#rule-identifier) (test) -- [Test Subject](#output-test-subject) (Web page) -- [Test Target](#output-test-target) (pointer) -- [Outcome](#output-outcome) (`Passed`, `Failed`, or `Inapplicable`) - -
            -Output data using EARL and JSON-LD. (See [Evaluation and Report Language](https://www.w3.org/WAI/standards-guidelines/earl/) [[EARL10-Schema]] and [Java Script Object Notation (JSON)](https://www.json.org).) - -```javascript -{ - "@context": "https://raw.githubusercontent.com/w3c/wcag-act/master/earl-act.json", - "@type": "Assertion", - "subject": "https://example.org/", - "test": "auto-wcag:rules/SC1-1-1-css-image.html", - "result": { - "outcome": "Failed", - "pointer": "html > body > h1:first-child" - } -} -``` +
            + **Note:** WCAG Evaluation Methodology (WCAG-EM) provides guidance on defining an [accessibility support baseline](https://www.w3.org/TR/WCAG-EM/#step1c) for test scenarios, which can help users of ACT Rules to select the appropriate rules for their own circumstance.
            -Test Subject {#output-test-subject} ------------------------------------ - -When a single URL can be used to reference the web page, or other test subject, this URL MAY be used. In scenarios where more complex actions are required to obtain the test subject (in the state that it is to be tested), it is left to ATT developers to determine which method is best used to express the test subject. +Test Cases {#test-cases} +======================== +Test cases are (snippets of) content that can be used to validate the implementation of ACT Rules. Each rule MUST have one or more test cases for `passed`, `failed`, and `inapplicable` [=outcome=]. A test case consists of two pieces of data, a snippet of each [test aspect](#input-aspects) for a rule, and the expected result from evaluating that rule. Test cases serve two functions, firstly as example scenarios for readers to understand when a rule passes, when it fails, and when it is inapplicable. It also serves developers and users of accessibility testing tools and methodologies to validate that a rule is correctly implemented. -Test Target {#output-test-target} ---------------------------------- -When representing the test target in the output data, it is often impractical or impossible to serialize the test target as a whole. Instead of this, a pointer can be used to indicate where the test target exists within the web page or other [test subject](#output-test-subject). There are a variety of pointer methods available, such as those defined in [Pointer Methods in RDF 1.0](https://www.w3.org/TR/Pointers-in-RDF/) [[Pointers-in-RDF]]. - -The pointer method used in the output data of an ACT Rule MUST include the pointer method used in [Test cases](#test-cases). - - -Outcome {#output-outcome} -------------------------- +Update Management {#updates} +============================ -The definition of a rule MUST always result in one of the following outcomes: +Change Log {#updates-changes} +------------------------------------- -- **Passed**: All [expectations](#expectations) for the [Test Target](#output-test-target) were true -- **Failed**: One or more expectations for the Test Target was false -- **Inapplicable**: There were no Test Targets in the [Test Subject](#output-test-subject) +It is important to keep track of changes to the ACT Rules so that users of the rules can understand if changes in test results are due to changes in the rules used when performing the tests, or from changes in the content itself. All changes to an ACT Rule that can change the [=outcome=] of a evaluation MUST be recorded in a change log. The change log can either be part of the rule document itself or be referenced from it. -
            - While *inapplicable* is a valid result for ACT Rules, it may not be a valid result for all [accessibility requirements](#structure-accessibility-requirements). Notably the success criteria of WCAG 2.0 and WCAG 2.1 can only be evaluated to true (passed) or false (failed). This translation of results is part of [rule aggregation](#rule-aggregation) -
            +Each new release of an ACT Rule MUST be identifiable with either a date or a version number. Additionally, a reference to the previous version of that rule MUST be available. For extensive changes, a new rule SHOULD be created and the old rule SHOULD be deprecated. -
            - In addition to `Passed` `Failed` and `Inapplicable`, [[EARL10-Schema]] also defined an `Incomplete` outcome. While this should never be the outcome of a rule when applied in its entirety, it often happens that rules are only partially executed. For example, when applicability was automated, but the expectations have to be evaluated manually. Such "interim" results can be expressed with the "Incomplete" outcome. +
            + **Example**: An example of when a new rule ought to be created is when a rule that tests for the use of a `blink` element changes to instead look for any animated style changes. This potentially adds several new failures that were previously out of scope. Using that same rule as an example, adding an exception to allow `blink` elements positioned off screen can be done by updating the existing rule.
            -Rule Quality Assurance {#quality} -================================= - -Test Cases (Atomic rules only) {#quality-test-cases} ----------------------------------------------------- - -Test cases are (snippets of) content that can be used to validate the implementation of an [=atomic rule=]. They consist of two pieces of data, a snippet of each [test aspect](#input-aspects) for a rule, and the [expected result](#output) that should come from that rule. Test cases serve two functions, firstly as example scenarios for readers to understand when a rule passes, when it fails, and when it is inapplicable. But also for developers and users of automated accessibility test tools to validate that a rule is correctly implemented. - -When executing a test, the test aspect(s), for instance an HTML code snippet, is evaluated by applying the rule's [applicability](#applicability) and [expectations](#expectations). The result is then compared to the expected result of the test case. The expected result consists of a list of [test targets](#output-test-target) and the expected [outcome](#output-outcome) (Passed, Failed, Inapplicable) of the evaluation. - - -Accuracy Benchmarking {#quality-accuracy} ------------------------------------------ - -The web is ever changing, and technologies are used in such diverse and creative ways that it is impossible to predict in advance, all the ways that accessibility issues can occur and all the ways they can be solved for. When writing ACT Rules, it is almost inevitable that exceptions will be overlooked during the design of a rule, or that new technologies will emerge that introduce new exceptions. - -This makes it important to be able to regularly test if the rule has the accuracy that is expected of it. This can be done by benchmark testing. In benchmark testing, the accuracy of a rule is measured by comparing its results to those obtained through accessibility expert testing. - -The accuracy of a rule is the average between the false positives and false negatives, which are in turn calculated as follows: - -- **False positives**: This is the percentage of [test targets](#output-test-target), that were failed by the rule, but were not failed by an accessibility expert. - -- **False negatives**: This is the percentage of test targets, that were passed by the rule, but were failed by an accessibility expert. +Issues List {#updates-issues} +------------------------------------- -There are several ways this can be done. For instance, accessibility test tools can implement a feature which lets users indicate that a result is in error, or pages that for which accessibility results are known, can be tested using ATT, and the results are compared. To compare results from ACT Rules to those of expert evaluations, [data aggregation](#output-aggregation) may be necessary. +An ACT Rule MAY include an issues list. When included, the issues list MUST be used to record cases when the [=outcome=] of an ACT Rule was `failed` where it ought to have been `passed`, or vice versa. There are several reasons why this might occur, including: +- Certain scenarios or the use of technologies that are very rare and were not included in the rule for that reason. +- Certain accessibility features are impossible to test within the test environment. For instance, they might only be testable by accessing the accessibility API, require screen capturing, etc. +- The scenario did not exist (due to changing technologies) or was overlooked during the initial design of the rule. -Rule Aggregation {#rule-aggregation} -====================================== +The issues list serves two purposes. For users of ACT Rules, the issues list gives insight into why an inaccurate result occurred, as well as provide confidence in the result of that rule. For the designer of the rule, the issues list is also useful to plan future updates to the rule. In a new version of the rule, resolved issues would be moved to the change log. -As described in section [[#output]] a rule will return a list of results, each of which contain 1) the [Rule ID](#rule-identifier), 2) the [test subject](#output-test-subject), 3) the [test target](#output-test-target), and 4) an [outcome](#output-outcome) (Passed, Failed, Inapplicable). Data expressed this way has a great deal of detail, as it gives multiple pass / fail results for each rule. -Most expert evaluations do not report results at this level of detail. Often reports are limited to giving a single outcome (Passed, Failed, Inapplicable) per page, for each success criteria (or other accessibility requirement). To compare the data, results from rules can be combined, so that they are at the same level. +Rule Accuracy {#accuracy} +========================= -When all rules pass, that does not mean that all [accessibility requirements](#structure-accessibility-requirements) are met. Only if the rules can test 100% of what should be tested, can this claim be made. Otherwise the outcome for a criterion is inconclusive. +This section is *non-normative*. -**Example:** An expert evaluates a success criterion to fail on a specific page. When testing that page using ACT Rules, there are two rules that map to this criterion. The first rule returns no results. The second rule finds 2 test targets that pass, and a 3rd test target that fails. +While [test cases](#test-cases) can be used to determine if an ACT Rule was correctly implemented, they do not guarantee that implementations will never produce incorrect results. When writing ACT Rules, it is almost inevitable that edge cases will be overlooked. Technologies are always evolving, and content authors are constantly coming up with new and unexpected ways to use those. -In this example, the first rule is inapplicable (0 results), and the second rule has failed (1 fail, 2 pass). Combining this inapplicable and fail, means the success criterion has failed. +There are two types of inaccuracies that can produce incorrect results. Inaccuracies in the **implementation** can be addressed with test cases, but inaccuracies in the **ACT Rule** itself can not. After all, rule inaccuracies come from the rule author being unaware of a particular edge case. -See [[#appendix-data-example]] on how this could be expressed using JSON-LD and EARL. +When a test result incorrectly indicates non-conformance to an accessibility requirement, this is known as a false positive. Opposite, when a rule incorrectly indicates conformance, this is a false negative. A percentage of false positives and false negatives can be calculated by comparing it to results from an accessibility audit: +- **False positives**: This is the percentage of [=test subjects=], that were `failed` by the rule, but conform to the [accessibility requirements](#accessibility-requirements). -Update Management {#updates} -==================================== +- **False negatives**: This is the percentage of [=test subjects=], that were `passed` by the rule, but were non-conformant to the [accessibility requirements](#accessibility-requirements). -Change Log {#updates-changes} -------------------------------------- +The possibility of false positives and false negatives with ACT rules mean they will likely require ongoing maintainence. Designing a process for maintaining ACT rules is outside the scope of the ACT Rules Format, which is limited to the rules themselves. Neverthless, it is suggested that rule authors work out a process for maintaining their rules. -It is important to keep track of changes to the ACT rules so that users of the rules can understand if changes in test results are due to changes in the rules used when performing the tests, rather than changes in the content itself. All changes to an ACT Rule that can change the [outcome](#output-outcome) of a test MUST be recorded in a change log. The change log can either be part of the rule document itself or be referenced from it. -Each new release of an ACT Rule MUST be identifiable with either a date or a version number. Additionally, a reference to the previous version of that rule MUST be available. For extensive changes, a new rule SHOULD be created and the old rule SHOULD be deprecated. +Harmonization {#harmonization} +============================== -**Example:** An example of when a new rule should be created is when a rule that tests for the use of a `blink` element changes to instead look for any animated style changes. This potentially adds several new failures that were previously out of scope. Using that same rule as an example, adding an exception to allow `blink` elements positioned off screen should be done by updating the existing rule. +This section is *non-normative*. +While the ACT Rules Format is designed to stimulate harmonization, there are no direct requirement in the ACT Rules Format that prevent a rule author from writing rules inconsistent with already established ACT Rules. Neither are there requirements for ACT Rules to have a certain number of implementations, or to have a certain level of accuracy. This allows quality requirements to be different for different user groups, and allows them to develop over time. -Issues List {#updates-issues} -------------------------------------- +Harmonization occurs when a group of rule implementors collectively accept the validity of an ACT Rule. For example, a community group of accessibility testing tool vendors could declare they've harmonized on a particular set of ACT Rules. Such a group can set acceptance criteria for rules, and have quality requirements that go beyond the ACT Rules Format. For example: -An ACT Rule MAY include an issues list. When included, the issues list MUST be used to record cases in which the ACT Rule might return a failure where it should have returned a pass or vice versa. There are several reasons why this might occur, including: +
            +

            Example: The ACT EPUB group might have the following acceptance criteria:

            +
              +
            • An ACT EPUB Rule is harmonized when it was approved members of 3 organisations, AND
            • +
            • An ACT EPUB Rule is harmonized when it has 2 independent implementations
            • +
            +
            -- Certain scenarios or the use of technologies that are very rare and were not included in the rule for that reason. -- Certain accessibility features are impossible to test within the test environment. For instance, they might only be testable by accessing the accessibility API, require screen capturing, etc. -- The scenario did not exist (due to changing technologies) or was overlooked during the initial design of the rule. +An example of such a process is the [WCAG ACT Review Process](https://w3c.github.io/wcag-act-rules/review-process.html). + + +Definitions {#definitions} +========================== + +
            +
            Test Subject
            +
            +

            A resource or collection of resources that can be evaluated by an ACT Rule.

            +
            +

            Example:

            +
              +
            • An HTML page, including all embedded scripts, style and images
            • +
            • An EPUB document
            • +
            • A Vue component file
            • +
            +
            +
            +

            Note: Implementers using the [[EARL10-Schema]] can express the test subject with the [subject property](https://www.w3.org/TR/EARL10-Schema/#subject)

            +
            +
            + +
            Test Target
            +
            +

            A distinct part of the [=test subject=].

            +
            +

            Example:

            +
              +
            • Nodes within an HTML page
            • +
            • A period of time within a video
            • +
            • A range of characters within a text document
            • +
            +
            +
            +

            Note: Implementers using the [[EARL10-Schema]] can express the test target with the [pointer property](https://www.w3.org/TR/EARL10-Schema/#pointer)

            +
            +
            + +
            Outcome
            +
            +

            One of three types of conclusions that come from evaluating an ACT Rule on a [=test subject=]. An outcome can be:

            +

              +
            • **Inapplicable**: No part of the test subject matches the applicability
            • +
            • **Passed**: The test subject meets all expectations
            • +
            • **Failed**: The test subject does not meet all expectations
            • +
            +
            +

            Note: While *inapplicable* is a valid outcome for ACT Rules, it might not be a valid result for all [accessibility requirements](#accessibility-requirements). Notably the success criteria of WCAG 2.0 and WCAG 2.1 can only be evaluated to satisfied (passed and inapplicable) or not satisfied (failed).

            +
            +
            +

            Note: Implementers using the [[EARL10-Schema]] can express the outcome with the [outcome property](https://www.w3.org/TR/EARL10-Schema/#outcome). In addition to `passed` `failed` and `inapplicable`, EARL 1.0 also defined an `incomplete` outcome. While this can not be the outcome of an ACT Rule when applied in its entirety, it often happens that rules are only partially evaluated. For example, when applicability was automated, but the expectations have to be evaluated manually. Such "interim" results can be expressed with the `incomplete` outcome. +

            +
            +
            + +Appendix 1: Expressing ACT Rule results with JSON-LD and EARL {#appendix-data-example} +================================================================================= -The issues list serves two purposes. For users of ACT Rules, the issues list may give insight into why an inaccurate result occurred, as well as provide confidence in the result of that rule. For the designer of the rule, the issues list is also useful to plan future updates to the rule. In a new version of the rule, resolved issues would be moved to the change log. +This section is *non-normative*. +**Example:** Output data using EARL and JSON-LD. (See [Evaluation and Report Language](https://www.w3.org/WAI/standards-guidelines/earl/) [[EARL10-Schema]] and [Java Script Object Notation (JSON)](https://www.json.org).) -Appendix 1: Aggregation examples, using JSON-LD and EARL {#appendix-data-example} -================================================================================= +```javascript +{ + "@context": "https://raw.githubusercontent.com/w3c/wcag-act/master/earl-act.json", + "@type": "Assertion", + "subject": "https://example.org/", + "test": "auto-wcag:rules/SC1-1-1-css-image.html", + "result": { + "outcome": "earl:failed", + "pointer": "html > body > h1:first-child" + } +} +``` -**Example:** +**Example:** ```javascript { @@ -395,17 +453,17 @@ Appendix 1: Aggregation examples, using JSON-LD and EARL {#appendix-data-example "subject": "https://example.org/", "test": "auto-wcag:SC1-1-1-css-image.html", "result": { - "outcome": "Failed", + "outcome": "earl:failed", "source": [{ "test": "auto-wcag:SC1-1-1-css-image.html", "result": { - "outcome": "Failed", + "outcome": "earl:failed", "pointer": "html > body > h1:first-child" } }, { "test": "auto-wcag:SC1-1-1-css-image.html", "result": { - "outcome": "Passed", + "outcome": "earl:passed", "pointer": "html > body > h1:nth-child(2)" } }] @@ -413,7 +471,7 @@ Appendix 1: Aggregation examples, using JSON-LD and EARL {#appendix-data-example } ``` -**Example:** Aggregate rules to a WCAG success criterion +**Example:** Aggregate outcomes to a WCAG success criterion ```javascript {