Tester issue report for: "Navigate to an unchecked checkbox in reading mode"

[jha11y]

Test file at exact commit

tests/checkbox/test-01-navigate-to-unchecked-checkbox-reading.html

Cycle:

Test Pilot (2020-05-27)

AT:

JAWS (version 2020.2004.66 - April 2020)

Browser:

Chrome (version 83.0.4103.61)

Description

Type your description here.

Conflicts with other results

Difference 1 - Unexpected behavior when testing X / Shift+X

• Your unexpected behaviors: Group Label not announced (for output "Lettuce �check box� �not checked� ")
• Other unexpected behaviors: No unexpected behaviors recorded (for output "check box not checked Lettuce")

Difference 2 - Unexpected behavior when testing X / Shift+X

• Your unexpected behaviors: Group Label not announced (for output "Lettuce �check box� �not checked� ")

• Other unexpected behaviors: No unexpected behaviors recorded (for output "check box not checked Lettuce")

• Other unexpected behaviors: No unexpected behaviors recorded (for output "Lettuce �check box� �not checked�")

Difference 3 - Unexpected behavior when testing F / Shift+F

• Your unexpected behaviors: Group label not announced (for output "Lettuce �check box� �not checked� ")
• Other unexpected behaviors: No unexpected behaviors recorded (for output "check box not checked Lettuce")

Difference 4 - Unexpected behavior when testing F / Shift+F

• Your unexpected behaviors: Group label not announced (for output "Lettuce �check box� �not checked� ")

• Other unexpected behaviors: No unexpected behaviors recorded (for output "check box not checked Lettuce")

• Other unexpected behaviors: No unexpected behaviors recorded (for output " Lettuce �check box� �not checked�")

Difference 5 - Unexpected behavior when testing Left Arrow / Right Arrow (with Smart Navigation on)

• Your unexpected behaviors: order of announcement of role name and state is reversed from other tests which may be confusing (for output "�check box� �not checked� Lettuce")
• Other unexpected behaviors: No unexpected behaviors recorded (for output "check box not checked Lettuce")

Difference 6 - Unexpected behavior when testing Left Arrow / Right Arrow (with Smart Navigation on)

• Your unexpected behaviors: order of announcement of role name and state is reversed from other tests which may be confusing (for output "�check box� �not checked� Lettuce")

• Other unexpected behaviors: No unexpected behaviors recorded (for output "check box not checked Lettuce")

• Other unexpected behaviors: No unexpected behaviors recorded (for output "�check box� �not checked� Lettuce")

[jha11y]

@jongund @jscholes can we figure out a way to resolve these conflicts?

[jscholes]

@jha11y Sorry, I'm not sure I understand what this issue is reporting, nor what I'm being asked to comment on.

[jha11y]

@jscholes these are the conflicts for the checkbox test that you,me and @jongund complete. Oddly it did not include the names of who had which conflict in this generated ticket. I'll update, but you can also look at your test results and it should show the conflicts

[jscholes]

The test description and included assertions prompted me to indicate whether the role, name and state were correctly conveyed (which they were), and made no mention of control grouping or speech ordering. I specifically only tested for those things explicitly prompted by each test (and control grouping was included in follow-up tests).

So in this case, the conflicts seem to have been caused by some testers including additional info for which they weren't prompted. It's not possible for each tester to guess at the myriad of other things other testers may think worth mentioning, and I think this highlights the importance of both:

• only responding to the assertions and instructions which have been specifically provided; and
• ensuring that each test is as explicit as possible about which aspects each tester should focus and comment on.

Please let me know if I'm reading this situation incorrectly. It would only be within my power to resolve the above conflicts by re-testing and agreeing with the above assertions, which I don't think is appropriate given that this test's materials make no mention of grouping or speech ordering.

[jscholes]

Additional note: while control grouping was the subject of some tests, speech ordering was never mentioned. E.g. it was asserted that a screen reader should identify a check box which was not checked and had a name of "Lettuce", but not in what order it would make sense for that info to be spoken or whether the order should or should not be consistent across navigation paradigms. I do think this could be a potential area to explore in the future though.

[jscholes]

For reference, here are the three assertions explicitly highlighted for this test:

• Role 'checkbox' is conveyed
• Name 'Lettuce' is conveyed
• State of the checkbox (not checked) is conveyed

[mcking65]

@jscholes wrote:

The test description and included assertions prompted me to indicate whether the role, name and state were correctly conveyed (which they were), and made no mention of control grouping or speech ordering. I specifically only tested for those things explicitly prompted by each test (and control grouping was included in follow-up tests).

James, Your interpretation of the test and assertions is exactly what we expected.

So in this case, the conflicts seem to have been caused by some testers including additional info for which they weren't prompted. It's not possible for each tester to guess at the myriad of other things other testers may think worth mentioning, and I think this highlights the importance of both:

• only responding to the assertions and instructions which have been specifically provided; and
• ensuring that each test is as explicit as possible about which aspects each tester should focus and comment on.

Exactly right.

It would only be within my power to resolve the above conflicts by re-testing and agreeing with the above assertions, which I don't think is appropriate given that this test's materials make no mention of grouping or speech ordering.

The unexpected results related to grouping and announcement order are not relevant to this test, so you should leave your results as they are.

Additional note: while control grouping was the subject of some tests, speech ordering was never mentioned. E.g. it was asserted that a screen reader should identify a check box which was not checked and had a name of "Lettuce", but not in what order it would make sense for that info to be spoken or whether the order should or should not be consistent across navigation paradigms. I do think this could be a potential area to explore in the future though.

The assertions are worded as they are to allow screen readers flexibility in the way they choose to present information. If we were to assert that a specific order were required, we would likely get way too close to dictating exactly how a screen reader should present a UI. Our goal is to ensure accurate and reliable rendering. We should avoid crossing the line into judgment of the usability of the rendering. In some cases, order could effect accuracy, and that would be worth highlighting if that were the case.

My view is that the results related to unexpected behaviors should be removed in order to resolve the conflicts.

I also see that the information from the tool about conflicts is quite inadequate. Fixing this will be a priority for the next version.

[jha11y]

Ok. I will delete any comments made that go beyond the assertions.

I guess this would be something we would want to make clear to future testers :-)