Proposal for simplifying user interface and test cycle administration

[mcking65]

The pressing need to provide feedback on Isaac's work on the wireframes has led me to think longer and more deeply about our discussions of process for arriving at consensus on results of tests so we can share them in reports. V1 of this UI is going to be built throughout the next 3 weeks. In our April 1 meeting, it seemed there was strong support for:

1. Equivalent results from two testers is a practical minimum for readiness to publish.
2. When there are substantive differences among testers, conversation about resolution should be held publicly on GitHub.

Designing this UI gets into the nitty gritty details of real life for steps 11-13 of the process in our working mode document:

• Step 11: (Tester) Report raw test results.
• Step 12. (Tester, Test admin) Resolve any differences between raw results from different testers.
• Step 13. (Test admin) Report draft test results.

I have been thinking about:

• Making the UI as minimal and simple as possible both for user and budget benefits.
• Keeping the process flexible by minimizing the number of process attributes that are encoded in the system.
• Optimizing for efficiency by:
  • enabling multiple asynchronous ways of working.
  • Minimizing dependency on specific people, especially system admins.

I've some ideas to float with the group. I'll share them in story form to hopefully make them easy to understand.

Scenario 1: Tester1 and tester2 resolve result differences via GitHub and Admin publishes draft report

Tester1 and tester2 are assigned to run the latest test plan for the simple checkbox example with JAWS and Chrome. Let's say that plan has 16 tests. In this scenario they get different results for the first test of 16 and the difference is resolved via discussion in a GitHub issue. Here is how life unfolds for them.

Note: For simplicity, only some system actions are described in this story.

1. Tester1:
   1. logs in, opens the test queue, and sees the checkbox test plan run for JAWS and Chrome with status of not started, awating results from tester1, awaiting results from tester2
   2. Activates the link for the checkbox test plan run.
2. System: Displays page for test 1 in the plan.
3. Tester1: Fills in all required fields for test 1 and presses a button to review results.
4. System: saves test result object, tags test result as raw results complete, displays review dialog.
5. Tester1:
   1. Reads results in dialog, decides they look good.
   2. Presses submit.
6. System: displays page for test 2 in the plan run.
7. Tester1: Repeats for tests 2 through 16 and takes a break.
8. Tester2:
   1. logs in, opens the test queue, sees the checkbox test plan for JAWS and Chrome status as raw results in progress, 50% complete, awaiting raw test results from tester2.
   2. Activates the link for the checkbox test plan run.
9. System: Displays page for test 1 in the plan.
10. Tester2: Fills in all required fields for test 1 and presses a button to review results.
11. System:
    1. saves test result object, tags as raw results complete.
    2. Notices tester 1 saved results for same test in the same test plan with same AT and browser and the test result is also tagged as raw results complete.
    3. Runs comparison, finds 3 differences.
    4. Displays review dialog with note: 3 results are different from results from other testers. Results that are different are marked with an asterisk (*).
    5. Note: tester1 results are not displayed in the review dialog; aiming for simplicity of design here!
12. Tester 2:
    1. Reads dialog, wonders what tester 1 got wrong :-)
    2. Presses button to copy a formatted string containing the differences to clipboard, opens text editor, pastes diff.
    3. Studies differences in editor and decides discussion is needed.
    4. Presses "Create GitHub Issue" button in review dialog.
13. System:
    1. creates GitHub issue with:
       • Title of something like "Differences in results for test_name using AT_name with browser_name
       • Appropriate labels
       • Comment that includes mentions of tester1 and tester2, link to test in the test plan, and results diff, e.g., @tester1 and @tester2 had the following differences in [test_name_link]: test diff formatted with markdown.
       • Assigns issue to tester1 and tester2 (and maybe admins)
    2. tags test result as blocked by issue, adds issue reference to test result.
    3. Opens GitHub issue in new browser tab.
14. Tester 2:
    1. Adds comment to issue with a question for tester1.
    2. Closes browser tab with GitHub issue.
    3. Presses submit in review dialog.
15. System: Displays test 2 in the test plan run.
16. Tester2: completes rest of run and takes a break.
17. Tester1 and Tester2: have discussion with others on GitHub. Consensus is that tester2's interpretation is correct.
18. Tester1: (wants to change result to match results from tester2)
    1. Logs in, opens test queue, and sees the checkbox test plan run for JAWS and Chrome with status of raw results complete and draft report blocked by differences.
    2. Activates link for checkbox test plan run.
19. System:
    1. Loads test 1 page and fills in the form using the test result object saved by tester1.
    2. Notices test results from tester2, runs comparison, adds button to test page for copying differences to clipboard.
    3. Notices blocked by issue tag on tester2 test result and Adds message at top that result reporting is blocked by differences described in issue_link.
    4. Displays the test 1page.
20. Tester1:
    1. Opens GitHub issue in new tab using link in the message at the top of the page, reads, figures out which commands to re-do and how to interpret the output.
    2. Changes fields in form.
    3. Presses review button.
21. System:
    1. Saves data and runs comparison to tester2 test result object for that test, finds results to be equivalent.
    2. Displays review dialog with message at top: Results are equivalent to results from other testers.
    3. Includes submit button labeled "Submit results and close related issue"
22. Tester1: Reads dialog and Presses submit button.
23. System:
    1. changes GitHub issue to closed.
    2. Removes blocked by issue tag from tester2 test result object.
    3. Displays page for test 1; the message at the top confirms the raw test results are complete and all results on file are equivalent.
24. tester1: Feels good; closes browser.
25. Admin: (wants to make results available to AT devs by publishing draft report)
    1. Logs in, opens test queue, sees checkbox test plan for JAWS and Chrome with status of raw results complete.
    2. Activates actions menu and chooses "View results report"
26. System: Displays results report (looks like report in current prototype)
27. Admin: reviews and presses button for "Mark as draft complete"
28. System:
    1. Adds tag for "Draft report complete" to all 32 test result objects created by tester1 and tester2.
    2. Displays test queue page where the checkbox test plan run now shows a status of "draft report complete"
29. Any unauthenticated user: (wants to see draft report)
    1. Opens reports page
    2. Chooses to view draft reports for specific report cycle.
    3. Sees list of all reports that can be built with data tagged as "draft report complete" for the chosen cycle.

Observations

• The resolution conversation was asynchronous and open.
• The admin was not involved.
• The form for creating results is re-used as the form for revising results.
• There is no complex UI for comparing or editing results.
• Diffs can be shared via email, chat, whatever.
• Once initiated, the entire test cycle could be managed from the test queue by extending the types of actions in the actions menu for a test plan run.

Scenario 2: Result differences resolved via GitHub and Admin makes corrections and publishes draft report

This scenario is identical to scenario 1 except that after consensus on how to resolve differences is reached, tester1 is not available to make changes. So, the admin completes the process of editing tester1 results and then making the draft report.

Starting with the admin resolving differences:

1. Admin: (wants to change tester1 result to match results from tester2)
   1. Logs in, opens test queue, and sees the checkbox test plan run for JAWS and Chrome with status of raw results complete and draft report blocked by differences.
   2. Opens actions menu for checkbox test plan run and chooses an admin-only option "Open plan with results from TESTER_NAME". There is a menu item like this visible to the admin for each tester that has saved results for the plan.
2. System:
   1. Loads test 1 page and fills in the form using the test result object saved by tester1.
   2. Notices test results from tester2, runs comparison, adds button to test page for copying differences to clipboard.
   3. Notices blocked by issue tag on tester2 test result and Adds message at top that result reporting is blocked by differences described in issue_link.
   4. Displays the test 1page.
3. Admin:
   1. Opens GitHub issue in new tab using link in the message at the top of the page, reads, figures out which commands to re-do and how to interpret the output.
   2. Changes fields in form.
   3. Presses review button.
4. System:
   1. Saves data and runs comparison to tester2 test result object for that test, finds results to be equivalent.
   2. Displays review dialog with message at top: Results are equivalent to results from other testers.
   3. Includes submit button labeled "Submit results and close related issue"
5. Admin: Reads dialog and Presses submit button.
6. System:
   1. changes GitHub issue to closed.
   2. Removes blocked by issue tag from tester2 test result object.
   3. Displays page for test 1; the message at the top confirms the raw test results are complete and all results on file are equivalent.
7. Admin: (wants to make results available to AT devs by publishing draft report)
   1. Activates button to go back to test queue.
   2. Activates actions menu and chooses option to "Mark draft results complete". (skips actually reviewing the data based on discussions with testers.) Note: this action is only visible to admins.
8. System:
   1. Adds tag for "Draft report complete" to all 32 test result objects created by tester1 and tester2.
   2. Displays test queue page where the checkbox test plan run now shows a status of "draft report complete"

Functions to support other scenarios

Discarding data:

• Data authors and admins have a discard or re-do button on each test page that destroys the data for that test (after confirmation)
• Data authors and admins have a discard results action in the actions menu in the test queue to destroy data for an entire run.

Change/Remove assignment:

• An assigned tester has an action of "Remove me from this test plan run".
• Admins have an action to remove a tester from a test plan run.
• Testers can assign themselves to a plan they are qualified to run simply by opening the plan and responding to a prompt asking if the tester would like to be assigned. They will not get the prompt if they are not qualified based on their user profile.
• Admins have an action to assign another tester to a test plan run. There is no limit to the number of testers that can be assigned to a given test run plan.

With these functions for discarding data and changing assignments, the system can accommodate a wide variety of scenarios, such as:

• If a tester has to leave the project before differences are resolved, discard their data where there are differences and assign another tester.
• Resolve differences by assigning a 3rd tester. If results match results from another tester, discard results from the one that differs from the other 2.
• Train or validate tester skills by having more experienced tester complete a test plan run. Then have the newer tester run through the plan and study differences. The new tester could even copy diffs into email or chat for discussion.

Publishing final reports

To get to the final report, the working mode summary is:

Step 14. (AT developer, optionally Web developer) Review the draft test results.
Step 15. (Community Group) Address any feedback on the draft test results. This can include changing test plans, tests, and re-running tests.
Step 16. (Community Group) Report reviewed test results.

If the basic requirement is the AT developer agrees with the data, one practical path might be:

1. Have an AT Vendor tester role in the system.
2. After test results are in draft state, assign AT Vendor Tester to run test plan. This assignment could be automatic.
3. AT Vendor Testers open test queue and see plans awaiting results from them.
4. If the AT Vendor saves a test result that is equivalent to the draft, make the submit button in the review dialog tag that test result as final.
5. After all tests in a plan have final results, they will show as final on the report page.

Note: the AT Vendor Tester would have the same resolution paths, e.g., create an issue, discuss offline and then come back and revise, etc.

That might work for some AT vendors. Others might just prefer to review the draft report and press some kind of agree button. Both paths would be possible.

Summary

The center-piece of all of this is that:

1. The runner is the only place where test results are entered or revised.
2. The test queue shows the status of a test plan run from not started all the way to final results and appropriate actions can be initiated from there for each plan run.

These 2 ideas should reduce the number of screens, and I believe that will simplify both the experience and the build.

Finally, the quick and dirty V1 way of seeing result differences only in a formatted text string that can be easily copied enables a lot of ways of working (that are fully accessible) and avoids building any kind of diffing/comparing UI. That same string can easily be formatted with markdown and used in the GitHub comments.

[mfairchild365]

I think this is fine for V1 and I don't want my following comments to delay the development of V1. However, I did have some thoughts as I have been processing this. I think it might be worthwhile to take a step back and review how this fits in the process as a whole.

From the working mode document:

Step 14: (AT developer, optionally Web developer) Review the draft test results. (Timebox: TODO)

Having two testers run the same test effectively doubles the effort (and potentially cost) required to produce reports. Is this level of effort and thoroughness necessary in order to produce sufficient results, that will be reviewed yet again by the AT developer? If we have limited resources, and we need to review the test results internally before sending them to an AT developer to review them, is there a more efficient way to review the results? For example, publish a draft of the results and ask the community as a whole to review them? Or perhaps, re-test only a sample of the results to identify where we should focus review and re-testing efforts?

[zcorpan]

Tester 2:

• Reads dialog, wonders what tester 1 got wrong :-)
• Presses button to copy a formatted string containing the differences to clipboard, opens text editor, pastes diff.
• Studies differences in editor and decides discussion is needed.
• Presses "Create GitHub Issue" button in review dialog.

It seems it assumes skills of testers beyond the ability to run tests -- copy and paste into a text editor, understand a text-based diff format, make a decision about whether the difference needs discussion or if they should just change their own result to match that of the other tester. Can we assume those things?

Can the tester choose to ignore the difference and continue to the next test (without filing an issue)? If yes, how do we deal with it?

[mcking65]

The functionality described in this issue, or variations on it, have been implemented in the test runner. Closing the issue.