Usability Test Quality Heuristics

The following indicators can help determine if a usability test will produce useful results. This is not an exhaustive list, but it should be helpful in planning and assessing usability tests.

Study design

Study purpose

Good: Researcher clearly articulates the purpose of the usability test (for example, as a specific question to answer, an area of inquiry, etc.).

Bad: Researcher does not specify a purpose for the test, or the purpose specified is very broad, like “testing the app” or “finding problems.”

Number of participants

Good: Tests include a sufficient number of participants necessary to see patterns, and there’s only one participant per session.

Bad: Studies includes only one participant, or studies include multiple participants tested simultaneously (like a focus group).

Sampling bias

Good: Participants are people who actually use the application (or do the task the application is supposed to support).

Bad: Participants are experts (vs. average users), friends, family, or close colleagues of the product team.

Moderator style

Question style

Good: Researcher asks follow up questions that are open-ended and non-leading.

Bad: Questions are leading or subtly suggest potential solutions or hypotheses.

Task variability

Good: Researcher asks participants to complete similar tasks related to the study’s purpose.

Bad: Researcher asks participants to complete different tasks, or tasks unrelated to the study’s purpose.

Framing

Good: Researcher asks participants to complete tasks with the product or service that align with their work-related goals.

Bad: Researcher asks participants to complete tasks unrelated to their work-related goals. For example, asking a participant how they might send a fax when their job doesn’t call for that.

Priming

Good: Moderator asks participants to complete tasks without indicating how that might be done. For example, “How would you view the status of your application?”

Bad: Researcher guides participants in completing tasks. For example, “Which of the links in the header would you click to login?”

Team participation

Number of teammates

Good: The team designates a single moderator for the test, and at least one member of the product team observes the usability test.

Bad; A single person from the product team participates in and leads the test.

Observer role

Good: Observers do not moderate. They are generally quiet, and ask open-ended questions after the test has concluded.

Bad: Observers interrupt the participant, or attempt to sell or explain the product. Observers debate the participant’s actions in their presence.

Sensemaking

Notetaking

Good: Tests are recorded or notes are taken for sharing with absent stakeholders.

Bad: Tests are not recorded, or test results are not documented.

Synthesis

Good: Moderator debriefs with teammates immediately after each interview. Researcher looks for patterns across multiple participants and surfaces problems that affected several people.

Bad: Moderator reports the most memorable problems without conducting affinity mapping or some other analysis technique.

Presentation of findings

Good: Researcher reports findings to team and stakeholders in an easy to follow, well prioritized way.

Bad: Researcher presents team a “basket of issues” or an unprioritized laundry list of potential changes.

Incorporation of findings

Good: Product team translates findings into future user stories or product refinements.

Bad: Researcher reports do not affect the product backlog or ongoing development work.