User Testing: Performance Criteria

[SachaG]

What criteria are we planning to use to decide which UI format to move forward with?

Personally, the metric I care the most about is cognitive load (because I assume the harder the survey is to complete, the more people will drop off midway through), which isn't really something that can easily be measured. But maybe a good proxy would be time spent per question? If my assumption that some of our UI formats require a lot more brainpower than others is true then you'd expect respondents to be slowed down by these formats.

I would also want to be on the look-out for any signs that respondents have trouble forming a correct mental model of how the UI works. For example, they might think the controls do something other than what they actually do, or think some questions are required when they aren't. I'm not sure how to measure that though, maybe through a post-survey quiz?

Are there any other objective criteria that can help us judge the results of the user testing?

[LeaVerou]

I don't think we can define an algorithm for making the decision since making the right tradeoffs is a judgement call. Here are some factors we should take into account.

There are three dimensions of usability, all of which are important:

• Learnability (how easy is it to figure out?)
• Efficiency (how fast is it to use?)
• Safety (how easy is it to make mistakes? Are mistakes recoverable?)

Participant sentiment should also be weighed, as that affects the experience (you'd expect this to be a function of the observed usability, but paradoxically it often is not). We can ask about this at the end (though some participants will likely express thoughts throughout).

Other factors, more specifically around survey answer UI:

• How well does this UI express the participant's actual answer? If we see a lot of "what I actually think is X, but this UI only allows me to input Y", then IMO that should be weighed very heavily.
• For optional things, what percentage of participants actually engages with them?

We should definitely capture metrics about time spent on particular questions, though the shorter the times we're measuring, the higher the error as it's hard to know exactly when their attention has shifted to the next question by just observing. It may make sense to measure time spent on several questions using the same template, and then just divide to find the average for each question template.

[SachaG]

OK, so given these dimensions here's specific criteria I would consider:

Dimension	Criteria	Measured in
Learnability	how long respondents pause or stumble for when first encountering the UI.	seconds
Efficiency	how long respondents take to go through a question after the first time	seconds
Safety	not sure I have a concrete criteria for that yet	?
Expressing the participant's answer	respondents voice concerns such as "what I actually think is X, but this UI only allows me to input Y" out loud.	yes/no
Response percentage	what percentage of respondents fill out the optional items (although I don't give this too much credit because I think completion rates will be artificially high during a user test since people know they are being watched).	percentage
Participant sentiment	whether the participant voices any thoughts about the UI (or could just ask them outright which one they preferred)	A vs B

We can take these 5/6 criteria, and for every respondent session score them either for Option A (5-option queston) or Option B (sentiment chips) – or as a draw. Then see which option has the most points overall. What do you think?

(cc @stubbornella)

[LeaVerou]

We can take these 5/6 criteria, and for every respondent session score them either for Option A (5-option queston) or Option B (sentiment chips) – or as a draw. Then see which option has the most points overall. What do you think?

These are inherently qualitative studies, we cannot define a precise algorithm that will tell us which UI we should go with. We should define factors and priorities, but not a full algorithm. Usually it becomes pretty obvious after a few sessions though 😄

Learnability	how long respondents pause or stumble for when first encountering the UI.	seconds

Learnability is more broad than how long they pause or stumble. Can they understand how to use the control and what it does? If they get confused at first, do they get stuck, or do they progress and figure it out quickly?

Safety	not sure I have a concrete criteria for that yet	?

Safety is things like, how many mistakes do they make with the interface? Are mistakes recoverable? How destructive are the mistakes?

(although I don't give this too much credit because I think completion rates will be artificially high during a user test since people know they are being watched).

This is correct, but % rates are still useful, just only in one direction: if everyone fills these fields in, that doesn't tell us much, because as you pointed out, they are more keen to fill everything in when watched. However, if despite being watched, they still don't fill them in, that's a pretty strong signal that % response rate in the wild would be pretty low.

---

I think if we want something quantitative-ish, maybe something more relaxed like:

• Learnability: 1-5
• Efficiency: Seconds to go through each question (median)
• Safety: 1-5
• Response %: 0-100% (where applicable)
• Data signal vs noise: 1-5 (the "I cannot express myself" issue)
• Participant sentiment: 1-5

Without a way to weigh these factors and produce a single score, this will not give us the answer, though but it may well give us a better picture.

[SachaG]

Well, I was trying to follow @stubbornella's suggestion that we use the user testing as a way to decide on UI issues to put an end to the debates, but I was afraid that if we don't have pre-existent success criteria then we won't be able to do that in a way that's satisfying for everybody.