-
Notifications
You must be signed in to change notification settings - Fork 9
/
how-to-evaluate-empirically.html
241 lines (185 loc) · 17.1 KB
/
how-to-evaluate-empirically.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<!-- Optional theme -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous">
<!-- Latest compiled and minified JavaScript -->
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
<link rel="stylesheet" href="style.css" />
<title>Empirical evaluation</title>
</head>
<body>
<p><a href="index.html">Back to table of contents</a></p>
<img src="images/usability-test.jpg" class="img-responsive" alt="A photograph of one person observing another person use a software application." />
<small>Credit: Media Barn Research</small>
<h1>How to evaluate empirically</h1>
<div class="lead">Amy J. Ko, Ph.D.</div>
<p>
<a href="how-to-be-critical.html">Critique</a> leverages intuition, expertise, and judgement.
These are powerful, invaluable sources of feedback, but they do have limitations.
Most notably, they can be subjective and sparse.
What if instead of expert speculation about whether a design will work (which is prone to "blind" spots (<a href="http://aer.sagepub.com/content/40/4/905.short">Nathan and Petrosino, 2003</a>), such as masking hard problems that experts view as easy), you want to actually <em>observe</em> whether it will work?
</p>
<p>
Observation, of course, requires empirical methods.
These contrast to critical methods in that they remove expert judgement from evaluation, leaving only observable phenomena in how someone interacts with a design.
This has the benefit of limiting subjectivity, which can, in some circumstances, be quite wrong in its interpretations and predictions.
</p>
<p>
There are numerous empirical methods for evaluating designs.
Here we'll just three general categories, and their tradeoffs.
</p>
<h3>Usability Tests</h3>
<p>
One of the lowest cost methods that works well for low-fidelity prototypes is a task-based evaluation (also called a "user" or "usability" test).
In a usability test, you define some common tasks to perform with your user interface and you invite several people who are representative of the people you're designing for to attempt to use your design.
Usability tests can help you learn about lower level problems in a user interface (layout, labeling, flow, etc.), but they generally can't help you learn about whether the design achieves its larger goals (whether it's useful, valuable, meaningful, etc.).
This is because a usability test doesn't occur in the context of someone's actual life, where those larger goals are relevant.
</p>
<p>
The goal of most usability tests is to discover aspects of a design that cause someone to fail at some task.
We call these failures <strong>breakdowns</strong>, the idea being that someone can be following the correct sequence of steps to complete a task, but then fail to get past a crucial step.
Once you've found the breakdowns that occur in your design, you can go back and redesign your interface to prevent breakdowns, running more usability tests after redesign to see if those breakdowns still occur.
Usability tests allow the designer to observe these breakdowns in person, helping them to make highly informed interpretations of what caused them, informing redesign.
</p>
<p>
Running a usability test has a few essential steps.
</p>
<p>
First, you need to decide who is <strong>representative</strong> of the stakeholders you are designing for and then find several people to invite to participate.
Who is representative depends entirely on whose problem you think you're solving and partly on your ability to get access to people that have that problem.
For example, if you are designing a course planner for students, you would want to recruit students (but what kind of students)?
If your representative users are challenging to recruit, you might have to get creative.
I've often had to walk into coffee shops and ask random strangers, or spam mailing lists to ask people to participate.
You have to be a bit bold and courageous to find participants, and find ways of compensating them for their time and attention.
If you're working for a company that invests in a whole team to find people to participate in user studies, you might be able to delegate this recruiting work to them.
</p>
<p>
In addition to finding representative users, you need to define <strong>tasks</strong> for the participants to attempt to perform.
Which tasks you choose depends on which tasks you think will be most important and common when people are using your solution.
Good tasks define the <strong>goal</strong> you want a user to achieve with your design without giving away any of the knowledge they need to achieve the goal.
If you <em>do</em> give away this knowledge, then it wouldn't be a fair test of your design in the real world, because you wouldn't have been there to help.
For example, if your goal is for someone to print a document with your app by using the "print" button, your instructions can't say "print the document", because then the user would know that "print" is the key word to find.
Instead, you might show them a printout of what you want and say, "Use this interface to make this".
The same applies if a participant asks you questions: you can't answer them, because you wouldn't normally be there to help.
The design should do the teaching.
</p>
<p>
Once you've defined your tasks, try to define what path you expect them to take through your design to accomplish the goal.
This way, as you are observing someone work, you can be monitoring where you expect them to go, and then note when they deviate from that path.
</p>
<p>
<img class="img-responsive center-block" style="width:50%" src="images/breakdowns.jpg" alt="A green line showing the designer's intended path, and a red line showing the user's actual path." />
</p>
<p>
When a participant arrives to participate in your user study, welcome them, explain that you're here to test the design and not them, and then explain what you'll have them do.
For example, you might have a script like this:
</p>
<blockquote>
Today we're interested in seeing how people use this new copier design...
We're here to test this system, not you, so anything that goes wrong is our fault, not yours.
I'm going to give you some tasks to perform. I won't be able to answer your questions during the test, because the goal is to see where people have difficulty, so we can make it easier.
Do you have any questions before we begin?
</blockquote>
<p>
To ensure the validity of the results that you get, you have to resist answering any of the questions that participants might ask unless it's about something other than the design.
It might reduce their discomfort and yours, but you won't be there in real life and you want to see how hard the task is.
Sit on your hands and close your mouth!
</p>
<p>
Once you conduct a test like this, and you observe several breakdowns, you may wonder <em>why</em> people were confused.
One strategy is to ask your participants to think aloud (<a href="http://psycnet.apa.org/doi/10.1037/0033-295X.87.3.215">Ericsson and Simon, 1980</a>) while they attempt to complete the task.
You can say:
</p>
<blockquote>
I need you to think aloud while working.
Tell me constantly what you're wondering about, confused about.
If you stop talking, I'll prompt you to resume talking.
</blockquote>
<p>
<img class="img-responsive center-block" style="width:50%" src="images/thinkaloud.png" alt="Two panel comic strip, on the left it says I don't understand what to do next and on the right it says I didn't understand what to do next" />
</p>
<p>
If your design requires too much concentration to have someone think aloud while they're using it (e.g., they're playing a game), you can also record their interactions and then conduct a <strong>retrospective interview</strong>, having them reflect on the actions that they took while watching the recording.
These recordings might just be the screen interactions, or they might also include the user's context, facial expressions, or other details.
Recording can also be useful for showing designers and engineers breakdowns, helping to persuade others in an organization that a design needs to be improved.
</p>
<p>
Not all think aloud is valid (<a href="https://dl.acm.org/citation.cfm?id=2379065">Gill and Nonnecke, 2012</a>).
Human beings cannot reliably tell you about <em>perceptual</em> phenomenon such as 1) why they didn't notice something or 2) what they noticed first.
People can share valid knowledge of what questions they have, what reactions they have to your design, what strategies they're trying, and anything else that requires explicit, conscious planning.
</p>
<p>
After finishing the user study, debrief with your participant, helping them to finish tasks they couldn't complete (so they don't feel like they failed), and reassure them that the design is still in progress, so any difficulties they experienced were your fault and not theirs.
This is also a great opportunity to ask for additional feedback.
</p>
<p>
While user studies can tell you a lot about the usability problems in your interface and help you identify incremental improvements to your design, they can't identify fundamental flaws and they can't tell you whether your design is <strong>useful</strong>.
This is because <em>you</em> define the tasks.
If no one wants to complete those tasks in real life, or there are conditions that change the nature of those tasks in real life, your user study results will not reveal those things.
The only way to find out if something would actually be used is to implement your design and give it to people to see if it offers real value (you'd know, because they wouldn't want you to take it away).
</p>
<h3>Technology Probes and Experience Sampling</h3>
<p>
One way to assess the usefulness of your design is to situate it in a real context and watch what happens.
Some designers will use a method called a <a href="http://dx.doi.org/10.1145/642611.642616">technology probe</a>, which deploys a prototype into a context, allowing designers to research how the prototype has changed their practices through interviews, logging, and other types of data collection.
It's also possible to use experience sampling (<a href="http://www.tandfonline.com/doi/abs/10.1080/10447310709336957">Consolvo et al., 2007</a>), which is a more narrow strategy of just interrupting users with brief surveys about how their experience with a prototype, gathered in the context of using it.
Both of these methods emphasize <strong>ecological validity</strong>, or the degree to which an evaluation generalizes to real situations.
By using actual real-life contexts, you can better understand how people use your design in their natural environments.
</p>
<p>
These situated forms of evaluation, while the most likely to lead to insights that will reflect reality, have high cost.
Your design needs to be implemented and reliable enough that someone can actually use it in daily life.
This means that these methods often come later in a design process, after a high fidelity, functional prototype exists.
</p>
<img src="images/ab-testing.png" class="img-responsive" alt="A digram showing A and B, Control and Variation, and a 23% and a 37%" />
<small>Credit: Wikipedia</small>
<h3>A/B Tests</h3>
<p>
While situated evaluations like experience sampling can reveal the context in which something is used, it can be challenging to evaluate whether a design is working at scale.
When an implementation is built enough to run at some scale, it is now common in industry to compare designs <em>experimentally</em>, giving one design to a percent of the user population and another design to the rest of the population, then measuring the difference in behavior of the two populations.
Experiments like these are quite popular in industry, in which one design (usually the current design, and therefore the "control" condition) is compared to some new design (usually a new design, and therefore the "experimental" condition).
You might know these by the name <a href="https://www.linkedin.com/pulse/ab-testing-vs-user-experience-research-yann-riche">A/B tests</a>, where the A usually represents a "control" condition and the B usually represents an "experimental" condition.
A/B tests are usually deployed in online settings, and the experiment is run remotely.
Experiments such as these can provide the strongest evidence of causality.
For example, a designer makes the "sign up" button larger and more prominent on the webpage she is testing in her experimental condition.
After running the experiment for some time, the designer looks at how many new accounts were created in the control and experimental conditions.
She finds that more accounts were created in the experimental condition, leading her to believe that the increased visibility of the "sign up" button <em>caused</em> more users to sign up compared to the control condition.
</p>
<p>
One challenge of designing good A/B tests is ensuing that the results can be trusted.
Industry is also still learning how to design good experiments; most A/B tests fail to meet even minimum standards of the kinds of randomized controlled experiments used in science.
Some data science education is beginning to prepare data scientists who can design sound experiments, working alongside designers to evaluate specific, measurable features of designs.
</p>
<p>
A major limitation of A/B tests is that because it's difficult to come up with holistic measures of success, the results tend to be pretty narrow.
Perhaps that's okay if your definition of success is increased profit.
Making more money is easy to measure.
But if your definition of success is harder to measure (e.g,. there's less hate speech on your platform), A/B tests might be much harder to conduct.
The ease with which A/B tests can run, and the difficulty of measuring meaningful things, can lead designers to overlook the importance of meaningful things.
A good designer will resist this path of least resistance, focusing on the outcomes that matter to a design, independent of what tools make easy.
</p>
<center class="lead"><a href="how-to-evaluate-analytically.html">Next chapter: How to Evaluate Analytically</a></center>
<h2>Further reading</h2>
<p>Ericsson, K. A., & Simon, H. A. (1980). <a href="http://dx.doi.org/10.1037/0033-295X.87.3.215">Verbal reports as data</a>. Psychological review, 87(3), 215.</p>
<p>Consolvo, S., Harrison, B., Smith, I., Chen, M. Y., Everitt, K., Froehlich, J., & Landay, J. A. (2007). <a href="http://www.tandfonline.com/doi/abs/10.1080/10447310709336957">Conducting in situ evaluations for and with ubiquitous computing technologies</a>. International Journal of Human-Computer Interaction, 22(1-2), 103-118.</p>
<p>Gill, A. M., & Nonnecke, B. (2012, October). <a href="https://doi.org/10.1145/2379057.2379065">Think aloud: effects and validity</a>. In Proceedings of the 30th ACM international conference on Design of communication (pp. 31-36).</p>
<p>Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., ... & Roussel, N. (2003, April). <a href="http://dx.doi.org/10.1145/642611.642616">Technology probes: inspiring design for and with families</a>. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 17-24). ACM.</p>
<p>Nathan, M. J., & Petrosino, A. (2003). <a href="http://aer.sagepub.com/content/40/4/905.short">Expert blind spot among preservice teachers</a>. American educational research journal, 40(4), 905-928.</p>
<p>Reiss, E. (2012). <a href="https://www.amazon.com/Usable-Usability-Simple-Making-Better/dp/1118185471">Usable usability: simple steps for making stuff better</a>. John Wiley & Sons.</p>
<p>Riche, Y. (2016). <a href="https://www.linkedin.com/pulse/ab-testing-vs-user-experience-research-yann-riche">A/B testing vs. User Experience Research</a>.</p>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-10917999-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</body>
</html>