just-enough-research.md

Just Enough Research (2013)

Erika Hall

───────────────

▪ In the digital age, creating meaningful design requires us to understand people who are different, anticipate what they will want to do, and provide them with the tools they need, exactly when they need them

▪ We are not saints, gods, or clairvoyants. We can only understand others through research, and that is the subject of this shiny little book.

▪ In a perfect world, research budgets are sufficient, ample research time is provided in advance of project definition, and accuracy is assured through shared models and processes

▪ In that perfect world, clients value research. They reward designers who ask difficult questions, and prioritize user needs over marketing directives, internal politics, or personal peccadillos.

▪ But that’s not the world we work in. In our world, budgets are constrained, schedules are absurd, there is little internal agreement about what constitutes worthwhile research, and “I don’t like yellow” is considered feedback.

▪ “A little learning is a dangerous thing.”

—Alexander Pope

▪ To design, to code, to write is to embrace danger, to plunge ahead into the unknown, making new things out of constantly changing materials, exposing yourself to criticism and failure every single day

▪ Yet you’re shadowed by the idea that the best designers and developers and writers are self-motivated, self-inspiring, hermetically sealed units of mastery. The myth of the creative genius makes it very difficult to say “I don’t know.”

▪ You might not be going the right way, but who cares because you need to get there fast.

▪ In these settings, research can be a very scary word. It sounds like money you don’t have and time you can’t spare, like some egghead is gathering wool in a lab or library when you could be moving forward and building something

▪ Scariest of all, it means admitting you don’t have all the answers.

▪ You may have a vague idea that research is a good thing, but the benefits are fuzzy while the costs are all too clear.

▪ Research is a tool—a periscope offering you a better view of your surroundings

▪ It can be very powerful if applied thoughtfully. Rather than piling on the costs, research can save you and the rest of your team a ton of time and effort.

▪ Because once you start getting answers, you’ll keep asking more questions. And that skeptical mind-set is more valuable than any specific methodology.

▪ And while we enjoy speculating about the future, we felt irresponsible taking our client’s money for guessing.

▪ Businesses and designers are keen on innovation, as well they should be. But the better you know the current state of things and why they’re like that, the better you will be positioned to innovate.

▪ Research is simply systematic inquiry. You want to know more about a particular topic, so you go through a process to increase your knowledge. The type of process depends on who you are and what you need to know

▪ A lot of personal research these days begins with a Google query (“Who is Mihaly Csikszentmihalyi?”) and ends on a Wikipedia page. (“Oh, so that’s how you pronounce it.”)

▪ Finding information is relatively easy. The knowledge already exists. You just have to find a trustworthy source for it. Assessing credibility is the hard part. (“Are all polar bears really left-handed?”)

▪ Pure research is carried out to create new human knowledge, whether to uncover new facts or fundamental principles. The researcher or investigator wants to advance a particular field, such as neuroscience, by answering a particular question, such as “Why do humans sleep?”

▪ Pure research is based on observation or experimentation. The results are published in peer-reviewed journals. This is science

▪ Rigorous standards and methodologies exist to preserve objectivity and ensure the credibility of conclusions. (Things get squishy when corporations fund ostensibly pure research, as they frequently do.)

▪ Applied research borrows ideas and techniques from pure research to serve a specific real-world goal, such as creating a supersoldier or improving the quality of hospital care or finding new ways to market pork-flavored soda

▪ While ethics are just as important, methods can be more relaxed. Maybe this means changing up the questions you ask partway through a study, or making the most of an imperfect sample group because you’re tight on time.

▪ The research is successful to the extent that it contributes to the stated goal.

▪ As with pure research, sometimes you accidentally discover something valuable you weren’t even looking for, and that’s a fantastic bonus.

▪ And then there is design research.

▪ Design research is a broad term with a long history. In the 1960s, design research referred to the study of design itself, its purpose and processes.

▪ If you’re interested in transformative concepts of spatial intelligence or the poetics of the sustainable kitchen, this field is for you

▪ However, when practicing industrial or interactive designers refer to design research, they typically mean research that is integral to the design work itself—inquiries that are part of designing, not about design

▪ This research focuses largely on understanding the people for whom we’re designing, often referred to by the dehumanizing but instrumental term end users

▪ Research is a core part of user-centered design.

▪ “Informing Our Intuition: Design Research for Radical Innovation” (http://bkaprt.com/jer/1/):

▪ Design research both inspires imagination and informs intuition through a variety of methods with related intents: to expose patterns underlying the rich reality of people’s behaviors and experiences, to explore reactions to probes and prototypes, and to shed light on the unknown through iterative hypothesis and experiment.

▪ For a design to be successful, it must serve the needs and desires of actual humans.

▪ Strangely, simply being human is insufficient for understanding most of our fellows

▪ Design research requires us to approach familiar people and things as though they are unknown to us to see them clearly. We need to peel away our assumptions like a gray alien shedding its encounter suit.

▪ Asking your own questions and knowing how to find the answers is a critical part of being a designer. If you rely on other people to set the agenda for inquiry, you might end up caught between fuzzy focus groups and an algorithm that scientifically chooses a drop shadow from among forty-one shades of blue.

▪ Discovering how and why people behave as they do and what opportunities that presents for your business or organization will open the way to more innovative and appropriate design solutions than asking how they feel or merely tweaking your current design based on analytics.

▪ You will find that when you ask the hard questions, your job gets much easier. You will have stronger arguments, clarity of purpose, and the freedom to innovate that only comes with truly knowing your constraints

▪ Research is not asking people what they like

▪ As you start interviewing people involved in business and design decisions, you might hear them refer to what they do or don’t like. “Like” is not a part of the critical thinker’s vocabulary

▪ On some level, we all want the things we do to be liked (particularly on Facebook), so it’s easy to treat likability as a leading success indicator. But the concept of “liking” is as subjective as it is empty. It is a superficial and self-reported mental state unmoored from any particular behavior. This means you can’t get any useful insights from any given individual reporting that they like or hate a particular thing. I like horses, but I’m not going to buy any online.

▪ Quash all liking, and hating too. Plenty of people habitually engage in activities they claim to hate.

▪ Don’t let your methods be guided by a desire to appear smart or conform to anyone else’s picture of research. Some clients will argue for doing interviews in a usability lab even when it isn’t appropriate, just because it feels research-y.

▪ You’ll need to explain to them why interviews with method and purpose are more valuable than having a social conversation with a random person—and why it really doesn’t matter where you do them, as long as they’re done right.

▪ Applied research is not science

▪ Avoid arguments about statistical significance; you will not win.

▪ Instead, keep the focus on gathering useful insights

▪ People who make design decisions at any level benefit from asking more and better questions

▪ It is a sampler rather than a survey, and a biased sampler in that I have included only the topics and approaches I personally have found most useful in my design career.

▪ WHO SHOULD DO RESEARCH? EVERYONE!

Ideally, everyone who is on the design team should also participate in the research

▪ The content strategist would notice the vocabulary real people used and the developer had good questions about personal technology habits. The visual designer was just really into motorcycles, and that helped sometimes too.

▪ Someone needs to be the research lead—the person who keeps everyone on track and on protocol and takes ultimate responsibility for the quality of the work

▪ One of our maxims at Mule is that every design project ultimately amounts to a series of decisions. What are the most important features? What is the best navigation scheme? How big should the logo be?

▪ For any given project, we include only the research activities that support the specific decisions we anticipate. If the client has only identified an audience and wants to explore ways to better serve them (“What can we offer of value to high school science teachers?”), our research will be more open-ended than if the design problem is already well defined (“How can we get high school science teachers to download and use our lesson plans?”).

▪ Organizational research will tell you which interactions benefit the museum most, while user research will indicate which are most plausible and the circumstances under which they will take place.

▪ To choose the best research tool for your project, you’ll need to know what decisions are in play (the purpose) and what you’re asking about (the topic).

▪ Then you can find the best ways to gather background information, determine the project’s goals and requirements, understand the project’s current context, and evaluate potential solutions.

▪ Generative or exploratory research: “What’s up with...?”

This is the research you do before you even know what you’re doing

▪ It leads to ideas and helps define the problem. Don’t think of this as just the earliest research. Even if you’re working on an existing product or service, you might be looking for ideas for additional features or other enhancements, or new products you could bring to an audience you’re already serving.

▪ Generative research can include interviews, field observation, and reviewing existing literature—plus feeling fancy about saying “generative research.”

▪ Your goal would be to see the new parent experience from their eyes, to understand what they do and what they need. Your generative research activities might include interviewing new parents on the phone, following new parents around on a typical day, or looking at the questions new parents ask on social websites.

▪ Once you’ve gathered information, the next step is to comb through it and determine the most commonly voiced unmet needs. This sort of research and analysis helps point out useful problems to solve. Your thinking might lead to a hypothesis, such as “Local parents of young children would value an app that offers ideas for science events and activities based on their child’s developmental milestones.” Then you can do further (descriptive) research on how parents recognize and commemorate those milestones.

▪ Descriptive and explanatory: “What and how?”

▪ Descriptive research involves observing and describing the characteristics of what you’re studying.

▪ This is what you do when you already have a design problem and you need to do your homework to fully understand the context to ensure that you design for the audience instead of yourself.

▪ While the activities can be very similar to generative research, descriptive research differs in the high-level question you’re asking. You’ve moved past “What is a good problem to solve?” to “What is the best way to solve the problem I’ve identified?”

▪ The Glaucoma Research Foundation offered a clear design problem to solve: how to create useful, accurate educational materials for people who had been newly diagnosed with an eye disease. So, a round of descriptive research was in order.

▪ For the Fantastic Science Center, descriptive research comes into play once we’ve identified a design problem, such as providing an online robotics course for students around the world. Maybe this supports the organizational goal to create a global robot army.

▪ Once you have a very clear idea of the problem you’re trying to solve, you can begin to define potential solutions. And once you have ideas for potential solutions, you can test them to make sure they work and meet the requirements you’ve identified. This is research you can, and should, do in an ongoing and iterative way as you move through design and development.

▪ The most common type of evaluative research is usability testing, but any time you put a proposed design solution in front of your client, you really are doing some evaluative research.

▪ Causal research: “Why is this happening?”

▪ Once you have implemented the solutions you proposed, and have a website or application up and running out in the world, you might start noticing that people are using it in a certain way, possibly a way that isn’t exactly what you’d hoped

▪ For example, you’ve noticed that ever since the Fantastic Science Center redesign launched, tickets for the Friday evening science-loving singles event are selling better, but ticket sales have completely dropped off for the Sunday afternoon film program. You need to do some causal research

▪ Establishing a cause-and-effect relationship can be tricky

▪ Causal research often includes looking at analytics and conducting multivariate testing (see Chapter 9). This means reviewing your site traffic to see how visitors are entering and moving around the site and what words they might be searching for, as well as trying design and language variations to see which ones are more effective

▪ Causal research might show that your film program traffic all came from one referring website that no longer links to you. Or, you might have to expand beyond looking at site performance to see what else is going on. Maybe a competing organization started an event with a very similar name and confused your target audience.

▪ Your research will benefit from a collaborative approach that includes assigning specific responsibilities to different people.

▪ Roles

Research roles represent clusters of tasks, not individual people. Often one person will take multiple roles on a study, or a single role can be shared

▪ Author

The author plans and writes the study. This includes the problem statement and questions, and the interview guide or test script.

▪ Interviewer/Moderator

The interviewer is the person who interacts directly with the test participants.

▪ Coordinator/Scheduler

The coordinator plans how time will be used during the study and schedules sessions, including arranging times with the participants

▪ Notetaker/Recorder

The notetaker is responsible for capturing the data during a test session. It is best that the interviewer and notetaker be two separate people so that the interviewer can devote full attention to the participant. If this is impossible, the interviewer can arrange to record the session as unobtrusively as possible

▪ Having both written notes and a recording makes data analysis easier

▪ Recruiter

The recruiter screens potential participants and identifies the respondents who would be good subjects. The recruiter and scheduler can easily be the same person.

▪ Analyst

The analyst reviews the gathered data to look for patterns and insights. More than one person should have this role

▪ Documenter

The documenter reports the findings once the research study is complete

▪ Observer

Often it’s useful for clients or other available team members to watch the research in progress. This is appropriate as long as the presence of the observers will not influence the research itself. As a substitute, you can make raw recordings available

▪ For the purposes of this section, what matters is that everyone working together has a shared understanding of how the work will proceed. This can be as simple as a checklist.

▪ Sometimes the people you’re working with or for will consider research somewhere between a threat and a nuisance.

You might have to get a certain amount of what they call organizational buy-in to proceed

▪ Get ready to advocate for your research project—before you start it.

▪ We don’t have time

You don’t have time to be wrong about your assumptions. What are your key assumptions? What if they’re all wrong? How much work would you have to redo? How long would that take?

▪ Even with little or no budget, you can usually locate some related research online, wrangle representative users to interview, and do a little usability testing

▪ One research methodology is superior (qualitative vs. quantitative)

What you need to find out determines the type of research you need to conduct. It’s that simple.

▪ You need to be a scientist

This isn’t pure science we’re talking about here. This is applied research

▪ Your desire to find out needs to be stronger than your desire to predict. Otherwise you’ll be a mess of confirmation bias, looking for answers that confirm what you already assume

▪ You need to be able to depersonalize the work. There are no hurt feelings or bruised toes in research, only findings

▪ You need infrastructure

I suspect you own or can borrow a laptop and have access to the internet. That is all you need

▪ It will take too long

Upfront research can provide a basis for decision-making that makes the rest of the work go much faster

▪ Nothing slows down design and development projects as much as arguing over personal opinions or wasting effort solving the wrong problem. And you can start small. A couple of weeks can mean very little to your overall schedule while adding significantly to your potential for success.

▪ Again, it’s a matter of where you want to invest and what you have to lose. Don’t waste anyone’s time or effort on untested assumptions if you don’t have to.

▪ Research will change the scope of the project

It’s better to adjust the scope intentionally at the start than be surprised when new information pops up down the road to amend your plans. Research is an excellent prophylactic against unexpected complexity.

▪ Actual reasons behind the objections

At the root of most of these objections is a special goo made up of laziness and fear.

I don’t want to be bothered

▪ Unless you are naturally curious about people, research can seem like annoying homework at first, but once you get into it, you’ll find it totally fun and useful. A little knowledge opens up a whole world of new problems to solve and new ways to solve the problems at hand

▪ If research is one more thing tossed on your already overfull plate, then someone needs to ask the “Who should be doing this?” question again—but the problem is you being too busy, not research being unimportant

▪ Research needs to be integrated into process and workflow or it will get shoved in a corner

▪ I am afraid of being wrong

The cult of the individual genius designer/developer/entrepreneur is strong. In certain “rockstar knows best” cultures, wanting to do research can come across as a sign of weakness or lack of confidence. Fight this

▪ Information and iteration are the keys to a successful design. Research is just one set of inputs

▪ I am very uncomfortable talking to people

You are creating a system or a service actual people are going to have to use. This system will be talking to people on your behalf, so it’s only fair that you talk to people on its behalf

▪ That said, some people on your team will have more comfort and skills when it comes to interacting with your research subjects, so consider that when you’re deciding who does what

▪ Describing the goals and potential of your research to people who aren’t sold on the value will actually help you focus and better articulate what you hope to uncover

▪ “Poor user experiences inevitably come from poorly informed design teams.”

—Jared M. Spool, founder of User Interface Engineering

▪ Design happens in context. And research is simply understanding that context

▪ Research happens in a context as well. Your professional environment should inform the research activities you choose and how you go about them

▪ If you are doing work, you need to get paid for it. “Just tossing in the research” is a terrible mistake designers who want to do good work make all the time. Instead, you have to commit to research personally as part of how you work, make your case for it, and be sure to include it in your fee

▪ Research will make your design stronger and enhance your ability to defend your decisions to the client

▪ In an established organization of any size, politics are a huge consideration. Challenging the assumptions of those in power can be incredibly threatening to those people

▪ If you’re at an organization that genuinely embraces critical thinking and clear communication in the design process, that’s terrific. I hope you’re also taking very good care of your unicorn desk mate. Otherwise, proceed with open eyes and discretion

▪ An existing product means that a glorious data trove exists: customer service! Customer service is where actual, individual human needs and expectations crash headfirst into reality

▪ In addition to, or instead of, direct access to the customer service people, get hold of the inbound support requests

▪ This will be a fantastic source of insights into the ways different types of customers think about their needs and the language they use to describe them.

▪ You can also practice seeing through what people say they want to what they actually need. Steve in Louisville may be asking for a more informative error message, but what he really wants is to be able to reorder his usual pizza and salad with a different credit card and no error message.

▪ You don’t just want insights; you also want a way to put those insights back into the product

▪ It’s very helpful to have a clear idea of how product and marketing decisions are made in your company. Do you have a truly customer-centered culture? When leaders talk about research, are they talking about market research, or do they have a more holistic view? Is there a market research group? Is design or user experience research already part of your process

▪ When you’re discussing the initial design and development of your product, discuss the role of research with the team. Document and review assumptions to identify the areas in which doing some research might be the most beneficial

▪ The approach and biases of the founder and the investors might dominate, so if you aren’t one of those, you will have to be very clear about the value of research to your endeavor and savvy about how to make your case

▪ Agile is a popular software development philosophy with the goal of building better software faster in a productive, collaborative working environment. Many short iterations of two or three weeks replace the traditional approach of multi-month or multi-year projects broken into distinct phases.

▪ On the surface, agile seems antithetical to design. The agile manifesto explicitly values “responding to change over following a plan.” Design is planning. However, any work with complex ideas and dependencies requires holding some ideas outside the development process

▪ While flexibility and responsiveness are certainly virtues that many project teams could use more of, let’s not discount the importance of having some sort of plan.

▪ From a user experience perspective, the primary problem with Agile is that it’s focused on the process, not the outcomes. It doesn’t offer guidance on what to build, only how

▪ Research is not antithetical to moving fast and shipping constantly. You’ll need to do some upfront work for background and strategy and the overall framework. Then, as the work progresses, do continual research

▪ It might sound counterintuitive, but the most effective approach may be to decouple the research planning from the development process—that is, don’t wait to start coding until you’ve answered all your research questions

▪ Once you have some basic tools and processes in place, such as observation guides, interview guides, recording equipment, and questions for analysis, you can take a Mad Libs approach and fill in your actual questions and prototypes on the fly

▪ Jeff Patton describes this continuous user research process in his article “Twelve Emerging Best Practices for Adding UX Work to Agile Development” (http://bkaprt.com/jer/2/). He offers a tidy three-point summary:

▪ Aggressively prioritize the highest-value users

▪ Analyze and model data quickly and collaboratively

▪ Defer less urgent research and complete it while the software is being constructed

▪ In other words, focus only on the essential user types, deal with your data as soon as you get it, involve your team in the analysis, and do the less important stuff later

▪ Prioritize those user types whose acceptance of the product is critical to success and those who least resemble the software developers on your team. Go learn about them

▪ Recruiting and scheduling participants is the most difficult part

▪ always be recruiting

▪ Set up windows of time with different participants every three weeks

▪ When you have them, you can either conduct an ethnographic interview (see Chapter 5) to understand their behavior before the next round of development or do some usability testing on the current state of the application

▪ Use what you learn from the initial user research and analysis to create personas that inform high-level sketches and user stories

▪ Then, when the team is working on a feature that has a lot more engineering complexity than interaction design complexity, you can fit in additional evaluative research

▪ Throughout the development cycle, the designers can use research to function as a periscope, keeping an eye out for new insights about users and competitive opportunities while doing usability testing on whatever is ready

▪ Cover your bias

Wherever there is research there is bias. Your perspective is colored by your habits, beliefs, and attitudes. Any study you design, run, or analyze will have at least a little bit of bias. Your group of participants will be imperfectly representative. Your data gathering will be skewed. Your analysis will be colored by selective interpretation.

▪ Don’t give up!

You can’t eliminate it completely—but the simple act of noting potential or obvious bias in your research process or results will allow you to weigh the results more appropriately.

▪ In lieu of a trained eye, use the following bias checklist, or make your own. Grade hard.

▪ Design bias

Design in this case refers to the design of the studies themselves, how they are structured and conducted. This is the bias that creeps into studies when you don’t acknowledge bias, or if you include or leave out information based on personal goals or preferences.

▪ Sampling bias

Since we’re talking about quick and dirty qualitative research, sampling bias is almost unavoidable

▪ If your app for science-minded new parents is intended to serve men and women in equal numbers but all your subjects are women, that’s a biased sample

▪ Interviewer bias

Conducting unbiased interviews is difficult. Inserting one’s opinions is easy. Make sure that interviewers remain as neutral as possible

▪ Practice interviews and critiques with an internal team are the best way to develop a neutral interviewing style.

▪ Sponsor bias

This is one of the biggest issues with onsite lab usability tests, because going onsite feels special and can be exciting or even daunting to a participant

▪ If the Fantastic Science Center is inviting you in to their facility, offering you snacks, and writing you a check, it is very possible you will be gentler in your evaluations

▪ To decrease sponsor bias without being deceptive, use a general description of the organization and goals of the study without naming the specific company until and unless it appears in materials you are evaluating

▪ For example, begin a phone interview with “We’re interested in how you select and plan activities for your family,” rather than “We want you to tell us what would entice you to visit the Fantastic Science Center

▪ Social desirability bias

Everyone wants to look their best. People want to be liked. It can be hard to admit to an interviewer that you don’t floss or pay off your credit card bill every month, so participants will sometimes give the answers that put them in the best light. Emphasize the need for honesty and promise confidentiality.

▪ The Hawthorne effect

The behavior of the people you are studying might change just because you are there

▪ Staff who typically goof around and chat during the day might clam up and shuffle files if you’re hanging about to observe their workflow. Do your best to blend into the background and encourage research participants to go about their normal day.

▪ What harm can come of asking people how they decide what to have for dinner or how they use their phones to find directions? We aren’t talking about clinical trials of dangerous, new cancer drugs, but all research that includes people and their personal information should be conducted ethically and conscientiously

▪ Below is a starter set of ethical concerns you should keep in mind whenever you are doing research

▪ (For more thorough guidelines, take a look at the ICC/ESOMAR Code on Market and Social Research, which is available in fifteen languages: http://bkaprt.com/jer/3/.)

▪ Maybe this goes without saying, but it is worth saying nevertheless. Is your overall goal, the project that the research supports, ethical? Will your success lead to harm for others?

▪ Conducting a completely above-the-board study on women to induce them to buy a diet aid with dangerous side effects doesn’t make it right

▪ The goals or methods of the research

A certain amount of user research and usability requires keeping certain facts from the participants

▪ Usually this is benign, such as hiding the name and description of the product you’re designing, but sometimes it’s a problem. Will concealing these facts lead those users to participate in anything they might not otherwise agree to?

▪ Consent and transparency

Informed consent is the rule. This means that participants must understand and agree in advance to the overall goals of any study and how their information will be recorded, used, or shared

▪ Let them know if they are being watched by unseen observers. Make sure that research participants are of sound mind and able to give consent to participate.

▪ This means that working with underage research participants is very tricky, and requires the parents’ consent.

▪ Be a skeptic

Get in the habit of asking a lot of questions. Question all your assumptions and determine whether you need to check your facts

▪ If you’re constantly on the lookout for threats and potential points of failure, you and your products will be stronger. This is a type of critical thinking that will serve you well at all times.

▪ As a designer or developer, you might have good reasons to avoid DIY and hire a trained professional.

These include:

A large, complex project. A large, complex organization. Complex or sensitive subject matter. A very specialized or challenging user base, such as children or neurosurgeons. Heinous organizational politics. Lack of team members with the time or inclination to acquire additional skills and duties.

▪ Discipline requires you to be ever-watchful for bad habits, shoddy thinking, and other human frailties that will undermine your efforts

▪ Checklists substitute the experience of others for your own. Discipline also requires that you don’t deviate from the checklists until you have sufficient experience yourself

▪ 1. Phrase questions clearly

This refers not to the questions you’re asking, but the big question you’re trying to answer. Unless you know and can clearly state what you’re trying to find out and why, applied research is a pointless exercise.

▪ Set realistic expectations

A successful study is preceded by expectation-setting for everyone involved, including the questions to be answered, the methods to be used, and the decisions to be informed by the findings. This is particularly important if you need to request time or budget especially for the work.

▪ 3. Be prepared

Research is like kitchen work: the better you prep, the faster and cleaner the work goes. If you don’t prepare, you end up with a huge mess and a kitchen on fire.

▪ Get your process and materials in order before you start. Set these up so they’re easy to reuse as needed.

▪ 4. Allow sufficient time for analysis

You need a little time for things to click into place. After doing the research, it’s tempting to just forge ahead to solutions without giving yourself enough time to digest.

▪ 5. Take dictation

Notes or it didn’t happen. Effective research requires effective reporting, and sharing your results and recommendations with others.

▪ A good report doesn’t have to be arduous to compile or read. It needs to be sufficiently informative and very clear to anyone who needs to make decisions based on the research.

▪ be honest with yourself and your team about your capacity for maintaining this level of rigor

▪ Otherwise you risk wasting both time and money, as well as spreading misinformation and decreasing the overall reputation of research as a necessary input into the work.

▪ There are things we know that we know. There are known unknowns—that is to say, there are things that we now know we don’t know. But there are also unknown unknowns—there are things we do not know we don’t know.”

—Donald Rumsfeld, former US secretary of defense

▪ research is essential to reducing your risk—the risk you incur by relying on assumptions that turn out to be wrong or by failing to focus on what’s most important to your business and your users.

▪ However, some assumptions are higher-risk than others.

To make the best use of your time and truly do just enough research, try to identify your highest-priority questions—your assumptions that carry the biggest risk.

▪ You could conduct research to understand the social sharing practices and motivations of people who shop online before diving into design and development. Or you could go ahead and design based on an optimistic assumption, then see what happens.

▪ Failing isn’t the only way to learn.

▪ No matter how much research you do, there will still be things you wish you’d known, and there are some things you can only learn once your design is out there in the world

▪ Design is an iterative process. Questions will continue to crop up. Some of them you can answer with research and some you can only answer with design.

▪ Even with research, you’ll need to create a few iterations of the wrong thing to get to the right thing.

▪ There is no answer to the question of enough, other than the point at which you feel sufficiently informed and inspired. The topics in this book can only offer a starter kit of known unknowns.

▪ Since human brains are pattern recognition machines, you might start seeing the patterns you want to see that aren’t actually there. Collaborating with a team to interpret the data will reduce the risk of overly optimistic interpretation.

▪ Whatever type of research you’re doing, and wherever it falls in your schedule, follow these six steps:

Define the problem. Select the approach. Plan and prepare for the research. Collect the data. Analyze the data. Report the results.

▪ 1. DEFINE THE PROBLEM

Just as you need a clearly articulated problem to create a solid design solution, a useful research study depends on a clear problem statement

▪ In design, you’re solving for user needs and business goals. In research, you’re solving for a lack of information.

▪ You want to know when you’re finished, right? So base your statement on a verb that indicates an outcome, such as “describe,” “evaluate,” or “identify.” Avoid using open-ended words like “understand” or “explore.”

▪ For example, if your topic is working parents of school-aged children and your question is, “How do they select and plan weekend activities?” then your problem statement could be, “We will describe how parents of school-age children select and plan weekend activities

▪ There are a lot of ways to answer a given question, and they all have tradeoffs

▪ Once you’ve selected the approach, write a quick description of the study by incorporating the question. For example: “We will describe how parents of school-age children select and plan weekend activities by conducting telephone interviews and compiling the results.”

▪ In the beginning, don’t worry about getting everything right. If you don’t know, go with your best guess. Since research is about seeking out new information, you’re going to encounter new situations and unpredictable circumstances. Make friends with the unexpected. And prepare to change the plan you’ve made to adapt once you have facts.

▪ You might plan for sixty-minute sessions but find that you’re getting all the information you need in half an hour. Or you might find that the name of a particular competitor keeps coming up during an interview, so you decide to add fifteen minutes of competitive usability testing to the interview so that you can observe your target customers using their service.

▪ Just be very clear about the points at which changes to your research plans might affect your larger project

▪ In addition to answering your research questions, you’ll continue to learn more about research itself

▪ Your research plan should include your problem statement, the duration of the study, who will be performing which roles, how you will target and recruit your subjects, plus any incentives or necessary tools and materials

▪ Recruiting

Recruiting is simply locating, attracting, screening, and acquiring research participants.

▪ Participants are good to the extent they represent your target. If participants don’t match your target, your study will be useless. You can learn valuable things by asking the right people the wrong questions. If you’re talking to the wrong people, it doesn’t matter what you ask. Bad participants can undermine everything you’re trying to do.

▪ Shares the concerns and goals of your target users

▪ Embodies key characteristics of your target users, such as age or role

▪ Can articulate their thoughts clearly

▪ Is as familiar with the relevant technology as your target users

▪ In theory, recruiting is just fishing. Decide what kind of fish you want. Make a net. Go to where the fish are. Drop some bait in the water. Collect the ones you want. It isn’t actually that mysterious, and once you get the hang of it, you’ll develop good instincts

▪ In practice, recruiting is a time-consuming pain in the ass. Embrace it. Get good at it and all of your research will be faster and easier, plus this part of the process will get progressively less unpleasant

▪ When designing web applications or websites, the web is a terrific place to find potential test participants

▪ If you happen to have a high-traffic website you can put a link on, that’s the easiest way to draw people in (unless you need to recruit people who have never been to that site

▪ Otherwise you can email a link to a screener—a survey that helps you identify potential participants that match your criteria—or post the screener where it will be visible

▪ Go anywhere you’re allowed to post a message that might be seen by your target users or their forward-happy friends and family. Twitter. Craigslist. Facebook. LinkedIn

▪ If you need people in a certain geographic area, see whether there are local community sites or blogs that would announce it as a service

▪ Referring to it as “design research” rather than “marketing research” goes a long way in the goodwill department.

▪ A screener is simply a survey with questions to identify good participants and filter out anyone who would just waste your time

▪ You can tell a good recruit immediately when you test. Good test participants care. When presented with a task, they get right into the scenario

▪ You could show them a fully functional, whiz-bang prototype and be met with stares and unhelpful critiques. (“Do all the links have to be blue? I find that really dull.”)

▪ The most efficient method of screening is an online survey. (See the Resources section for suggested tools for creating surveys and recruiting participants

▪ What are all of the specific behaviors you’re looking for?

▪ Behaviors are the most important thing to screen for. Even if you’re designing something you believe to be totally novel, current behaviors determine whether your design has a chance of being relevant and intelligible to the participants

▪ If you’re designing an app for cyclists, you need to test the app with people who ride bikes, not people who love bikes and wish they had time to ride.

▪ Be realistic about the amount of skill and comfort you’re targeting. And if you need them to have certain equipment or access to participate, make sure to mention that

▪ To usability-test a mobile app, you need people who are sufficiently familiar with their device to focus on the app’s usability

▪ What level of knowledge about the topic (domain knowledge) do they need

▪ If you’re truly designing something for very general audiences in a familiar domain—say, reading the news—you should verify that they actually do the activities you’re asking about, but you don’t have to screen for knowledge about the subject matter

▪ and prevent professional research participants from trying to read your mind to get the study incentive

▪ If you recruit people from the site you are testing, then just refer to “an interview about this website.”

▪ Asking age, gender, and location allows you to avoid certain biases, but you also need to get at differences in behavior patterns that may have implications for your design

▪ This question serves two purposes: it gauges museum-visiting frequency without giving away the topic of the study, and it offers a way to assess general habits around getting out of the house

▪ At the same time, you should make the screener as short as possible to make it less likely potential participants will bail before they get to the end.

▪ Just like formulating queries in Google, writing screeners and reviewing the results you get makes you better and more accurate at screening

▪ Research makes data. You might have photos, videos, screen captures, audio recordings, and even handwritten notes (the power goes out, but the interview goes on).

▪ A simple interview remains the most effective way to get inside another person’s head and see the world as they do. It is a core research technique with many applications

▪ Once you are comfortable conducting research interviews, you can apply this skill to any situation in which you need to extract information from another person.

▪ Being a good interviewer requires basic social skills, some practice, and a modicum of self-awareness

▪ Introverts might want to start out as observers and notetakers, while extroverts may need to practice shutting up to let the other person talk

▪ In the research lingo, the type of interview covered in this book is a semi-structured interview, meaning that you will have prepared questions and topics, but not a strict script of questions to ask each participant in the same order and manner

▪ This allows more flexibility to respond to the individual perspective and topics that come up. You might find out some very useful things you would have never thought to ask.

▪ Usability testing

Usability testing is simply the process of conducting a directed interview with a representative user while they use a prototype or actual product to attempt certain tasks

▪ The goal is to determine to what extent the product or service as designed is usable—whether it allows users to perform the given tasks to a predetermined standard—and hopefully to uncover any serious, resolvable issues along the way.

▪ If you have a thing, or even a rough facsimile of a thing, you can test it. If your competitor has a thing, you can test that to figure out what you need to do to create a more usable alternative

▪ If you’re about to start redesigning something, usability-testing the current version can provide some input into what works and what doesn’t work about the current version. The test will tell you whether people understand the product or service and can use it without difficulty. This is really important, but not the whole story where a product is concerned

▪ As philosophers would say, usability is necessary, but not sufficient

▪ Usability testing can

▪ Uncover significant problems with labeling, structure, mental model, and flow, which will prevent your product from succeeding no matter how well it functions

▪ Let you know whether the interface language works for your audience

▪ Reveal how users think about the problems you purport to solve with your design

▪ Demonstrate to stakeholders whether the approved approach is likely to meet stated goals

▪ Some people criticize usability testing because aiming for a usable product is tantamount to aiming for mediocrity. But remember, usability is absolutely necessary, even though it is in no way sufficient

▪ If your product isn’t usable then it will fail. However, usability testing won’t absolve you of your responsibilities as a designer or developer of excellent products and services

▪ Usability testing absolutely cannot

▪ Provide you with a story, a vision, or a breakthrough design

▪ Tell you whether your product will be successful in the marketplace

▪ Tell you which user tasks are more important than others

▪ Substitute for QA-testing the final product

▪ We live in the future. There is no reason to test in anything called a “usability lab,” unless there’s a danger your experiment will escape and start wreaking havoc

▪ A usability lab gives you the illusion of control when what you are trying to find out is how well your ideas work in the wild. You want unpredictability. You want screaming children in the background, you want glare and interruptions and distractions.

▪ Just like the corporate VP who is always tweaking the clip art in his presentation slides rather than developing his storytelling skills, it’s easy for researchers (especially us introverted nerds) to obsess about the perfect testing and recording setup rather than the script and facilitation

▪ You can have a very primitive setup and still get a good result, identifying as many usability issues as possible.

▪ Usability issues aren’t preferences and opinions, but issues that make a given design difficult and unpleasant to use

▪ Recruiting and observing or interviewing people one at a time is incredibly valuable. It can also be time-consuming. If it’s just not possible to talk to representative users directly, or if you’re looking for additional background information, you can turn to documented studies by other researchers

▪ Both qualitative studies and surveys can increase your knowledge of the needs and behaviors you should consider.

▪ 5. ANALYZE THE DATA

What does it all mean? Once you have collected the data, gather it all together and look for meaningful patterns

▪ Refer to your initial problem statement and ask how the patterns answer the questions you originally posed

▪ You can use the same qualitative data in different ways and for different purposes. For example, stakeholder interviews might yield business requirements for a redesign and a description of the current editorial workflow that you can use as inputs to the content strategy. Usability testing might indicate issues that need to be fixed, as well as data about current customers that you can use to develop personas.

▪ Returning to data from previous studies can yield new insights as long as the conditions under which they were conducted remain relevant and new questions arise

▪ If you are working with a design team, get as many members as possible involved in the analysis. A group can generate more insights faster, and those insights will be shared and internalized far more effectively than if you simply circulated a report.

▪ Working together to examine specific behaviors and concerns will help your team be more informed, invested, and empathetic with users from the start

▪ It will save time if you give all the participants advance access to the notes or recordings so they can come prepared.

▪ Even if the session includes only one interviewer and one notetaker, it’s useful to follow an explicit structure to make sure that you cover everything and work productively

▪ Here’s a good baseline structure. Feel free to modify it to suit your project’s needs:

▪ Summarize the goals and process of the research. (What did you want to find out? Who from your side participated and in which roles?)

▪ Describe who you spoke with and under which circumstances (number of people, on the phone or in person, etc.).

▪ Describe how you gathered the data.

▪ Describe the types of analysis you will be doing.

▪ Pull out quotes and observations.

▪ Group quotes and observations that typify a repeated pattern or idea into themes; for example “participants rely on pen and paper to aid memory,” or “the opinions of other parents are trusted.”

▪ Summarize findings, including the patterns you noticed, the insights you gleaned from these patterns, and their implications for the design.

▪ Document the analysis in a shareable format

▪ Acknowledge that the goal of this exercise is to better understand the context and needs of the user. Focus solely on that goal.

▪ Respect the structure of the session. Refrain from identifying larger patterns before you’ve gone through the data

▪ Clearly differentiate observations from interpretations (what happened versus what it means).

▪ No solutions. It will be very tempting to propose solutions. Stick to insights and principles. Solutions come next.

▪ Goals (what the participant wants to accomplish that your product or service is intended to help them with or otherwise relates to).

▪ Priorities (what is most important to the participant).

▪ Tasks (actions the participant takes to meet their goal).

▪ Motivators (the situation or event that starts the participant down the task path).

▪ Barriers (the person, situation, or thing that prevents the participant from doing the task or accomplishing the goal).

▪ Habits (things the participant does on a regular basis).

▪ Relationships (the people the participant interacts with when doing the tasks).

▪ Tools (the objects the participant interacts with while fulfilling the goals).

▪ Environment (what else is present or going on that affects the participant’s desire or ability to do the tasks that help them meet their goals).

▪ No matter how rigorous your screening, some outliers may have gotten through. You will know that a participant was an outlier if their behaviors and attributes would rule them out as a target user

▪ There is no reason to accommodate any of Dan’s stated needs or priorities in our design because there is no overlap between his needs and the museum’s goals. So, his data falls outside the patterns we’re looking for.

▪ There will be some people who would never realistically use your product. Don’t try to accommodate them in your model just because you talked to them.

▪ Within a small, closely knit team you can document more informally than if you need results to influence executive decision-making at a larger organization

▪ Given good data, a quick sketch of a persona or a photo of findings documented in sticky notes on a whiteboard in a visible location is far superior to a lengthy written report that goes ignored

▪ Always write up a brief, well-organized summary that includes goals, methods, insights, and recommendations. When you’re moving fast, it can be tempting to talk through your observations and move straight to designing, but think of your future self. You’ll be happy you took the trouble when you want to refer to the results.

▪ The only way to design systems that succeed for imperfect humans in the messy real world is to get out and talk to people in the messy real world

▪ Design doesn’t happen in the deep, cold vacuum of space. Design happens in the warm, sweaty proximity of people with a lot on their minds

▪ A design project is a series of decisions, and making sure the right decisions get made can seem tricky in a complex organization. Take heart. You have more influence than you might think, as long as you take the opportunity to understand the inner workings

▪ It’s inescapable that the nature of an organization matters to the design process. Budgets, approvals, timing, and resource availability can all depend on successfully negotiating an organization. The ultimate success of a product or service depends on how well it fits into everything else the organization is doing and how well the organization can and will support it.

▪ However, researching an organization is very similar to traditional user research and can be incredibly helpful to interactive design and development projects.

▪ Most user-centered design studios interview client stakeholders—people whose jobs or roles will be directly affected by the outcome of the project—as a part of the standard requirements-gathering process. Doing this is essential when you’re coming in cold to work with an unfamiliar organization.

▪ When you run into participants who resist going along, that is often an indicator of some deeper issue worth probing, obliquely and tactfully, of course. The question to answer is why that individual resists your potential scenario.

▪ If you are at a smaller, more nimble organization, such as a very early-stage or rapidly growing startup, the enemies aren’t complexity and stasis. Rather, you may have to contend with the desire to maintain momentum and “fail fast.”

▪ As for how to go about organizational research, it’s pretty straightforward and covers the same principles discussed in the previous chapter. The major difference is that you’re talking to current stakeholders instead of potential customers

▪ The stakeholder concept emerged in a 1963 internal memorandum at the Stanford Research Institute (http://bkaprt.com/jer/7/). It defined stakeholders as “those groups without whose support the organization would cease to exist.” Your research should include anyone without whose support your project will fail. This might include executives, managers, subject matter experts, as well as staff in various roles.

▪ Be generous in your selection. A few additional hours in conversation will help ensure you’re both well informed and protected from an overlooked stakeholder popping up too late.

▪ Executives

The leaders will help you understand the overall company mission and vision and how your project fits into it.

▪ Managers

Managers will frequently be concerned with resource allocation and how your project affects their incentives, monetary or otherwise, and their ability to do the work

▪ Subject matter experts

These are people who have specialized knowledge about the industry or business. You can find them by identifying those design-critical areas where you have the least background knowledge and by asking for introductions. They’ll provide you with essential background information

▪ Staff in various roles

Overlap with the subject matter experts. You will need to balance out the executives with people who do the day-to-day work. In particular, find anyone who has knowledge of the end users. Customer service people and salespeople are as valuable as they are overlooked

▪ Investors and board members

In some organizations the board members are either influential or highly knowledgeable. In others, they are more removed and of less utility. Inquire about level of interest or concern with the project before arranging a conversation

▪ The term stakeholder is a bad bit of jargon, but there really isn’t a better alternative. Think of them as people who could potentially put a sharp stick in your back unless you get them on your side. But don’t fear them! Stakeholder interviews—sitting down individually with people who will be directly affected by the project—have many benefits.

▪ For example, the Fantastic Science Center’s marketing director might actually know a lot about visitor behavior, or know where the organization has been making a lot of assumptions. Or you might find out from customer service that potential museum visitors have been expressing a set of needs that the marketing department doesn’t know about at all. This means you have one great insight about potential visitor needs, and another one about an organizational disconnect

▪ Stakeholder interviews will help you understand the essential structure of the organization, how your work fits into the organization as a whole, and the approval process for various aspects of your project

▪ Organizational politics are a depressing reality in most companies. If you pretend they don’t exist, you’re in for a world of pain. A significant benefit of organizational research is political. You don’t want your hard work to get trampled in a turf war you didn’t know existed

▪ Business requirements are frequently defined as though the project were taking place in a frictionless ideal state, but the application you’re developing doesn’t exist in a vacuum. You have your own reasons for wanting to build or design it in a certain way. Similarly, you need to understand how your work might solve or create problems throughout the organization, and how the organization will prioritize those problems and solutions.

▪ It’s shocking how many projects get underway lacking clear, or even any, business requirements. How do you know whether your work has succeeded? If it’s fully functional? If the users are happy? If your work doesn’t support the business, you have failed, no matter how good the design.

▪ Don’t forget to inquire into technical requirements and take the time to locate anyone who might have particular knowledge about them.

▪ How important is the work to the organization, really? The answer might be surprising. It makes a big difference whether the project at hand is genuinely valued by the organization. At Mule, we have a maxim based on repeated observation: the more important a project is to an organization, the more successful it will be. There might be a higher stress level among people working on an absolutely critical, number one priority project, but you can be more sure that the people working on it will be giving it their full attention.

▪ During interviews, make sure to ask about the typical workday as well as how decisions are made within the team and the organization. This is especially critical if the project at hand brings together cross-functional teams, or teams who have never worked together before, or an internal team and one or more outside vendors.

▪ For the definitive word on making influential people feel heard, I encourage you to read Paul Ford’s excellent essay “The Web Is a Customer Service Medium” (http://bkaprt.com/jer/9/).

▪ “Why wasn’t I consulted,” which I abbreviate as WWIC, is the fundamental question of the web. It is the rule from which other rules are derived. Humans have a fundamental need to be consulted, engaged, to exercise their knowledge (and thus power).

▪ Asking someone for input before you get started is a peerless prophylactic against that person rearing up late in the game with insurmountable objections. Inquiry is flattery. Inviting people to participate empowers them

▪ Take it from the stalkers and internet trolls. Never underestimate the ability of a single individual—no matter how seemingly unimportant or obscure—to really fuck things up for you once they set their mind to it.

▪ What are your assumptions about how the organization functions, about how different disciplines interact, about what the workflow is and how well it’s working, about how much people know or need to know about what you’re doing? Now think of the worst-case scenario if you’re wrong. What happens if marketing doesn’t understand how your work supports the brand, if the salespeople can’t see the benefits, if the production team has no incentive to give you any of their time? This is your opportunity to educate as well as listen, and to get everyone on board.

▪ The purported customers or audience members are not the only users of the product you’re building. Founders may be using it as proof of concept to raise more capital from investors. Salespeople may rely on prospects interacting with it before they can close the deal. Company representatives might expect to be asked questions about it when they’re out at conferences. You’ll benefit from gaining their perspectives and knowing their priorities in that regard.

▪ Don’t wait for people inside the organization to come to you, and don’t rely on a higher-up to tell you who to talk to. Based on the goals of this project, it’s up to you to determine whose input you need.

▪ Similarly, those people will affect what you’re creating. Even if you’re the sole author of an application, you require the participation of others for it to succeed.

▪ You can identify which people or departments will have to put in the time, effort, money, and other resources to cope with the changes. You can learn whether the resources will be available or whether the organization will need to buy more servers or hire more writers

▪ Understanding how what you’re proposing to build relates to the organization responsible for it means that you can anticipate changes to workflow and minimize them where possible, or prepare people for changes when they’re necessary.

▪ Workflow is the set of processes through which complex work gets done. Unless you’re the sole developer of this application, your work doesn’t happen in an organizational vacuum. It has to fit into the work that everyone else is doing. You need to take into account the ways that design and development decisions will affect operations, and make very intentional decisions and recommendations based on that.

▪ Sharpen your tact

Build a wide list of people to interview: founders, executives, managers, technical staff, customer service, etc. Then prioritize it. In addition to people who are directly valuable to the project, you will likely have to speak with some for purely political reasons. This is also an opportunity for learning.

▪ As a rule, and as time permits, it’s best to interview stakeholders individually. The more political an organization, the more important private conversations are to get an accurate picture of the organization

▪ You may have to fight a manager who “just wants to sit in.” This sentiment indicates some combination of fear and curiosity—fear that you’ll be gossiping behind that person’s back, and curiosity about what will be said.

▪ Explain the observer effect—that person’s presence is likely to change the responses—and hold your ground. You’ll need to assure the interviewee that their answers will not be directly attributed and assure the interested parties that they will get all the information they need in an aggregated report.

▪ Email interviews

In a pinch, for a stakeholder who is remote or impossible to get time with, it’s better to send a few key questions via email than not get any information from them at all.

▪ The interviewer should be a calm and confident person, preferably someone who is genuinely very interested in what these people have to say

▪ Have someone else taking notes so that the interviewer can focus on the conversation. You can record the conversation, but this may make the interview subject more nervous about speaking freely

▪ Put the participant at ease and demonstrate respect for their time. Send an agenda and the key questions ahead—not all the questions, but the ones the participant will benefit from knowing in advance. More complex topics might require some forethought. It’s best to avoid making people feel ambushed or unprepared.

▪ Introduce yourself and restate the purpose of the meeting. It should be something like: “We’re starting to work on a complete redesign of the Fantastic Science Center website and we want to get your input. We’ll use your input to make sure that the design meets your needs as well as those of the visitors.”

▪ Explain to what extent the information will be shared, by role or business function. “Please feel free to be totally frank. Honest answers are essential to this process. We’re talking to people throughout the organization, and will group answers together rather than focusing on what one person said. If we use a direct quote, we will not attribute it to you personally.”

▪ Like a good journalist, don’t narc on your sources. Get something in writing from the person directing or approving this research, stating that people can speak freely without fear of reprisal.

▪ Ask questions and let the conversation follow its natural course. It’s very important to keep the interview feeling informal. It’s not an interrogation.

▪ At the end of the interview restate how you’ll use the information and verify the level of the participant’s participation throughout the project. You definitely want to make sure that your expectations match. Make sure that it’s OK to follow up if you need more information or clarification.

▪ Dealing with a hostile witness

It’s in the name. Stakeholders have a personal stake in the process or outcome of the project. They might be in competition for resources, or they might have a larger or smaller workload if the project is successful

▪ Stakeholder interviews tend to be interesting when they go well. People enjoy being consulted and treated as experts. However, sometimes stakeholder interviews take a turn for the ugly. This can be very unpleasant, particularly when you’re interviewing in person. The participant you’re interviewing will turn the tables and start attacking the process, or you personally

▪ Practice, practice, practice. If you’re new to doing these sorts of interviews, practice with members of your team before doing it for real. Have them throw in some challenging or unproductive responses:

▪ “Why are you asking me this?”

▪ “I don’t understand that question. It doesn’t make any sense.”

▪ “I don’t feel comfortable talking to you about that.”

▪ “No one pays attention to anything I have to say, so I don’t know why I should bother talking to you.”

▪ “How much more time is this going to take?”

▪ Create a clear statement of what you need to accomplish for the project to be considered a success by the organization. These are the business requirements. Design and development are how you satisfy the business requirements. It’s best if everyone who cares about the project agrees.

▪ The goal of gathering and documenting business requirements is to ensure that all the stakeholders agree on the purpose and limitations of what you’re doing. You want to increase your chance of success, connect what you’re doing to the goals of the business, increase collaboration, and save costs, particularly those associated with changes. Note that business strategy must remain constant for the duration of a project.

▪ Requirements must be:

▪ Cohesive. The requirements all refer to the same thing.

▪ Complete. No missing information. No secret requirements that didn’t make it onto the list.

▪ make it onto the list. Consistent. The requirements don’t contradict each other.

▪ ontradict each other. Current. Nothing obsolete

▪ Current. Nothing obsolete. Unambiguous. No jargon. No acronyms. No opinions

▪ Feasible. Within the realm of possibility on this project.

▪ Concise. Keeping them short and clear will increase the chances that they are read, understood, remembered, and used. Aim for no more than two to three pages.

▪ The document should not contain specific solutions or design requirements

▪ The type of organization determines the level of detail required in the business requirements documentation

▪ What to include in your documentation

Problem statement and assumptions

What needs to be solved or improved from a business perspective?

▪ Goals

Every project begins with a rough set of goals, or concepts of success. Every individual in an organization sees them a little bit differently. Gathering these and reconciling them is essential to a functioning project

▪ Success metrics

What are the qualitative and quantitative measurements that tell you whether the project is hitting the mark? These should support the goals.

Metrics can include things like “boosts reputation of Fantastic Science Center among peers,” or “increases online sales in the store thirty percent by the six-month point.”

▪ Scope

Scope refers to the amount of work included in any project. “Scope creep” is what happens when more work just keeps getting tacked on and the scope grows uncontrollably. The best way to avoid scope creep is to document what is included in as much detail as possible and in language everyone understands.

▪ Scope is a boundary, so it’s also very useful to note that which any of the stakeholders might assume to be included but is out of scope.

▪ Not touching the logo this time around? Note that! Not changing the registration process? Write it down. Detailed scope documentation makes for happy teams and functional projects.

▪ Risks, concerns, and contingency plans

Want to increase your chances of project success? Then acknowledge anything you’ve uncovered that might lead to failure or unmet expectations

▪ Some organizations are more functional and well resourced than others. Every organization has its challenges. If the team understands and acknowledges these, they will be able to work around them more effectively.

▪ Maybe key decision-makers will have limited availability. Or perhaps two departments who need to collaborate very closely have historically had a poor working relationship

▪ If these challenges are not openly acknowledged, which is sometimes the case, be very sensitive in how you talk about them. For your work to succeed, you have to address them.

▪ Verbatim quotes

The specific words used are highly valuable in revealing an individual’s personal perspective and attitudes. If possible, share these without attaching identifying information.

▪ Unpack the baggage

A solid understanding and honest assessment of an organization and its business is necessary for the success of any significant design project.

▪ Organizational habits and capabilities are just as relevant as target user behaviors and needs, although they’re less frequently included as fundamental topics of design research. And the true nature of workflow and interpersonal relationships is just as ripe for ethnographic exploration.

▪ Even just the process of conducting research can be beneficial if only because it provides the motivation to open atrophied or nonexistent communication channels

▪ Every delightful and every frustrating artifact that exists, exists because of a series of design decisions.

▪ Design as a job is similarly delightful and frustrating. Whatever you create has to work for a diverse array of people who may not be anything like you. Your work must be sufficiently novel to attract attention while fitting into each user’s existing world of objects and situations over which you have no control. How do you create one design that solves a problem for an endless combination of people and environments?

▪ You do user research to identify patterns and develop empathy. From a designer’s perspective, empathy is the most useful communicable condition: you get it from interacting with the people you’re designing for.

▪ When we talk about user research as distinguished from usability testing, we’re talking about ethnography, the study of humans in their culture

▪ We want to learn about our target users as people existing in a cultural context. We want to understand how they behave and why. This is very different from gathering opinions. It isn’t just surveying or polling. And it’s definitely not focus groups.

▪ Understand the true needs and priorities of your customers/readers/target audience/end users.

▪ Understand the context in which your users will interact with what you’re designing.

▪ Replace assumptions about what people need and why with actual insight

▪ Create a mental model of how the users see the world.

▪ Create design targets (personas) to represent the needs of the user in all decision-making

▪ Hear how real people use language to develop the voice of the site/application.

▪ For you to design and develop something that appeals to real people and reflects their priorities, you’ll need to talk with or observe representative users directly in their context—their regular environment

▪ This reduces your risk of making bad assumptions based on your own experiences, hopes, or subjective preferences. That context includes the physical environment, mental model, habits, and relationships.

▪ Physical environment

This is the physical context in which someone will use your product or service. This could be at their office at a desk, at home on the sofa, at home at a kitchen table, or outside in an unfamiliar city. Is your target user likely to be alone, or surrounded by others, subject to interruptions? Needs can change vastly with setting.

▪ Mental model

A mental model is an individual’s pre-existing internal concept of and associations with any given institution, system, or situation. Every one of us has an imperfect, idiosyncratic map of reality in our head. Without it, we would be utterly lost. With it, we rely on assumptions based on previous experiences we consider analogous. The better the analogy, the more useful the map. This is why interfaces that strive for novelty are often unusable. With no hooks into an existing mental model, we have to figure things out from scratch.

▪ What do they expect, and how do those expectations make it more or less likely that they would interact with the website in the ways we want them to?

▪ Habits

This includes habits of body and mind. How does the user already solve the problem you are trying to solve for them, if indeed they do?

▪ We frequently hear from entrepreneurs that are trying to create a habit around a new product. Habits are hard to change, as anyone trying to kick one will attest, but inserting a new hook into an existing habit is much easier.

▪ For a science museum website, relevant habits might include their current weekend activities, or the sources of information they rely on for entertainment, education, events, or keeping up to date on technology changes

▪ Relationships

Social networks are merely the most obvious intersection of human relationships and digital products. People are social animals and every interactive system has an interpersonal component. Your product or service will exist within a web of human relationships.

▪ ASSUMPTIONS ARE INSULTS

There are about seven billion people on the planet. As of this writing, about two-thirds of them have no internet access. Wrap your head around that.

▪ When you make assumptions about your users, you run the risk of being wrong. When you embed wrong assumptions in the design of your product or service, you alienate people—possibly before they even have a chance to hear what you have to offer. The more obvious that wrong guess is, the more annoying it is.

▪ Your assumptions about the age, gender, ethnicity, sexuality, and physical or cognitive abilities of your users might lead to barriers you don’t actually intend—barriers that don’t serve your business goals or ethics.

▪ Every product doesn’t need to be all things to all people. However, every design decision should be a well-informed, intentional one that welcomes your intended users rather than pushing them away or making them feel bad. That’s why identifying and understanding your target audience or user base—to the point of true empathy—is the most important and useful design research you will do.

▪ It seems like a simple formula:

If your goal is to make things people buy and use, you should design what people want. To do that, you need to know what people want. So just find some people and ask them what they want. Then go off and make what they tell you.

▪ No. This does not work

▪ The first rule of user research: never ask anyone what they want

▪ Your challenge as a researcher is to figure out how to get the information you need by asking the right questions and observing the right details.

▪ The questions you ask directly and the questions you want answered are two different sets of questions

▪ Similarly, to create a good fit between what you’re designing and what your target users need, you have to know about the aspects of their habits, behaviors, and environment that are relevant to your work, and then turn that knowledge into insights you can act on

▪ WHAT IS ETHNOGRAPHY?

Ethnography is a set of qualitative (descriptive rather than measurable) methods to understand and document the activities and mind-sets of a particular cultural group who are observed going about their ordinary activities in their habitual environment.

▪ Radically simplified, the fundamental question of ethnography is, “What do people do and why do they do it?”

▪ In the case of user research, we tack on the rider “...and what are the implications for the success of what I am designing?”

▪ Humans and their habits and material culture are endlessly complex. Ethnography is an equally deep and nuanced field

▪ The practices outlined in this chapter are merely a pragmatic simplification of a few core ideas intended to help you apply useful insights about people to your product design.

▪ Deep dive

You want to get to know a small but sufficient number of representative users very well

▪ Walk in their shoes, live in their skins, see through their eyes...choose the creepy spiritual possession metaphor that works for you.

▪ Daily life

Fight the urge for control and get into the field where things are messy and unpredictable

▪ As you’re probably well aware from how your day is going so far, life for everyone is messy and unpredictable in ways both good and bad. It’s very easy to think up ideal scenarios in which everything is smooth and simple. These are as useful to your work as a game of SimCity is to allocating actual resources in New York City.

▪ Everyone’s behavior changes with the context and the circumstances. Soak in your subject’s actual environment. It’s of limited utility to learn how people behave in your conference room. No one is going to act naturally in there.

▪ Personas will keep you honest. You design for them, not for you or for your boss.

▪ INTERVIEWING HUMANS

The goal of interviewing users is to learn about everything that might influence how the users might use what you’re creating

▪ Good interviewing is a skill you develop with practice. The great myth is that you need to be a good talker. Conducting a good interview is actually about shutting up. This can be very hard, especially when you’re enthusiastic about the topic.

▪ When you’re interviewing someone you know nothing. You’re learning a completely new and fascinating subject: that person.

▪ The interview guide should contain:

▪ The brief description and goal of the study. This is for you to share with the participant and use to remind yourself to stay close to the topic.

▪ The basic factual or demographic questions for putting the participant’s answers in context. These will vary depending on the purpose of the interview, but often include name, gender, age, location, and job title or role.

▪ A couple of icebreaker or warm-up questions to get the participant talking. Most people know this as “small talk.” Feel free to improvise these based on the demographic information

▪ The questions or topics that are the primary focus of the interview.

▪ You should also gather a bit of background information on the topic and people you’ll be discussing, particularly if the domain is unfamiliar to you. Talking to homeowners about how they selected their mortgage brokers?

▪ Interview structure: three boxes, loosely joined

An interview has three acts, like a play or a spin class: the introduction and warm-up, the body of the interview, and the conclusion.

▪ Introduction

Introduce yourself with a smile, expressing genuine gratitude that the person you are interviewing has taken the time to talk (even if they’re getting a large incentive and especially if it’s a busy staff member who has taken time out of their workday).

▪ Describe the purpose of the conversation and the topic without going into so much detail that you influence the answer

▪ Explain how the information will be used and shared. Obtain their explicit permission to record the conversation.

▪ Ask whether they have any questions about the process

▪ Move on to the demographic information or facts you need to verify. Use the collection of this information as the basis for the warm-up questions.

▪ “Oh, you live in San Diego. What do you like to do for fun there?”

▪ Once you’ve covered the formalities and pleasantries, it’s time to dig into the interview meat

▪ Ask open-ended questions that encourage the subject to talk, not closed questions that can be answered with “yes” or “no.” (Closed question: “Do you communicate with the marketing department often?” Open question: “Tell me about the internal groups you communicate with as part of your job.”)

▪ If the subject doesn’t offer enough information on a topic, ask a follow-up or probing question, such as “Tell me more about that.”

▪ Use your list of questions more as a checklist than as a script. If you read the questions verbatim, you’ll sound like a robocall survey

▪ Conclusion

Once you have the information you were looking for, and hopefully even more, make a gentle transition to the wrap-up. Say something like “That’s it for my questions. Is there anything else you’d like to tell me about what we discussed?”

▪ Thank them for their time and cover any administrative topics such as incentives or next steps on the project.

▪ Don’t be afraid to shut it down early if you find yourself in an unproductive interview situation

▪ It happens and the best thing you can do is move on to the next one. There is no rule that says you need to hang in until you’ve attempted to have every single one of your questions answered.

▪ You, the interviewer, play the dual role of host and student

▪ The more comfortable a participant feels, the more and better information you will get. A relaxed participant will open up and be more honest, less likely to worry about putting on a good impression.

▪ Think of them as the world’s foremost expert on themselves, which is the all-absorbing matter at hand. Insert yourself only when necessary to redirect back on topic or get clarification. You will know when your interview is going particularly well because you won’t be able to get a word in, but you will be getting answers to all your questions.

▪ Practice active listening

As long as you’re breathing, make interested mm-hmm sounds

▪ If you’re interviewing in person, make sure to look at the speaker directly and nod

▪ Unrelated thoughts might start to pop up, especially if an answer goes on at length. Stay alert and focused on the other person.

▪ Keep an ear out for vague answers

You want details and specifics. Always be ready to bust out a probing question such as “Why is that?” or “Tell me more about that.”

▪ Handy checklist

This checklist for effective user research was adapted from the Ethnography Field Guide produced by the Helsinki Design Lab, powered by Sitra, the Finnish Innovation Fund (http://bkaprt.com/jer/10/):

▪ Create a welcoming atmosphere to make participants feel at ease

▪ Always listen more than you speak

▪ Take responsibility to accurately convey the thoughts and behaviors of the people you are studying

▪ Conduct your research in the natural context of the topic you’re studying

▪ Start each interview with a general description of the goal, but be careful of focusing responses too narrowly

▪ Encourage participants to share their thoughts and go about their business

▪ Avoid leading questions and closed yes/no questions. Ask follow-up questions

▪ Prepare an outline of your interview questions in advance, but don’t be afraid to stray from it

▪ Whenever possible, snap photos of interesting things and behaviors

▪ Also note the exact phrases and vocabulary that participants use

▪ Pay attention after you stop recording. You might get a valuable revelation.

▪ Tell me about your job. Walk me through a typical week in your life. How often are you online? What computers or devices do you use? When do you use each of them? Do you share any of them? What do you typically do online? What do you typically do on your days off? How do you decide what to do? Tell me about how your children use the internet. How do you decide what to do on your days off with your kids? What are your particular non-work interests? What do you read online besides the news? How frequently do you visit museums in your town? Which ones? What prompts you to go?

▪ The interview is the basic unit of ethnographic research

▪ Once you’ve completed your interviews, analyze them all together to find themes, including user needs and priorities, behavior patterns, and mental models

▪ If you are doing generative research, look to the needs and behaviors you discover to point out problems that need solving

▪ Turn the clusters around user types into personas that you can use for the life of the product or service you’re working on.

▪ Once you’re comfortable doing ethnographic interviews, you can take your skills into the field

▪ You enter the participant’s actual environment and observe as they go about the specific activities you’re interested in studying. By doing this you will be able to see actual behaviors in action and learn about all of the small things you might not hear about in an interview, such as a janky work-around so unconscious and habitual the individual has completely forgotten it.

▪ Contextual inquiry is a deeper form of ethnographic interview and observation. It is particularly useful for developing accurate scenarios, the stories of how users might interact with potential features, as well as identifying aspects of the user’s environment that will affect how someone might use a particular product.

▪ He would quite literally hang out in Staples office supply stores waiting for someone to purchase Quicken, and then follow them home to observe them using the software. He learned where they had difficulty setting up the program, which allowed him to make improvements to the initial experience.

▪ Things to keep in mind

▪ Travel. Allow plenty of time to get to the site and set up

▪ Get situated. Find a comfortable spot that allows you to talk to the participant without interrupting their normal routine

▪ Interview. Establish trust and learn about what you will be observing. Find out when it will be least disruptive to interrupt and ask questions

▪ Observe. It’s a show. You’re watching. Note everything in as much detail as possible. The relevance will be apparent later. Pause to ask questions. Stay out of the way.

▪ Summarize. Conclude by summarizing what you learned and asking the participant to verify whether your observations were correct

▪ Note: even if the participant disagrees with your assessment, you might still be correct, and the contradictory description is a very interesting data point.

▪ Contextual inquiry can be very inspirational. You might observe problems and opportunities you had no idea existed and open the door to some innovative and interesting ideas

▪ FOCUS GROUPS: JUST SAY NO

▪ A handful of “ordinary” people around a conference table engaged in a lively discussion about how various brands make them feel. A cheerful, yet authoritative moderator.

▪ Observers wielding clipboards behind a two-way mirror. Focus groups are synonymous with qualitative research in popular culture, and it isn’t uncommon to hear all user research lumped as “focus groups.”

▪ Unlike the interviews and contextual inquiry mentioned above, focus groups don’t provide insight into behavior or the user’s habitual context. But because they’re so common, it’s worth mentioning them.

▪ Focus groups evolved from the “focused group interview” developed by American sociologist Robert K. Merton. (Fun fact: he also coined the terms “role model” and “self-fulfilling prophecy”; http://bkaprt.com/jer/12/).

▪ Merton himself deplored how focus groups came to be misused. As he said, “Even when the subjects are well selected, focus groups are supposed to be merely the source of ideas that need to be researched” (http://bkaprt.com/jer/13/).

▪ Focus groups are the antithesis of ethnography. Unlike interviewing participants individually or observing people in their natural environment, the focus group creates an artificial environment that bears no resemblance to the context in which what you’re designing would actually be used.

▪ The conversation is a performance that invites social desirability bias and gets in the way of finding out what people need and how they behave outside of this peculiar group dynamic. Participants are more likely to proclaim or conceal behaviors for the benefit of those around them.

▪ Recruiting and screening participants is the most time-consuming and least informative aspect of user research. If you are doing a focus group, one bad recruit in the group can tank the entire session. In one-on-one interviews, at least that recruit won’t taint the pool.

▪ There may be some group activities that might yield useful insights as part of the design process

▪ However, focus groups are simply research theater. And your research time is too precious to squander on a sideshow

▪ Accept no substitute for listening to and observing real people who need to do the things you’re designing a thing to help people do

▪ Who is the competition?

“No one! No one is doing anything that even comes close to what we are doing!” “The top five companies by market share in our vertical.” “The first page of search results for ‘[relevant term]’ on Google. All of those guys.”

▪ The hardest competitor to beat is the one your potential customers are using right now. If they have to stop using that one to start using yours, they may incur a switching cost. People are lazy, forgetful creatures of habit. Your target customers have to love you more than they hate change.

▪ This is not the place for wishful thinking. It’s a jungle out there, a hostile and constantly changing ecosystem, and you want the thing you’re building to have the best chance to adapt and survive—like the creature from Alien, but with a more pleasant user interface. You need to know the landscape and the competition.

▪ By taking a hard look at the role you want to play in your target customer’s life and the advantages and disadvantages that affect your ability to do so.

▪ Competitive research begins with a broad perspective on the competition. You may be looking for things to steal, like approaches and customers. You need to see how other people are solving similar problems, and identify opportunities to offer something uniquely valuable.

▪ You need to do this frequently and quickly; get in the habit of constantly asking not only “What matters to our customers?” (the user question) but “How are we better at serving that need than any competitor?” (the product question) and “How can we show our target customers that our product is the superior choice?” (the marketing question).

▪ When you look at what your competitors are doing, you only see what is visible on the outside, unless you have a mole. That’s what your users see as well, so user research won’t help you here. It will take some deeper digging, critical thinking, and extrapolation to determine (or make a good guess at) why your competitor is doing things a certain way.

▪ Albert S. Humphrey was a management consultant who devised something called SWOT analysis (http://bkaprt.com/jer/14/): strengths, weaknesses, opportunities, and threats

▪ Your work with your own organization (or the research you’ve done into your client’s organization) should have provided you with a good sense of your (or their) internal strengths and weaknesses.

▪ Once you’ve enumerated these characteristics, you can identify the aspects of the user experience that serve to amplify strengths and exploit opportunities as well as those that mitigate weaknesses and counteract threats.

▪ If you do competitive research and your competitor doesn’t, you have an advantage.

▪ Once you have identified a set of competitors and a set of brand attributes, conduct an audit to see how you stack up. In addition to those organizations you think of as competitors, conduct a web search to see who else comes up

▪ Add in any product or service that was mentioned repeatedly in user interviews and anyone you admire as a leader solving a similar type of problem.

▪ For each competitor and each site, product, service, or touchpoint, answer the following:

▪ How do they explicitly position themselves? What do they say they offer?

▪ Who do they appear to be targeting? How does this overlap or differ from your target audience or users?

▪ What are the key differentiators? The factors that make them uniquely valuable to their target market, if any?

▪ To what extent do they embody each of your positive/negative attributes?

▪ How do the user needs or wants they’re serving overlap or differ from those that you’re serving or desire to serve?

▪ What do you notice that they’re doing particularly well or badly?

▪ Based on this assessment, where do you see emerging or established conventions in how they do things, opportunities to offer something clearly superior, or good practices you’ll need to adopt or take into consideration to compete with them?

▪ Your brand is simply your reputation and those things that signify your identity and reputation to your current and potential customers

▪ That reputation offers a promise of all the good things you do for your customers, most of which exist only in the customer’s mind. The stronger the brand, the more awesome associations pop up in more people’s minds.

▪ Coca-Cola is a phenomenal brand, producing positive emotional associations across the globe on a product that is fundamentally caffeinated sugar water. Tremendous continuous effort goes into brand marketing. You probably don’t need that.

▪ For many interactive products and services, there is no “brand” apart from the service itself. The brand experience is the user experience. The visual design of the interface is the brand identity. The brand personality is the voice of the interface language.

▪ Here are the questions you need to ask about your brand:

▪ Attributes: which characteristics do you want people inside and outside the company to associate with the brand or product, and which do you want to avoid?

▪ Value proposition: what does your product or service offer that others do not and how does your brand communicate this?

▪ Customer perspective: when you conduct ethnographic interviews with existing or potential customers, what associations do they have with your brand?

▪ All of this is important to keep in mind as you do a competitive brand analysis. Make sure you’re comparing apples to apples, not Starbright Cleaners to Apple.

▪ The name is the single most important aspect of a brand. What makes a good name varies from market to market like everything else, but at a minimum, it needs to be unique, unambiguous, and easy to spell and say

▪ The logo is simply the illustrative manifestation of your brand, which can take several forms: wordmark, bug, app icon, favicon, etc. Which logo you choose and how much you spend on it depends on the contexts in which people are going to have to identify your stuff and distinguish it from your competitors.

▪ Native mobile apps represent a new level of challenge, since the logos are so constrained in size and dimension and do have to work very hard in that small uniform space to help a user distinguish one app from another. You don’t want to look at your phone’s desktop and get confused about which icon opens which program.

▪ To conduct an effective logo assessment, list all of the contexts in which the target users are likely to encounter it, and review your competitors’ logos in the same contexts

▪ Don’t just test your own product—test the competitor’s! You can use task-based usability testing (described in Chapter 7) to evaluate a competitor’s website or application

▪ This allows you to understand their strengths and weaknesses directly from the user’s point of view, identify opportunities to develop your advantages, and gain insight into how target users conceptualize core tasks and key features.

▪ The competitive landscape and how what you’re designing fits into it may be the fastest moving target of all research topics. New options are appearing and product categories are collapsing every day.

▪ “Within 30 minutes I realized, Oh my God, it’s broken. Holy shit, we totally fucked up.”

—Bill Nguyen, founder of photo-sharing service Color (http://bkaprt.com/jer/15/)

▪ Your initial forays into clarifying requirements, understanding users, and checking out the competition helped you think up an appropriate design solution

▪ Evaluation is assessing the merit of your design. It’s the research you never stop doing. There are several ways to go about it, depending on where you are in the project.

▪ In the early stages, evaluation takes the form of heuristic analysis and usability testing. You can test an existing site or application before redesigning. If you have access to a competitor’s service or product, you can test that. You can test even the very earliest sketches.

▪ Once a site or application is live, even if it’s in private alpha, you can start looking at quantitative data and use site analytics to see how people are actually interacting with the system and whether that meets your expectations.

▪ The best way to assess a functional design is through a combination of quantitative and qualitative methods. The numbers will tell you what’s going on, and the individual people will help you understand why it’s happening.

▪ Despite the fancy name (which is from the Greek heuriskein, to find out), heuristic analysis is the most casual method of evaluating usability

▪ Heuristic” in English simply means “based on experience”; a heuristic is a qualitative guideline, an accepted principle of usability. The more you know about using and designing interactive systems, the better you’ll be at heuristic analysis.

▪ Godfather of usability Jakob Nielsen and his colleague Rolf Molich came up with the idea for heuristic analysis way back in 1990 (http://bkaprt.com/jer/16/). The method is very simple: evaluators (at least two or three, ideally) individually go through a site or application with a checklist of principles in hand and score the site for each one.

▪ Nielsen’s ten heuristics (http://bkaprt.com/jer/17/) are

▪ System status visibility. The system should provide appropriate feedback

▪ Match between system and real world. Use language familiar to the user and follow conventions.

▪ User control and freedom. Provide emergency exits, undo, and redo.

▪ Consistency and standards. Things that appear the same should behave the same.

▪ Error prevention. Don’t just let users escape from errors: help users avoid them.

▪ Recognition rather than recall. Options should be visible. Instructions should be easy to find. Don’t make the user have to remember information.

▪ Flexibility and efficiency of use. Support shortcuts for expert users.

▪ Aesthetic and minimalist design. Avoid providing irrelevant information.

▪ Help users recognize and recover from errors. Error messages should be helpful.

▪ Help and documentation. Ideally, the system should be usable without documentation, but help should still be available and task oriented.

▪ Several of these heuristics focus on error prevention and recovery, which remains the most neglected area of system design

▪ Every time an application displays “Unknown Error” or an unhelpful error code number with no instruction, you know someone should have done a little heuristic evaluation.

▪ The advantage of heuristic analysis is that it’s a quick and cheap way to identify potential issues. You don’t need to recruit users. You can just get two colleagues to sit down and do it in an hour. It’s a good way to deal with obvious issues in early prototypes before bringing in users.

▪ The downside is that it’s very simplistic and might not catch all the issues that would come up in context. Less experienced evaluators may not see all the problems. Different evaluators will find different issues. Some expert evaluators may find issues that don’t present a problem to actual users. It focuses on the system itself rather than the relationship between the user and the system. The advantages were greater back in the day when fewer people were familiar with technology and recruiting people was much more difficult.

▪ Heuristic inspection is not a substitute for usability testing, but it can be a good sanity check. The number of sites and applications that launch with major usability flaws is evidence of its continued usefulness.

▪ Usability is the absolute minimum standard for anything designed to be used by humans. If a design thwarts the intended users who attempt the intended use, that design is a failure from the standpoint of user-centered design

▪ Despite the vast amount of knowledge we possess about usability, unusable objects are all around us: the completely unintelligible “universal” remote, the spiteful web form that discards every piece of entered data, the deceptive door that only appears to open outward until you walk into it

▪ The more complex a system is to design and build, the more work is required to ensure that it’s usable—but that work is always worth doing

▪ (This is also an argument for keeping feature sets simple.) If the desire to rush to market trumps usability, you might see your first mover advantage dissolve as soon as a competitor copies all your functionality and leapfrogs your ease of use.

▪ According to Nielsen (http://bkaprt.com/jer/18/), usability is a quality attribute defined by five components

▪ Learnability: how easy is it for users to accomplish basic tasks the first time they come across the design?

▪ Efficiency: once users have learned the design, how quickly can they perform tasks?

▪ Memorability: when users return to the design after a period of not using it, how easily can they reestablish proficiency?

▪ proficiency? Errors: how many errors do users make, how severe are these errors, and how easily can they recover from the errors?

▪ Satisfaction: how pleasant is it to use the design?

▪ Cheap tests first, expensive tests later

▪ Usability testing can be more or less expensive. Don’t use expensive testing—costly in money or time—to find out things you can find out with cheap tests

▪ Find out everything you can with paper prototypes or quick sketches before you move to a prototype. Find out everything you can in the comfort of your own office before you move into the field. Test with a general audience before you test with specific audiences who take more time and effort to find.

▪ In fact, start even earlier than that. Test a competitor’s product before you even put pencil to paper. Then you should test some sketches. And then test at every stage as much as you can allow.

▪ You can test every two weeks in conjunction with development sprints, if that’s how you roll. I’m not going to tell you when to do usability testing in your design and development cycle, but I will tell you when not to do it: right before you are about to launch.

▪ The second most expensive kind of usability testing is the kind that you put off until very late in the process, when you risk finding out that there are huge usability problems that will be very difficult to fix.

▪ The most expensive of all is the kind your customers do for you after launch by way of customer service.

▪ The most difficult part of usability testing is determining how it fits into your process as a decision-making input

▪ Build usability practices into your workflow from the start, the same way you account for internal reviews of work in progress

▪ Create a testing process and checklist that includes all of the information and equipment you need

▪ Always be recruiting.

▪ Maintain a database, even just a Google doc, of potential participants and their contact information

▪ Decide who’s in charge of this stuff. A point person makes everything operate more smoothly

▪ What you will need

A plan. A prototype or sketch. Four to eight participants of each target user type based on personas (ideally) or marketing segments. A facilitator. An observer. One or more methods of documentation. A timer or watch.

▪ Ideally you have personas that you have been using throughout the design process and you can use them and their core tasks as a jumping off point for usability.

▪ The features you want to test should likewise have associated scenarios and tasks. For each feature, write a very brief story that offers background on how the user arrived there and what they are trying to accomplish.

▪ For example, if you wanted to test the new ticket purchase process design, you might use the following scenario: a science-minded school friend is in town for the weekend and wants to visit the Fantastic Science Center. You did a web search and landed on this page. What would you do to find and purchase tickets for the weekend your friend is visiting?

▪ Not all tasks are created equal. When you go into a usability test, you should have a clear idea which failures are a bigger deal.

▪ If a user can do anything at all on your site, they need to be able to successfully give you money.

▪ For websites with the goal of marketing a physical location, such as the Fantastic Science Center, finding the address and operating hours is generally the most essential task.

▪ Once you have your tasks, make a checklist test plan that you use to run and document each round of testing. The test plan lays out what you’re going to do, how you’re going to conduct the test, which metrics you’ll capture, the number of participants you’re going to test, and which scenarios you’ll use

▪ Helpfully, the US Department of Health and Human Services maintains usability.gov, which is a resource for making useful and usable websites

▪ This checklist can be used for both planning the test and writing your report. Modify it to fit your needs:

▪ Objectives. Subject of the test: what are you testing and what state is it in? Methodology. Participants and recruiting. Procedure. Tasks. Usability goals. Completion rate (the percentage of tasks the user was able to complete). Error-free rate (the percentage of tasks completed without errors or hiccups).

▪ Participants are the fuel that makes usability tests go, and they are single use, so you need a good supply of them. You can bring people back to see if your improvements have really improved things for them, but they might be tainted—influenced by their previous experience with your design—and won’t necessarily give you an accurate depiction of how someone is going to approach this system for the first time.

▪ Recruiting for usability testing is substantively the same as for ethnographic interviews

▪ It is essential that the people you select for the test share some key goals with your target users. Otherwise, they won’t be able to immerse themselves sufficiently in the scenarios you give them.

▪ This requires a balance of sociability and self-awareness. Making small talk is fine and helpful up front. Once the test starts, you’ll need some self-control so you don’t intervene.

▪ It’s one of those things that gets easier with practice.

▪ In particular, avoid leading the user and helping them when they get lost. Embrace the uncomfortable silences.

▪ Frequently, users who encounter a usability issue are quick to blame themselves rather than the system. This is how people have been conditioned by frequent exposure to less than usable products. If this happens, ask the participant to describe how they expected the system to work and why they had that expectation.

▪ Be very honest with your team about who should be facilitating. If you don’t have a good facilitator on your team, you can always contract with someone or try to get a volunteer from another department. And again, practice

▪ Even if you are set up to record, it’s very important to have a second person observing the tests and taking notes. This allows the facilitator to be responsive and the observer to be as observant as possible, creating the smallest number of distractions.

▪ Audio recording is fantastic.

▪ Screen capture with an audio track is very useful and relatively easy

▪ If you add an additional camera on the participant’s face and body, this can be helpful, but you have to ask yourself whether the additional information is worth the additional overhead.

▪ Consider having the notetaker snap a few photos; accurately time-stamped photos combined with audio recording might be sufficient to capture information like how the user was holding their phone when they were experiencing that difficulty.

▪ Usability testing applications on mobile devices is a free-for-all right now, so it’s a terrific place for innovation. There is a great need for evaluating the usability of mobile interfaces, particularly in their context of use (walking around outside, rather than seated in your conference room), but there is no one clear, comfortable way to both observe the user over the shoulder and capture the activity on the user’s screen.

▪ MailChimp’s solution to this conundrum, which they detail on their blog (http://bkaprt.com/jer/19/), is to have a user set up a video chat on a MacBook and then hug it from the back so the iSight camera catches video of their interaction on the phone and the audio through the microphone (Fig 7.1).

▪ Fig 7.1: A little awkward, but effective—remote test mobile usability by having participants hold their devices in front of a laptop webcam

▪ The participant’s reaction to the task.

▪ How long it takes to complete the task.

▪ If the user failed to complete the task

▪ Any terminology that presented a stumbling block.

▪ The most important items to note are areas where the user exhibited nonverbal frustration, verbatim quotes, and any features that were particularly successful or unsuccessful. If the notetaker can manage an approximate time code, that will make analysis easy.

▪ Eye-tracking measures where someone is looking, how long, and in what direction. Observation and analytics can tell you where a user taps with a finger or hovers with a mouse, but where that user directs their gaze is a mystery only a non-trivial amount of cash can reveal. Whether paying top dollar for this data is worthwhile remains a deeper mystery still.

▪ The only case where this could be worthwhile would be testing with populations who have trouble articulating what is drawing their attention on the page.

▪ The aim of usability testing is to identify specific significant problems in order to fix them. The outcome is essentially a ranked punch list with a rationale

▪ Rate each problem users encountered during the test on each of the following two scales: severity and frequency. You must look at both to ensure you’re prioritizing real obstacles, rather than chasing a fluke.

▪ Severity:

High: an issue that prevents the user from completing the task at all. Moderate: an issue that causes some difficulty, but the user can ultimately complete the task. Low: a minor problem that doesn’t affect the user’s ability to complete the task.

▪ Frequency:

High: 30% or more participants experience the problem. Moderate: 11–29% of participants experience the problem. Low: 10% or fewer of participants experience the problem.

▪ Once you’ve conducted the tests, and rated the issues, sort them into three tiers. Each represents the combination of severity and frequency. Also take into account how core the related task is to your application (for example, confusion over changing a profile picture may be less core than obstacles to entering payment information).

▪ Need to convince someone before you can make any changes? Watching actual users struggle with the system is more convincing than reading a report, and offers all the agitation of a suspense film.

▪ Verbatim quotes and video clips of failure presented in conjunction with a report can also be effective.

▪ Just make sure to connect the tasks you tested and the problems you found to high-priority business goals.

▪ To conduct a benchmark usability study, identify a small common set of tasks to test across your website and those of your competitors

▪ Use a common scoring system across all sites and tasks to identify which of the competitive group was most usable overall, and most usable per key task.

▪ Following a redesign, you can run the study again to verify improvement relative to competitors.

▪ You take all this messy data and begin to organize it, and group it, and label the groupings. Through conversation, clarity will start to emerge. Clarity in the data analysis will translate to clarity of concept, content relationships, navigation, and interactive behaviors. And best of all, if you work collaboratively that clarity and deep understanding will be shared.

▪ Any models or maps you create will simply serve as documentation of what everyone already knows.

▪ Closely review the notes.

▪ Look for interesting behaviors, emotions, actions, and verbatim quotes

▪ Write what you observed on a sticky note (coded to the source, the actual user, so you can trace it back).

▪ Group the notes on the whiteboard

▪ Watch the patterns emerge

▪ Rearrange the notes as you continue to assess the patterns.

▪ You will end up with a visual representation of your research that you can apply toward your design work in a few different ways

▪ Your first pass—and if you don’t have a lot of time, your only pass—should be to extract general design mandates from your interviews. Then you can prioritize those mandates based on business goals. This also requires the least diagramming skill

▪ An affinity diagram helps turn research into evidence-based recommendations.

▪ The participants in the analysis build clusters of related observations. Once a cluster starts to take shape, you can extract the insights and the overarching mandate or recommendation.

▪ The diagram itself can be a handy visual reference or a tool for communicating with a larger team about your research and the principles you’ve uncovered

▪ As you review the notes or recordings, write down anything interesting you observed on a sticky note. An observation is a direct quote or objective description of what the user did or said.

▪ Pull out all of the particularly interesting quotes. Flag those that seem to represent the particular needs of each user type. These will be useful for your personas.

▪ Also note the vocabulary that participants used to describe their goals and the elements of the tasks or systems you are working with, particularly if they differ from those used in your organization.

▪ Note all stated or implicit user goals.

▪ Implicit goals can be found in quotes or actions that indicate a particular desire. For example, starting the weekend with some good activities in mind. In particular, flag those you didn’t anticipate, but that your product might readily meet.

▪ Example observations:

▪ “I reset my password every time I visit the website because I never remember it.

▪ Participant’s four-year-old daughter interrupted three times during the thirty-minute interview.

▪ “I take care of the kids for the whole day every Saturday to give my partner some alone time.”

▪ Participant reports checking email on her phone every morning before getting out of bed.

▪ Example goals:

▪ “I like to start the weekend with some good activities in mind.”

▪ “I want my kids to keep learning even when they’re not in school.”

▪ Name the pattern and identify the user need that emerges from it, such as “Needs reminders for organized activities.”

▪ “I see signs around town for events that look interesting, but I never remember before it’s too late.”

▪ it’s too late.” “The week rushes by and then I wake up on Saturday morning with no good ideas.”

▪ The final step of the analysis is to identify the actionable design mandate or principle.

▪ When announcing a new exhibit, offer the ability to sign up for a reminder

▪ Allow members the option of digital access to all services (e.g., online member newsletter instead of print, email guest passes to their friends).

▪ Improve promotion of and navigation to activities and lesson plans

▪ Create a stronger voice for the museum based on the quality of its scholarship and expert status (e.g., offer the museum perspective alongside the feed of technology news).

▪ In addition to serving as a useful input to other tools, such as personas, and a nifty visual representation of your research and analysis, the affinity diagram helps you make decisions. You can decide which features and functionality to prioritize based on the patterns of needs you recognize. You can decide to do additional research based on the questions it raises.

▪ A persona is a fictional user archetype—a composite model you create from the data you’ve gathered by talking to real people—that represents a group of needs and behaviors.

▪ In the usual course of product development, every interest other than the user has a say: business leaders will provide business goals and requirements, marketers will speak to marketing targets, engineers will speak to the technical constraints and level of effort required to develop particular features. Personas allow designers to advocate for users’ needs

▪ Good personas might be the most useful and durable outcome of user research

▪ Design, business strategy, marketing, and engineering can each benefit in their own way from a single set of personas.

▪ If you’re following an agile process, you can write your user stories based on a particular persona.

▪ Personas exist to represent the user in user-centered design, because there is no generic user

▪ They embody the behavior patterns and priorities of real people and act as a reference point for decision-making. A persona is a tool for maintaining an empathetic mind-set rather than designing something a certain way just because someone on the team likes it.

▪ Design targets are not marketing targets. Stamp that on every persona document you create. Market segments do not translate into archetypes

▪ And the user type with the highest value to your business may not be the one with the most value to the design process.

▪ Design for the users with less expertise and you can meet the needs of those with more.

▪ How many personas do you need? As few as possible, while representing all relevant behavior patterns

▪ You can often reduce the number by creating relationships among them and assigning multiple roles to one persona.

▪ For the Fantastic Science Center website you might consider an out-of-town visitor, a local parent, a teacher, and a staff member. Could the out-of-town visitor also be a teacher? Try it.

▪ All fictional user profiles are not created equal. A truly useful persona is the result of collaborative effort following firsthand user research. Otherwise you’re just making up a character that might be as relevant to the design process as any given imaginary friend.

▪ If you have interviewed some real people and worked collaboratively with your team to identify some patterns, you should be able to create some useful personas.

▪ Once you’ve created a set of personas, you can reuse them over time, even for different products.

▪ A persona description should have just enough detail to capture those aspects of a target user most useful and inspiring for the designers to keep in mind

▪ Consider your personas as a set. You don’t have to capture all concerns in a single one. And the personas can have relationships to each other, just like people do in real life.

▪ Use a real photo of a real, relatable person, not a stock photo. Creative Commons-licensed photos from Flickr or other photo-sharing websites are very useful for this.

▪ Give the persona a name that fits the demographic information and that everyone on the team can remember and pronounce

▪ Select the set of demographics that fit the role and behavior pattern. Be realistic without stereotyping. The persona must be plausible and representative

▪ Ideally, the gender, age, ethnicity, education, job, marital status, and location are derived from actual users you’ve interviewed. However, recruiting can be unpredictable and the lack of a complete match needn’t stop you from creating a suitable persona.

▪ Increase your knowledge by finding people whose online profiles match the criteria you do have.

▪ Try searching for local news stories about teachers to get useful background details, quotes, and even pictures of actual classroom environments.

▪ Role

For the most accurate personas, select a role that closely matches that of one of the participants you interviewed and is also one of the identified target user types, such as the aforementioned teacher, parent, or tourist.

▪ Use an actual quote from a user interview that embodies a core belief or attitude that is essential to keep in mind to meet their needs

▪ Goals

Goals and behavior patterns are the core of a persona. Identify three to four key goals for that persona based on what you heard in your user research. These will be the goals that the product or website will serve or relate to.

▪ The local parent’s goals might include finding weekend activities, keeping kids learning when they aren’t in school, and keeping up to date with advances in science.

▪ Behaviors and habits

Note the specific and habitual behaviors that constitute the pattern that defines the persona

▪ Skills

Skills include the level of technical expertise and experience this persona has. Be realistic about the level of skill you’re targeting with your design

▪ How much experience do you expect them to have based on their profession and educational background? This is a crucial area not to make assumptions.

▪ She could be a very good proxy for everyone who has a lower skill level, but absolutely doesn’t want to be made to feel stupid.

▪ Environment

Note all aspects of the environment that will affect this persona’s interaction with the product. Include the relevant hardware, software, and internet access. Do they go online at work or home? Surrounded by people or in private? Is their time online continuous or does it happen in specific chunks? The teacher might have half an hour during the day using the classroom computer. The parent might have an office job with a browser window always open.

▪ Relationships

Note any relationships this persona has that will affect their interaction with your product

▪ Is there a partner who influences decisions? Will children or coworkers be present or otherwise influence the use of your design? Relationships should be based on real-world information, either from your study or demographic information available through surveys or other research.

▪ Scenarios

If personas are your characters, scenarios are your plots. Each scenario is the story of how a persona interacts with your system to meet one (or more) of their goals.

▪ Running a persona through a scenario helps you think through your design from the user’s point of view.

▪ You can use scenarios at several points in your process:

To flesh out requirements. To explore potential solutions. To validate proposed solutions. As the basics for a usability test script.

▪ As long as a scenario hews closely to actual data gathered in user research, you have a lot of flexibility in the actual format

▪ While personas should remain reasonably constant in their characteristics and priorities, scenarios can evolve and deepen over time and change as your understanding of the system changes.

▪ You can write a scenario as a short text narrative, a step-by-step flow, or even a set of comic panels—whatever is easy for your team to create and use to keep each persona represented in design and technology decision-making

▪ If you find anyone on your team resenting the effort necessary to work with personas and scenarios, you’re doing it wrong. Simply drawing out scenarios on a whiteboard can work.

▪ Scenarios are not themselves use cases or user stories, although they can influence each

▪ A use case is a list of interactions between a system and a user, and is typically a way to capture functional requirements

▪ Scenarios are from the perspective of the individual human user represented by the persona, not the perspective of the system or business process.

▪ Goal: she wants to find local activities that will be entertaining for her son and relaxing for her and her husband

▪ Motivation: when she was driving home from the office on Friday evening, she saw banners for the Fantastic Science Center’s new exhibit on super storms. Sitting in her driveway, Diane Googles the science center on her iPhone.

▪ Task: she needs to determine whether visiting the Fantastic Science Center will meet her needs.

▪ You will know your personas are working when they become the first people you want to see any new idea. Rather than asking “Does this work for me?” or “Does this make my boss happy?” you can ask “Does this address Dana’s concerns about privacy? Would Neven understand what to do? Would Anna find time for this in her busy schedule?”

▪ All of us carry around a library of mental models in our heads. Without them, every new experience would be a complete surprise and we would have to painstakingly figure out each situation

▪ Using a term from cognitive science, a mental model is an internal representation of something in the real world—the sum total of what a person believes about the situation or object at hand, how it functions, and how it’s organized.

▪ People have mental models of how stoves work, how dogs behave, and what happens at a rock show. (Band plays, band says “Thank you and goodnight,” band waits offstage while audience applauds, band returns to play popular favorites.)

▪ In design, “intuitive” is a synonym for “matches the user’s mental model.”

▪ The closer an interface fits that image, the easier it will be to learn, use, and navigate. This is a concept with a lot of practical value.

▪ However, particularly following consultant and author Indi Young’s work in this area (Mental Models: Aligning Design Strategy with Human Behavior; http://bkaprt.com/jer/20/), people in the business tend to use the one term as a catchall.

▪ So there are two types of mental models: the type each of us holds in our head to help us deal with the world, and the type designers sketch out to better create that world

▪ To design an application or a website, think about the mental models of the activities you want to support

▪ If you’re designing a mobile application to help commuters find the best way to get to work on public transit, it’s useful to look at the mental model of “getting to work.” If you’re redesigning buses, you’d want to look at the mental model of “bus.”

▪ As a designer, you have your own mental model of what you’re designing, and you have a mental model of the users themselves, your set of assumptions about what they know and how they will interact with your design. It’s easy to overestimate how well your view matches their reality

▪ Documenting the user’s mental model allows you to not just get inside their head but get the inside of their head out of your head for everyone else to see.

▪ You can use a mental model diagram to collaborate with your team, prioritize features, better organize information, and identify areas where users have needs that aren’t being served.

▪ A mental model diagram can help resolve issues that arise if different user types have widely divergent mental models, or if the actual design of the system is significantly different from the one that was originally proposed.

▪ How to create a mental model

Do user research. Make an affinity diagram (see Fig 8.1). Place affinity clusters in stacks representing the user’s cognitive space to create the model. These groups will include actions, beliefs, and feelings. Group the stacks around the tasks or goals they relate to (Fig 8.3).

▪ Mental model diagrams illustrate your users’ thought processes in detail. This information helps you identify relevant and necessary content and functionality.

▪ Conceptual modeling/site mapping

For a new website or service design, you can translate the mental model to a conceptual map that relates content and functionality according to the target user’s view (Fig 8.4). The model will form the application framework or the basis of the information architecture as you proceed into more detailed design.

▪ Gap analysis

If you have an existing product or service, you can use a mental model to identify gaps, or mismatches between what you offer and what the user needs or expects. This will help you design features that fill those gaps.

▪ For example, when designing the app for urban commuters, you might find that their mental model of getting to and from work includes changing plans suddenly based on contingencies like bad weather, local events, and transit system delays. If your application only offers route suggestions based on optimal rather than actual conditions, you may recommend a route that’s influenced by rain or event traffic.

▪ TASK ANALYSIS/WORKFLOW

Task analysis is simply breaking one particular task into the discrete steps required to accomplish it

▪ Contextual inquiry is the best prelude to task analysis, but you can also use data from user interviews as long as you’ve collected sufficient detailed information about how the participants work toward their goals step by step

▪ Any given task has both cognitive and physical components that may be more or less important given the domain and the goal of the analysis. For example, making a complex purchase decision such as buying a new car typically has a series of cognitive activities surrounding identifying the need or desire for a car and conducting research online, as well as the physical component of actually going to the dealership and test-driving the car itself.

▪ For example, “purchasing tickets” sounds simple, but the online process is often a complex and stressful multistep flow with a lot of decision points

▪ Break it down

Using the information from user interviews or contextual inquiry, identify each step the participants reported or you observed them taking to complete a given task.

▪ Note the initial state, the event prompting the user to begin the task, the information or tools the user needs at each step, and any steps at which the task is likely to be interrupted or resumed. Put all of these steps back together as a workflow.

▪ Receive postcard advertising fall event calendar. Go to website. Locate event information on homepage. Click on link to see all available upcoming events. Identify event. Verify ticket availability and price. Enter number of tickets desired. Enter preferred delivery method. Review information and total cost. Select “Buy Now.” Enter credit card information. View confirmation page and instructions for receiving tickets.

▪ Communicating the meaning and value of research is a design activity itself. And the act of working together to synthesize individual observations will ensure that your team has a better shared understanding than a report could ever deliver.

▪ You may also benefit from the fact that a clear, economical diagram is viscerally appealing. If you’re promoting the value of research among skeptics in your organization, don’t underestimate the accessibility and appeal of your analysis, visualized.

▪ When you set out to optimize, you will run up against one of the oldest and thorniest philosophical problems, that of the Good. What is good? How do you know it’s good? What does it mean to be best? What are you optimizing for? How will you know when you have attained that optimal state and have reached the best of all possible worlds?

▪ What if, in optimizing for one thing, you cause a lot of other bad things to happen?

▪ Optimistic people talk as though there is some sort of obvious, objective standard, but once you start thinking about what is truly optimal, you will find that it’s always subjective and you’ll always have to make trade-offs.

▪ Qualitative research methods such as ethnography and usability testing can get you far. You’ll begin to understand how people make decisions and may even get a peek at habits they might not fully admit to themselves.

▪ No matter how much research and smart design thinking you did up front, you won’t get everything right out of the gate, and that’s OK. Because here come the data...I mean, the visitors.

▪ Once your website or application is live and users arrive in significant numbers, you’ll start getting some quantitative data

▪ Every interaction each of those individuals has with your website can be measured. All those people with the particular needs and quirks you lovingly studied fade away in the face of the faceless masses.

▪ You were in the realm of informed assertions. Now you’re in the big time. Actual data.

▪ You can see how well your design is performing out there in the world. How many people? How long do they stay? What do they see? Where do they exit? Do they return? How frequently? And do they click the button?

▪ A user is said to convert any time they take a measurable action you’ve defined as a goal of the site. For many websites there is an obvious primary raison d’être.

▪ On an application marketing website, conversion is clicking “sign up”; for ecommerce sites, “buy now”; on a hotel site, “make a reservation.” The success of the design can be measured by how many people click that button and do the thing that makes the money.

▪ Analytics refers to the collection and analysis of data on the actual usage of a website or application to understand how people are using it. Based on data from analytics, you can identify areas where your website is not as effective as you’d like it to be.

▪ For example, analytics could tell you that a thousand people visit the homepage every day, but only five people click on any other link on the page. Whether this is a problem depends on your goals for the site.

▪ Over half of the world’s websites have Google Analytics installed. It’s an excellent place to start and will give you a variety of pleasing charts and graphs. After you sign up, you or a friendly developer will need to insert a snippet of JavaScript into the source code of the site you want to measure.

▪ Some of the basic stats to look at include:

Total number of visits. Total number of pageviews. Average number of pages per visit. Bounce rate (the percentage of people who leave after viewing one page). Average time on site. Percentage of new visitors.

▪ In general, you want to see the total number of visits and unique users go up over time as your site becomes more and more popular

▪ More pageviews could mean increased engagement, or it might mean that you have a highly motivated audience who can’t find the information they’re looking for right away.

▪ Getting a lot of traffic through search is good, but visitors will bounce if your site wasn’t quite what they were looking for

▪ Some of the most interesting data points aren’t about what’s happening on your website at all, but where your traffic is coming from. Google Analytics will let you know how many people are coming from search results, other websites, or direct navigation—that relatively rare occurrence when someone just types in your URL.

▪ Clearly, access to data is no longer the issue. The question is what to do with all of these numbers. If you don’t already have quantitative goals, define some. You can start by looking up averages for your type of site or industry. Those are the numbers to beat.

▪ If you aren’t making your numbers, review the data and prioritize changes. Bounce rate is a good place to start. Before you get around to fine-tuning your message, you need people not to run screaming and stick around long enough to hear it. A high bounce rate is often an indicator of unmet expectations or uncertainty about where to go next.

▪ You can use analytics to see which pages are the most frequent entry points

▪ A split test is a clinical trial for a particular page or set of elements on your website. Some visitors are served the control—the current design—and others get a variation. The variation that performs significantly better for a specific metric is the winner.

▪ Like an international criminal, split testing has a lot of aliases, including A/B testing, A/B/n testing, bucket testing, multivariate testing, and the incredibly Panglossian “whole site experience testing,” which promises to deliver the best of all possible website experiences.

▪ This approach is of special interest to marketers, and actually derives from a technique first used in ancient times when special offers were sent on paper by direct mail. Send a flyer with one offer (“Free dessert with every pizza”) to a thousand houses, and a flyer with a different offer (“Free salad with every pizza”) to a thousand other houses, and see which one generates the better response.

▪ As a designer, you may be asked to participate in the process or create variations. So even if you aren’t in charge of running them, it’s helpful to know the basics so you can deal with the effects.

▪ Run the experiment until you’ve reached a ninety-five percent confidence level

▪ You will need a specific, quantifiable goal. This is a track race, not rhythmic gymnastics. No room for interpretation. You have to know the current conversion rate (or other metric) and how much you want to change it.

▪ Small, incremental changes will have a more significant influence on a high-traffic site (one percent of one million versus one percent of one thousand) and tests will be faster and more reliable with a larger sample size.

▪ How large a sample do you need? It depends on the sensitivity and how large an improvement you want to see. If you are looking at making a small change, you will need a larger sample to make sure that you can be confident in the result. A small change in a small sample size is more likely to be merely the result of chance.

▪ Approach this process with patience and confidence. The confidence in this case is statistical confidence, the probability that the winner is really the winner, rather than the outcome of chance events.

▪ The standard is ninety-five percent, meaning that there is a ninety-five percent chance that you can rely on the result.

▪ On a high-traffic site, you can get to this level within a couple of days. Lower traffic, and the test will take longer.

▪ To rule out the effect of other variables, such as day of the week, you would ideally let the test run over a two-week holiday-free period, allowing you to make day-over-day comparisons.

▪ If you have less patience, you open yourself up to more potential errors, both false positives and false negatives.

▪ Perhaps the Fantastic Science Center received an unusual mention in the New York Times, and the website variation you’re testing is particularly popular with Times readers but not with your typical population of site visitors. You need to let the test run long enough to counter these kinds of outliers.

▪ It’s also important to keep in mind that if you want to test variations of a particular page against the current state, someone has to design those variations. Even if they’re small changes, it’s still work.

▪ The wording, size, color, and placement of the button.

▪ Any piece of copy on the page and the total amount of copy.

▪ The price or specific offer.

▪ The image or type of image used (photo vs. illustration).

▪ After a number of tests you might see patterns begin to emerge that you can apply to your design work when solving for specific conversion goals.

▪ By the same token, remember that specific conversion goals are frequently just one aspect of the overall success of a website or a business.

▪ More precautions apply to split testing than to most other information-gathering approaches for a couple of reasons.

▪ Testing can be seductive because it seems to promise mathematical certitude and a set-it-and-forget-it level of automation, even though human decision-making is still necessary and the results remain open to interpretation within the larger context. The best response to a user interface question is not necessarily a test.

▪ Additionally, these are activities that affect the live site itself, so that presents a little risk.

▪ A consistent online experience can help build trust and habit, and split testing by its very nature introduces inconsistency. Keep this in mind as you decide what and how to test.

▪ This is an incremental process—tweaking and knob-twiddling—not a source of high-level strategic guidance. Since you’re changing things up, it’s best suited for aspects of your design where users might expect to see variation, and where there is a single clear user behavior you want to elicit in a given context.

▪ Search engine marketing landing pages? Fantastic. Those are generally intended for new users. Global navigation? Maybe not.

▪ Focusing on small positive changes can lead to a culture of incrementalism and risk aversion. How will you ever make a great leap that might have short-term negative effects?

▪ On his excellent eponymous blog (http://bkaprt.com/jer/21/), entrepreneur and adviser Andrew Chen invokes the concept of the local maximum, which you may be excited to remember from calculus. The gist is that you can only do so much optimizing within an existing design system. If you focus on optimizing what you have rather than also considering larger innovations, who knows what vastly greater heights you might miss (Fig 9.1).

▪ This is why understanding context and all the qualitative factors matters. Yahoo! could do all the split testing in the world and it wouldn’t turn into Google, and Google’s mathematical acumen is not turning Google+ into Facebook.

▪ Fig 9.1: Split testing can help you optimize your current design until you reach a local maximum, but it can’t tell you how much you can accomplish with a different approach

▪ There is a tension between strategic design thinking and data-driven decision-making. In the best case this is a healthy tension that respects informed intuition and ambitious thinking and knows how to measure success. When data rules the roost, this can leave designers feeling frustrated and undervalued.

▪ Doug Bowman left Google to become the creative director at Twitter in part because of A/B testing run rampant, saying: “When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data” (http://bkaprt.com/jer/22/).

▪ You can optimize everything and still fail, because you have to optimize for the right things. That’s where reflection and qualitative approaches come in.

▪ By asking why, we can see the opportunity for something better beyond the bounds of the current best.

▪ If this book raised more questions than it answered, fantastic. I want you to be excited about asking questions. Questions are more powerful than answers. And asking often takes more courage than sticking with comfortable assumptions.

▪ Every time you find a product or service that’s a joy to use, meets a need maybe you didn’t even know you had, and fits seamlessly into your life, you know that someone on the other end asked hard questions. Why should this exist? Who benefits? How can we make this better?

▪ Make friends with reality. Cultivate a desire to be proven wrong as quickly as possible and for the lowest cost. If you work in a culture that prizes failing fast, there is no faster way to fail than by testing an idea that’s still on the drawing board.

▪ How much research is just enough? You’ll need to do just enough to find out.

▪ Helsinki Design Lab: The government of Finland cares deeply about strategic design. This website is a trove of guides and templates, including an ethnography field guide (http://bkaprt.com/jer/23/).

▪ Service Design Toolkit: The Belgians, on the other hand, have focused on human-centered service design. Posters, guides, and workshop materials (http://bkaprt.com/jer/24/).

▪ Service Design Tools: Roberta Tassi’s thesis work in in the design department of the Politecnico di Milano resulted in this orderly collection of communication tools and methodologies (http://bkaprt.com/jer/25/).

▪ Remote Research: From the genuinely nice people who created Ethnio, a site to help you conduct remote research and testing (http://bkaprt.com/jer/26/).

▪ Design Staff: Google Ventures Design Studio publishes a fantastic blog full of smart and practical ideas for lightweight research, from recruiting to testing (http://bkaprt.com/jer/27/).

▪ Userfocus: This London-based usability consultancy publishes articles and ebooks. While many are free, some will cost a few pounds. Enjoy Usability Test Moderation: The Comic (http://bkaprt.com/jer/28/).

▪ Nielsen Norman Group: Jakob Nielsen’s evidence-based usability pronouncements are legendary. They have probably started as many arguments as they’ve settled (http://bkaprt.com/jer/29/).

▪ Getting People to Talk: An Ethnography & Interviewing Primer: This video is handy because there are limits to how much you can glean about interviews from reading (http://bkaprt.com/jer/30/).

▪ ICC/ESOMAR Code on Market and Social Research: Professional and ethical rules defined by the International Chamber of Commerce and an international organization for market researchers. Works just as well as a code for design research (http://bkaprt.com/jer/31/).

▪ Human-Centered Design Considered Harmful: This critique of Human-Centered Design brings up several critical points about context (http://bkaprt.com/jer/32/).

▪ An Ethnography Primer: AIGA and Cheskin put together a downloadable primer on design ethnography with concise text and pretty photos. Very handy for certain internal audiences (http://bkaprt.com/jer/33/).

▪ Observing the User Experience (2nd Edition), Goodman, Kuniavsky, Moed.

▪ Designing and Conducting Ethnographic Research, Margaret D. LeCompte and Jean Schensul.

▪ Mental Models: Aligning Design Strategy with Human Behavior, Indi Young.

▪ Interviewing Users: How to Uncover Compelling Insights, Steve Portigal.

▪ Designing for the Digital Age: How to Create Human-Centered Products, Kim Goodwin.

▪ Screeners

A recruiting screener is just a particular type of survey, so you can use Google Docs or SurveyMonkey or any other similar tool to make one if you have some in-house survey-writing expertise. (Web)

▪ Recording is easier when you make phone calls over VOIP. And always, always remember to get permission before you start recording.

▪ iTalk Recorder: Dead simple. Press the big red button to record audio. Press it again to stop. (iOS, Android)

▪ Split testing services cost between $0 (Google) and $300,000 (Accenture) per year depending on the vendor, level of service, and amount of traffic

▪ For not too much money, KISSmetrics is an analytics startup with reasonable, tiered pricing (starting at less than $50 per month), a refreshingly people-oriented philosophy, and a user-friendly interface.

▪ When you turn your research into models, you’re going to need to make some diagrams. This is really an area to use what you’re comfortable with