- Course Documents
- Course Summary and Key Take-aways
- Introduction and Qualitative Research Overview
- Interview Protocols
- Interviews, Observations, and Data Extraction
- Affinity Walls and Analysis
- There are a number of reason you'd do a user needs assessment but it's generally because you want to understand:
- why people want to use a product/service
- how people actually use product/service
- see what people like/dislike about a product/service
- hear what else people might want
- User needs can be actual needs, wants, preferences, quirks, etc.
- Qualitative research is non-numerical, direct interaction with people and context, and not designed to be considered aggregate, large-scale descriptions of user behavior
- This course focused on semi-structured interviews with a good chunk feeling very intuitive.
- There is a good section on interview protocols but feels very intuitive. It feels like this information could be condensed into a good checklist / cheat sheet of items to do before an interview.
- Some of the best tips on conducting an interview:
- Ask open-ended questions to really engage the user
- Don't be leading them to a certain place and don't judge! Users need to feel that the system is being tested, not their intelligence
- Closed-ended questions aren't great
- Overly abstract or generalized (i.e. "what do you normally do when..."); you want to be concrete
- Be Prepared. In general, being prepared and understanding who you're interviewing and having your questions ready will go a long way (again, not rocket science but good to remember)
- Post-interview, it's best to take notes immediately. The idea behind affinity notes and the affinity wall is really cool.
- The short slide about qualitative data and learning was interesting:
- You can condense information by clustering, sorting, finding patterns, and relating things to each other
- Condensing information is understanding
- Understanding is intelligence
- Why do a user needs assessment?
- To understand why people would want to use a product or service
- To learn how people actually use or don't use a product or service
- To see what people like or dislike about a product or service
- To hear what else people might want from a product or service
- When to do a user needs assessment?
- Before an design phase
- before designing something new
- before a re-design
- Any time questions or problems arise?
- What causes users to want our service?
- Why don't customers go beyond the home page?
- What frustrates users of our product?
- How could we improve the client's experience?
- Before an design phase
- "Needs" are not always urgent
- Origin of "needs assessment"
- 1960's
- Roger Kaufman
- In UX, "needs" = needs, wants, preferences, quirks
- often focused on a product or service
- pros and cons to taking a broader view
- Origin of "needs assessment"
- Course Focus: Qualitative Research
- Methodologies taught in this course:
- Interviews
- Observations
- Affinity diagrams
- Strengths of qualitative research:
- in-depth
- broad range
- hypothesis generation
- causal explanations
- Methodologies taught in this course:
- What this Course Does NOT Cover
- Other methodologies like:
- work models
- personals
- market research
- surveys
- log analysis
- document analysis
- Other methodologies like:
-
Quantitative research helps to understand...
- overall trends
- existing product or service
- issues you can anticipate
-
It is less useful for...
- detailed understanding
- new possibilities
- unanticipated
-
Qualitative Research
- Characterized by...
- non-numerical data
- direct interaction with people and contexts
- richness and detail
- Strengths of qualitative research
- in-depth
- broad range
- hypothesis generation
- causal explanations
- Characterized by...
-
Qualitative Vs. Quantitative:
- Qualitative research generates hypotheses that can be verified with quantitative research.
- Qualitative research offers potential explanations for outcomes from quantitative research.
- Qualitative research tends to provide close-up, in-depth description while quantitative research tends to provide aggregate, large-scale description.
-
Data gathering:
- interviews
- field oservations
- participant observations
- focus groups
-
Data analysis
- Coding
- content analytsis
- thematic analysis
- recursive abstraction bold: focus for this course
-
Quatitative Research Caveats
- not best for questions such as:
- What proportion of users do X?
- Would usage increase if we did Y instead of Z
- Does program V cause effect W?
- Weaknesses
- no numerical data or analysis
- relative comparisons
- Qualitative Research => you are trying to get the rich detail of particular uses
- not best for questions such as:
- I won't be doing this
- Types of Interviews
- Group vs. Individual
- Remote vs In-person
- Degree of structure
- structured
- semi-structured
- unstructured
- Semi-structured -> balancing what you know about the project but more free-form conversation
- balance between:
- researcher's interests and focus
- open, free-form discussion
- There is often a script or interview protocol
- conversational style
- expect to go OFF-script
- balance between:
- Interviews for Needs Assessment
- Who
- user / potential user
- "typical" member of the group
- Where
- where the participant normally uses product or service (or might do so)
- where participant feels comfortable
- When
- scheduled in advance
- mutually convenient time
- when participant normally uses the product or service
- Who
- Components of an Interview Protocol
- Overarching question
- Introduction
- Main Interview*
- background / warm-up questions
- core questions
- Observation*
- think-aloud protocol
- observation questions
- Conclusion
-
- => one set for every type of interview participant
- Overarching Question
- this is the big question you are trying to answer
- keep it in mind throughout protocol development and actual interview
- you will not necessarily ask the interview participant this question directly
- Introduction
- script for you to explain process to interview participant:
- self-intro
- goals of interview
- duration
- overview of components
- confidentiality
- voluntary
- permissions => if necessary (waiver, release, audio recording release, etc.)
- script for you to explain process to interview participant:
- Main Interview
- Background / warm-up questions
- short
- simple
- close-ended
- Core questions
- open-ended
- conversational
- focused on overarching question
- Background / warm-up questions
- Observation
- Think-aloud protocol
- encourage participant to verbally explain what they are thinking and doing
- awkward at first but it gets easier
- Observation questions
- prompts to perform tasks
- questions you can anticipate
- focused on overarching questions
- Think-aloud protocol
- Conclusion
- script for the end of the interview and observation:
- "anything you'd like to add?"
- Thank you
- Follow-up plans
- additional questions
- report back
- Your contact information
- "Any questions?"
- Thank you, again!
- script for the end of the interview and observation:
- Note
- these are guidlines; no hard rules
- most important: interview ethics
- second most important: is your key question being answered?
- Interviewing as Science and Art
- Preparation
- interview participant background
- topic background
- interview protocol
- overarching question
- Improvisation
- rapport with interview participant
- attention to the unexpected
- unscripted questions
- focus on overarching question
- Preparation
- Good Questions
- Open-ended
- why did you...
- How did you go about...
- Can you tell me about...
- Why do you think that is?
- How did you handle that situation?
- ask questions in which you really engage the user to tell you as much as they can about the situation
- specific, concrete
- "tell me about the last time that you did..."
- Non-leading, judgment free
- what option do you prefer?
- What do you think was the reason for that?
- How did you try to address the problem?
- Open-ended
- Not So Good Questions
- Closed-ended
- Do you like X?
- How many times a week do you?
- What is your favorite...?
- sometimes they are appropriate but they shouldn't dominate the protocol
- Abstract, generalized
- what do you normally do when...?
- what is the typical way in which?
- Thesea ren't good because humans aren't good at accurately describing what they do or have done.
- Leading, judgmental
- Do you prefer X over Y?
- Was it because of a bad process that the task was difficult?
- Why didn't you use a feature Z?
- Closed-ended
- Organization and Flow
- Cluster related questions together
- Within each cluster, arrange them in a meaningful order:
- chronological
- narrative
- broad to narrow, or narrow to broad
- Arrange clusters in a meaningful order
- Save any sensitive questions for the end
- Follow-up Questions
- Under each core question, add follow-up questions
- use bulleted list
- Follow-up question types:
- Generic: "Why?" "How did that happen?" "Tell me more."
- Prompts for specific issues: "Did you think about X? or Y?"
- Repetition: "Could you tell me about another time..."
- Think through the order of these, as well
- You may not ask all of these follow-ups
- Under each core question, add follow-up questions
- Final thoughts
- goal of protocol
- preparation
- guide during interview
- Memorize the overall flow of the protocol
- You will not read protocol questions one after the other
- goal of protocol
This interview protocol was developed for a 15-minute interview with a user of GPS mapping applications, seeking to understand unusual or unexpected things about their usage habits.
What unusual or unexpected uses of GPS does the participant make when navigating from one place to another, especially in the context of informing new or improved GPS features?
Hi, my name is Kentaro Toyama, and I’m here to understand better how you use GPS technology, particularly for navigating from one point to another. This interview will take about 10-15 minutes, during which time we’ll go through some questions. Throughout, I’d like you to treat me as if you’re describing the situation to someone who isn’t familiar with GPS devices. I’m here to learn from you.
A couple of things before we start. To the extent possible, I will take your comments to be confidential. My research team and I will aggregate all the comments from several interviews we’re conducting so that your comments are not easily traced to you. If we quote you in our final report, we will do so without identifying your name or specific role. If there’s anything you really don’t want on the record, even if it’s anonymized, please let me know that, too. Also, this interview is entirely voluntary on your part – if for any reason you want to stop, please let me know. We can end the interview at that point with no repercussions for you of any kind. I can also throw out anything you’ve told me until that point.
Do you have any questions for me? All right, then, let’s proceed.
Oh, and by the way, do you mind if I take an audio recording? This is just so that I don’t miss anything – no one other than the research team will have access to the recording. Thanks.
How often do you use GPS for navigating?
In general, are you happy with GPS technologies?
I’d like you to think back to the most recent time when you used a GPS to navigate from one place to another. Can you tell me a bit about that trip?
- When did the trip take place? From where to where did you go? Roughly how long was the trip?
- When exactly did you turn the GPS on?
- How did you input the destination location? [E.g., by address, by landmark, etc.]
- Did you have to interact with the GPS once you set the destination, and if so, when and why?
- Did you have other passengers, and did any help with navigation?
- Do you remember turning the GPS off, and if so when was that?
- Did anything unusual happen on that trip?
- Would you say that that was a typical trip? If not, what was unusual about it?
[Repeat above, for other trips, if fruitful.]
Now, I’d like you to think about a recent trip that was unusual or uncommon for you. For example, if you were on vacation, or in an accident, or part of a group of drivers going to the same location, and so on.
- Can you think of anything like that?
- What made the trip unusual?
- Did you use a GPS device? And if so, did the unusual nature of the trip change how you used the GPS?
- Who was in the car with you, and did they help with navigation?
One of the things I’m most interested in are situations under which the GPS is used in an unusual way. Can you think of any instances when you used the GPS in a way that isn’t typical?
- [Use follow-up questions from above.]
- Is there anything else that might be relevant – unusual uses of GPS, for example?
Thank you – those are all the questions I have for you. If anything else occurs to you after I leave, please don’t hesitate to let me know by email. I may be in touch with you again to ask a few follow-up questions. If you’d like, I can send a version of the report that we’ll write based on this interview. Do you have any questions? Thanks again!
- Memorize the overflow / prep for the interview
- participant background
- interview protocol
- Rapport with Interview
- make them feel comfortable
- do not jump right in
- start with small talk
- Adopt a Learning Mindset
- they are the expert
- Observe
- Adapt
- Listen
- They are the expert in their experience; you need to understand what they do and how they have tried to solve the problem
- Follow Interesting Threads
- Advanced prep pays off but willing to abandon the script if needed
- Ask follow-up questions until satisfied
- Listen for unexpected but relevant things
- improvise unscripted follow-up questions
- Note-taking
- take notes with pen and paper; devices can create a barrier
- print a list of questions with gaps for your notes
- Write down:
- key points in short phrases
- things that audio recording can't capture (like facial expressions, physical context)
- Follow-up questions
- Practice Interviewing
- maintain conversational tone
- keep overarching question in mind
- Be curious!
- Observation Review
- think aloud protocol
- observation questions
- Observation starts even before the interview
- Observation Tips
- Ask participant to perform relevant tasks
- Think Aloud Protocol
- what are you doing onw? why are you doing that?
- Take good notes on think aloud protocol
- what are the unexpected things?
- pauses
- missteps / multiple steps
- detours
- intentional detours
- Ask questions if you don't understand the motivation for something
- what are the unexpected things?
- Similar to interviews
- maintain conversational tone
- keep overarching question in mind
- be curious
- practice it
- watched
- From Notes to Qualitative Data
- As soon as possible after an interview, write down affinity notes
- an affinity note is:
- a sticky note
- a participant code (type and number: C01, G04, M02)
- statement or question
- from notes or audio recordings and relevant to the user needs assessment
- Include....
- factural statement
- participant quotes
- observations, as statements
- interpretations
- Questions
- Tips:
- make each note understandable on its own
- be concrete but concise
- one affinity note per minute of interview
- Interpretations
- repeating themes
- contradictions or conflicts
- key findings
- root issues
- anything else
- Good Quotes:
- unexpected
- representative
- The Secret to intelligence
- By clustering, sorting, finding patterns, and relating things to each other, you can condense information
- condensing is understanding
- understanding is intelligence!
- Affinity Wall Guidelines
- at first, make clusters quickly
- move notes around as you see new patterns
- don't get too attached to clusters
- Aim for clusters of 3-7 notes
- break up if more than 7 notes
- merge if less than 3 notes
- Writing the next-level notes is the crux:
- summarize the knowledge in the cluster
- summary should be a full sentence and make sense on its own
- balance abstraction and precision
- Stop creating level when summaries become uninformative
- at first, make clusters quickly
- watched
- watched