DIY Mobile Usability Testing

What is this?

The "DIY Mobile Usability Testing" project started back in 2009. We set out to prove that running usability studies for mobile software doesn't need to be difficult or expensive, it can be done by anybody, and it is a lot of fun!

The project has 2 parts:

1. It lists some basic principles and rules of thumb for planning usability studies with mobile devices
2. It provides clear instructions on how to capture video footage of usability testing sessions with mobile devices

We deliver the content as a workshop or presentation. So far, we have been to:

• Over the Air (2010)
• EuroIA (2010)
• IA Summit (2011)
• UX Sofia (2012)
• South by Southwest Interactive (2012)
• UX Spain (2013)
• MozFest (2014)
• FOSDEM (2015)
• ParisWeb (2016)

This repository contains the latest version of the presentation (the one for ParisWeb 2016), and this README includes the "script" we used for the talk.

Who are we?

• Belén Barros Pena (belenbarrospena at gmail dot com)
• Bernard Tyers (bernard at ei8fdb dot org)

The license

Creative Commons Attribution-ShareAlike (CC BY-SA)

The script

Hello everybody. Thank you very much for coming. Let us introduce ourselves. This is Bernard, who is a user researcher in Mordor... I mean, the UK Home Office. And I am Belén. Up to yesterday I was the design lead of something called the Yocto Project. Today I am blissfully unemployed.

Our talk is called do it yourself mobile usability testing. And as you might have guessed from the title, it is a talk about .... usability testing. This is one of the definitions of usability testing out there, from one of the most popular books on the subject. It is a process that employs people.

People. I wonder where we could find some people ...

[AFTER A BIT OF SEARCHING]. It looks like you might be the only people in this room, so one of you will need to help us today. And to pick just the right person, we have designed a highly scientific (and virtually infallible) recruitment exercise that I am going to ask you to go through with me.

The first step of the exercise is kind of tricky. It requires excellent co-ordination and fine motor skills. Are you ready?

What you need to do is ... to stand up. Take out your mobile phones. Now, sit down if you don't have a French phone that can surf the Interwebs. Sit down if you don't speak any English at all. Sit down if you don't like beer. Yes, I know is 11:30 in the morning but, you know, I am from Spain, and over there this is beer time. And finally, sit down if you are absolutely terrified by the idea of being our usability testing participant today.

And now, my beautiful assistant is going to choose the lucky one between you.

Wonderful. Now that we have a victim ... I mean, a participant, let's go back to this usability testing business. The definition is full of big words but the idea behind it is quite simple. It goes pretty much like this. You pick someone who looks like might want to use your software. And you put them in front of whatever you have of that software: drawings on paper, an interactive prototype, an alpha version, the thing that you already released. Whatever you have. And you ask that person to do something with the software. And you watch ... as they struggle to do even the simplest things. You learn tons from that.

This is how a usability testing session looks like. The blurry bit is the participant's face, which we have hidden for privacy reasons. You can't see his face, but he was mightily confused with the prototype he was trying, the poor thing.

So, as you can see, we record our usability testing sessions. And we do that for 2 reasons. 1) Because video footage allows us to go back to what happened during the session and clarify any doubts we might have. And 2) because video footage constitutes unequivocal evidence of the existence of usability issues. Even the most recalcitrant teams will concede defeat when confronted with actual people failing to do things with the software they are building.

But I personally believe there is a third reason to record your tests. It is the ability of video footage to generate empathy between software teams and their user base. You see, for the vast majority of people building software "the user" is an abstract entity, a blurry blob like the one I showed you before in the video. By watching video clips of usability testing sessions, that abstract "user" becomes instantiated, it acquires a face and a voice. And because of it software teams will stop talking about "the user" and will start saying things like: "Remember that girl? The girl that couldn't find the log in button? Would SHE find it now? Is it in the place where SHE would expected to be?". And by this “instantiation”, users are brought into the centre of the software-making process, something we normally attempt to achieve using personas. I must confess that, where my personas have failed miserably, I have succeeded using videos from usability testing sessions.

With the video recording we are actually capturing 2 separate things: we are capturing user's actions, the clicks and taps and swipes on the screen. Do yo see the orange circle? That’s a click of the mouse, the user action. And we capture reactions: software reactions to users' actions, which happen on the screen; but also users’ reactions. We human beings react with our faces, and that’s why we record facial expressions.

Now, having reached this point I have a confession to make: mobile usability testing is pretty much the same as desktop usability testing. All I've said so far applies to both.

Only that, in reality, they are not the same thing at all. Because when you throw mobile into the equation, you must face a few extra challenges.

And simplifying the whole thing a little bit, those extra challenges can be reduced to 3 big questions you need to answer before running any usability study with mobile devices. Which phone will you use for the test? In which context? And with which connection?

Now, I'm afraid there are no definitive answers to these questions. If you were expecting a formula answer that you can apply every single time you better leave now. As with almost anything in our user-centered design world, the answer depends on what you are trying to do.

Let's go through them in turn. First, which phone.

These are the results of an experiment run by Nielsen back in 2009. They took about 50 people and asked them to do some tasks with a few websites using their mobile phones. Then, they calculated the success rate and broke it down by the type of mobile phone the participants used, and this is what they found.

If participants were using a feature phone, something like a Nokia 6300, they managed to complete 38% of the tasks.

If participants were using what they called then a smartphone, which was something like a Blackberry Bold, they managed to complete 55% of the tasks.

And if participants were using what they called a “touch phone”, something I guess like the iPhone or the HTC Desire, they managed to complete 75% of the tasks.

The conclusion they reached was that the usability of the handset will affect the results of your test. Screen size, input and navigation mechanisms, memory and processor power, operating system design, the browser: all those factors influence how your users perform with your software, and their perception of it.

We must find a way of neutralising the effects of handset usability on the test results, or at least minimise them. And to do so, you need to remember 2 things.

1. Whenever you can, ask participants to use their own phones for the test. The phone might be a design and usability disaster, but your participant has probably become used to all its flaws, and has developed work arounds. It might be a horrible phone, but it’s your participant’s horrible phone.

2. If this is not possible, and loads of times it’s not... what if you are developing an application that only runs on a particular platform and your target users don't have that kind of handset? Or you have a prototype that only runs on your test phone? If it’s not possible, include time to train participants on how to use the phone, and add some dummy tasks so that participants can become familiar with the handset and the functions in it they will be using during the test.

And that's it about phones. The next question is which context. And this is about that old controversy that goes: should we test in a usability lab or should we test in the real users’ environment? For desktop software, this discussion is not really that important, because a lab environment is not that different from the real users’ environment. In both you have a desk, and a chair, and a computer with a mouse and a keyboard.

The field vs. lab controversy has come back to life with mobile, because a lab and the real users’ environment couldn’t be more different. We use our phones while walking on the street, while riding public transport, in public spaces, noisy places full of interruptions, too dark or way too bright. Far from those immaculate, comfortable, quiet and perfectly lit spaces that are our usability labs.

To determine what’s best, testing in the field on in the lab, academics use comparative studies. They take the same experiment, and they run it once in the field and once in the lab, and then they compare the results to see which one reveals more usability problems. So we have set up a little competition here using the comparative studies out there. Who thinks the field is going to win? Who thinks the lab is going to win? Well, let’s find out....

Kjeldskov et al, 2004, the added value of evaluations in the field is very little. Field 0, lab 1.

Kaikkonen and company, 2005, there was no difference in the two test settings. Field 0, lab 2.

Nielsen and colleagues, 2006, field can reveal problems not identified in laboratory evaluations. Field 1, Lab 2.

Duh and friends, 2006, there were more usability problems found in the field than in the laboratory.

And it’s a tie: field 2, lab 2.

So when it comes to where you should test your mobile software, in the lab or in the field, experts disagree... which is a polite way of saying they have no idea. But all of them agree on something. Testing in the field is more difficult, more expensive and more time-consuming than testing in the lab. In fact, testing in the field is so much hassle, that for the majority of us, working under industry time and budget constraints, is not even an option.

But don't let that discourage you: testing in the lab is always better than no testing at all. The real question is when and how to go into the field. So, remember: for most software, lab testing will do handsomely. But if you must do field testing... and sometimes you must. What if you are developing a field data gathering application, for example for geologists, or emergency services. Or a geocaching game. Or if you are developing mobile money applications. In all those cases, you absolutely need to run field tests. And if you do: do it late, after you have already done testing in the lab, you have fixed the most glaring usability problems and you have a somehow solid interface. Some authors suggest to piggyback on field trials when possible.

Plan, plan your head off, and run pilot tests to make sure the location you have selected is appropriate, that all equipment works as planned, that you have scheduled enough time (field tests tend to take longer than lab tests), that the chosen test protocol is understood and works (how to handle moderator prompting, timing between questions, how to react to external interruptions, etc, etc), to identify factors that might impact the test results (noise, lighting conditions, etc).

Be prepared, like the Scouts. Field tests are vulnerable to unexpected events, such as rain or bus delays. More than likely something will go wrong, and you better be ready when it does.

And finally, embrace the wilderness. The most interesting thing of field testing is that you have no control over the environment. Give up you controlling instincts, relax, enjoy and learn as much as you can.

And that's it about context, so let's tackle the last question: which connection. This is about what to use during your tests: wifi, or the mobile phone network. Unless your software only runs over wifi, (or usage data tells you otherwise) the safe assumption is that most access will happen through the mobile phone network. Mobile phone networks are becoming pretty good here in Europe, but we still have blind spots. Ofcom reckons a whopping 65% of the Scottish Highlands have no reliable 3G signal, and this is the UK ladies and gents. I am not talking about India or Kenya here.

So, in general, do not test over wifi. Wifi networks are usually faster, more reliable and have less latency than mobile phone networks. If you test over wifi, you are likely to miss usability issues related to performance, responsiveness and unreliable connectivity.

And of course, if your participants use the mobile phone network during the test, they might incur data costs. Make sure to compensate them accordingly, or they will hate you.

So that wasn’t so bad, was it? Wouldn’t it be great if the mobile world was that simple? Fat chance: the formula is still incomplete, and this is what’s missing. The small detail of how on earth are you going to record the whole thing.

Yes, of course, you could make do without recording the tests, but... do you remember why we record our usability tests? Do you remember the power of video to generate empathy?

We’ve found 4 main approaches to recording usability studies with mobile devices. The first one is wearable equipment, which usually involves some form of headrest with a camera attached to capture the mobile phone screen. It might also include microphones and headphones, to give instructions to participants or to record their comments, and unfortunately, also batteries and video mixing equipment, because you must power it all somehow and merge the different video sources. All this stuff being carried in belts or backpacks. Some teams have done a great job of reducing the size and weight of the equipment. This one here weights just 2 kilos.

Wearable equipment is great because it allows you to run tests in the field, which we know sometimes we must do, but ... the required equipment is not easy to set up, it’s intrusive, heavy and in general quite uncomfortable.

The second approach is screen capture, either by mirroring the phone screen to another device using something like Reflector, or by capturing the screen using the phone itself with applications like UX Recorder or UXTester.

Screen capture is great because it provides really good quality footage, but it imposes constraints on the devices you can use. It must be Android, or iOS, or a certain version of any them, or have some application installed. And this complicates the business of using your participants' own phones.

The other problem with screen capture is that it doesn't record the user's fingers, which are quite expressive things by the way, as we will see later on. Without the fingers, you are bound to miss some valuable information. This is why many people in the industry are still refusing to use the screen capture approach.

The third approach is document cameras. Document cameras stand on a desk and point downwards. They have been the approach of choice by very important people like Scott Weiss, Google and Nielsen Norman.

Document cameras are great because they combine good recording quality with easy set up. There was a time when document cameras were expensive things, but there are now affordable alternatives, like the cameras from this company called Ipevo. This is one of them, which we have brought with us for you to see. And this is another one. But although document cameras are good, they also have some problems.

Participants must keep within the camera range, which is often drawn as a square on the desk surface. So your poor participant, who is already terrified and self-conscious because they are doing this thing you have called ‘a test’ in front of a stranger who is watching them while they are trying to use this website they have never seen before and makes no sense whatsoever, must also remember not to move the phone beyond the funny square or they’ll ruin the whole thing. Cognitive overload!

The other problem is that the phone must lay on a desk or be hold at a flat angle, which is not how we normally use our phones. And I have seen this with my own eyes: when you are concentrating hard to get something done on the phone, your first instinct is to pick the thing up from the table. We just can't resist.

The fourth approach is mounted devices, a frame attached to the phone that holds one or more cameras. They come in 2 flavours: ready-made, the ones you buy; and diy, the ones you make yourself.

6 years ago, when we first started doing this talk, there were only 4 examples of DIY mounted devices on the Internet: Nick Bowmast's, Barbara Ballard's, Google's and one from Usability Sciences. How things have changed. The Internet if now full of them. But we all stand in the shoulders of those 4 pioneers.

Mounted devices are cool, because they solve the problem of document cameras: since you can hold the phone with them, they allow a more natural interaction with the device.

However, the ones you buy tend to be expensive things. This DigiZoom one no longer exists. Not surprising, given the price tag. Luckily, there is now a more affordable alternative out there, thanks to the same Nick Bowmast that pioneered one of the first DIY rigs. It's called Mr Tappy, and with a webcam included, it sells for USD 349. If this is too much for you, and it very well might be, you can always make one yourself. But you must know that they can be messy to build. If they are too bulky, they can prevent one-handed use of the phone. And if they are too heavy, they can be tiring for study participants.

At this point, I hope it is clear that none of these 4 recording approaches is perfect. But how would the perfect recording set up look like? Well, it would need to be like this: easy to put together and cheap, so everybody can do testing; repeatable, so that you can easily replace it when it breaks; it must allow holding the phone and one-handed use, to ensure natural interaction with the device; it should support all form factors, and run tests with participants' own phones, to minimise the impact of the device on the results of the study; it should capture the mobile screen, but also the participant's face and fingers; and it should give enough video quality, and the 'enough' bit is important here. It doesn't have to be 4K video: it just needs to be intelligible, so that you can hear what participants are saying and follow what they are doing.

Once we figured this out, we checked the recording approaches against these criteria, and put it all neatly on a table like this one. It turns out that wearable equipment and screen capture have the highest number of red balls, so we discarded those two. The ready made mounted devices did great, with a single red ball. Unfortunately, it's the price criteria, which is completely out of our control and we can do nothing about. So we discarded that one too. Document cameras do very well, but the red balls have all to do with the natural interaction with the phone, and we really wanted that. So we discarded document cameras, and with that there was only one option left: the DIY mounted devices.

We just needed to build one that was easy to make and easy to replace when it broke. And so we did, and this was the spirit of our enterprise. This photo shows an iPad stand made with pencils. It looks absolutely ridiculous ... but it works! And that was the spirit of our DIY mounted device: it doesn't matter how stupid it looks, as long as it does the job.

This is what we used to build it.

• 2 Meccano trunions
• A few Meccano strips
• Some screws and nuts
• A jubilee clip, a piece used by plumbers that is the real star of the show
• A HUE webcam
• A second webcam (any will do)
• A USB extension cable
• Some Blu Tack
• A few tools: an allen key, a Meccano wrench and a screwdriver
• A computer (it will be Linux today)
• A webcam app. We will use GUVCView
• Screen recording software. We will use recordMyDesktop

This is how it looks like: absolutely ridiculous. But guess what? It works! And we are going to prove it to you today with the help of our selected victim ... I mean, participant.

[RUN THE TEST]

We are almost done. The only thing left is to see how our DIY recording gizmo fares when checked against the ideal recording set up.

The ideal thing should be easy to put together. If Bernard can do it here in front of you, it sure must be easy. To make it even easier, we have made a video with instructions

So, green ball.

The ideal thing should be cheap. So how much does ours cost? A total of GPB 62.48. That turns out to be about EUR 73.00 right now, but with the pound in free fall with all this Brexit nonsense, tomorrow it could be just 20! I think that qualifies as cheap, so green ball.

The ideal thing should be repeatable, which means it should be easy to fix / replace when it breaks. This is a measure of how straightforward is to obtain the pieces required to build it. Let's see:

• both webcams and the USB extension cable we bought from Amazon

• all Meccano pieces and tools came from an amazing online shop called Meccano Spares

• jubilee clips, blu tack and screwdrivers can be purchased from any diy store

So really easy to replace any damaged pieces, and green ball.

The ideal thing must allow natural interaction with the mobile device, which means participants must be able to hold the phone and use it with one hand. A good measure of this is how heavy the rig is. Ours comes to 125 grams. And just to provide some context, I've added the weight of a few random things, also in grams. Users can also hold the phone and move it around: because the camera is attached to it, the image will remain stable, as I hope you saw earlier during our test.

I am going to put a green ball, with the caveat that 125 grams added to a mobile phone is quite a bit. Participants will get more tired than when they use a PC, so make your sessions as short as possible, and schedule breaks.

The ideal thing must support all form factors. There was a time when mobile phones came in all kinds of shapes and sizes. But we seem to have settled down on a black box with as few hardware buttons as possible (they are expensive, you see). If you ask me, this is rather boring, but it does have an advantage: building a recording device that fits all phones is now much easier than it used to be. The only significant variation will be the width of the phone, so the only thing you need is a few different Meccano strips to fit them all. 5, 6 and 7-hole ones should do. Green ball.

The ideal thing must run tests with participants' own phones. Well, we took someone at random from the audience, with some random phone, and ran a test. So green ball.

Finally, the ideal thing must capture screen, face and fingers (very important the fingers, as we saw); and must give you enough video quality. And here is the outcome of a test like the one we just ran in the room. Screen, face and fingers are all there, and I hope you'll agree with me the video quality is about enough. Green balls!

And with this we are done. Let us finish with a call to arms.

Mobile software is hard: mobile phones are still rather new, and come with enormous interaction challenges, from a huge and heterogenous user base, to ever-changing context of use. We cannot expect to build useful, usable and beautiful mobile software without engaging our users in the process. Usability testing has proved to be an excellent tool for that, and now you know how to do it for your mobile software. So no excuses. Go back to work on Monday, and start planning that usability study.

Thank you for listening.