Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
UX testing results #1470
Recently I made a small UX testing run. In other words I found people willing to use this application as I looked at their phone. Hopefully it is OK to post here what I learned.
Conclusions based on testing (note that I focused on what should be changed/improved, I unfortunately mostly failed to note what was going well):
(1) people declared that they like the application, design and how it works. One person had a strong negative impression but it was caused by an OSM registration form broken in many ways.
(2) UX testing already resulted in multiple improvements. Many small issues were identified. Some of them are easy to fix and are already fixed. Fixed issues include improvements to a Polish translation, pull requests #1437, #1441, #1442, #1444, #1446, #1444, #1442, #1437, #1441, #1448, #1451, #1491, #1484, #1478, #1476, #1471. Some further pull requests are open: #1445, #1450, #1452, #1460, #1462, #1455 + some not yet included improvements of Polish translation.
(3) two of testers had contact with earlier version and considered current one as a massive improvement
(4) for people new to OSM registration on mobile is a step barrier. Especially as registration page has broken layout on mobile, registration with Gmail account failed completely (this is now fixed), registering with Gmail account still asks about password and mail. Registration page was describes as "it looks like something done by students for a late project, not a website of a serious project".
I opened openstreetmap/operations#311 (now fixed), openstreetmap/openstreetmap-website#2288 (kept open), openstreetmap/openstreetmap-website#2287 (rejected, but I feel that it is worth rediscussing, but only after fixing one that was not refused), openstreetmap/openstreetmap-website#2285 (rejected, but alternative was proposed in a comment)
(5) Missing tutorial/explanation was identified as a major issue. It turns out that for new users it is not clear how app works.
were all NOT obvious and NOT clear. Nearly all people were clearly confused.
(6) some form of gamification was widely requested, proposed, suggested or implied as needed. One memorable comment was "it is a more boring Pokemon Go". It was not mentioned by single older person and strongly pushed as important by the youngest one.
Youngest proposed that stars should unlock something in app. For example by allowing to spend in-app currency to build a house, or to build your own map "or at least achievements"
(7) overall, based on UX test it is clear that at this moment application would benefit more from tweaking what exists (including also work like #1329, #1438) than from adding new quests. This was a bit surprising to me as I in past focused on and planned to continue work mostly on adding more quests
(8) problem shared by every single person was gluing map to a current location. Every single person attempted to pan map and failed to do it. Every single person failed to notice about ungluing. Every single person considered option used to unglue as hidden and not in the expected place. All were expecting that clicking on that button will locate them, not that it will trigger ungluing. Two suggested to detect attempts to pan map freely and to show message in such case.
(9) problem shared by all three who encountered quest about no longer existing feature - it was not clear to any of them that expected path was to select "can't say", select note, create note.
(10) Both who encountered lit quest and all three who triggered surface quest were really unsure which answer should be used, especially for borderline ones.
(11) Both people who wanted to go back from note creation menu had problem to do it. One of them proposed adding explicit back button in case where there are other buttons present. Note that Google recommends not doing this - see https://material.io/design/navigation/understanding-navigation.html#reverse-navigation linked from https://developer.android.com/design (though I would not trust advice about navigation from page with broken navigation)
(12) Quest loading indicator is generally not recognized as one. I admit that I personally looking at app interface I am unable to say whatever it is downloading quests, finished downloading or waits for new quota from Overpass API. Especially rolling back of red bar is a bit confusing.
(13) All people who tried to create note were unsure which language for note text was preferable. Unfortunately I have no good idea how to explain this one without pushing too much text into note creation dialog. (EDIT: now one version of fix is in #1471, thanks to @rugk for an idea)
(14) issues #1456, #1457, #1463, #1485 were created for more complicated problems that are more complicated to fix (and maybe are not worth fixing or fix will not make situation better, so I started from issue rather than coding immediately)
(15) not reported as issue for now - on Xiaomi Mi A2 lite in evening night style enabled itself, but only partially. Map style remained old, text style remained old, but background was dark, resulting in poorly readable text (not reported for now as I was unable to discover how night mode is enabling itself in auto mode despite looking at #1306)
UX testing info:
UX testing done on biased, small and unrepresentative sample.
But it is good enough to detect low hanging fruit and resulted in a very useful feedback.
To be more specific - I tested on group of 7 people. All were my friends or my family. All from Kraków, Poland and the same cultural group, almost all of them are young adults.
One person is an active OSM contributor, second had an unused OSM account. I also asked for additional feedback on some OSM forums (talk-us, talk-au, slack-us). I also checked comments at Google Play in Polish and English. I based some things on comments at HN in OSM-related submissions that were linked in slack-us.
Despite that group was biased, small, unrepresentative sample multiple clearly problematic issues were identified. Fixing ones that have clear and obvious solution and releasing new version is at this point the best use of energy. It will be more useful than testing further and confirming that identified problems are actually problems.
So I plan to do at least tiny step to fix each of issues identified by more than one person. Then wait for a release that will include all or nearly all UX improvements, and test again with next group of people. I plan also to ask for feedback in more communities - talk mailing list, reddit/r/openstreetmap, HN, maybe also other...
My work on UX testing was sponsored by a NGI Zero Discovery grant
Similar issues that are articles rather than issues:
I think that this wouldn't really be necessary to make StreetComplete more fun. Something simple like a leaderboard with world, national, and local "best StreetCompleters" would give users a sense of community and importance, but not to the extent of bribing users to enter incorrect information for points.
I also thought about "mappy days" - solving quest, any quest, on given day makes that day successful.
In general, certainly not all suggestion from users will be implemented (especially as some were contradictory). I mentioned only ones shared by many or in some other way valuable. Many things expressed only by a single person were not mentioned at all.
Hi @matkoniecz, you've made an impressive work here !
About item 6 (gamification) discussed by you and @BjornRasmussen :
Other thing in my mind:
@Binnette it's possible to get the scores of each player by analysing osm changeset tags. Every changeset created by SC leaves tags indicating what quest was answered. By looking at the changesets, it would be possible for an external system to create a StreetComplete leaderboard without any changes to the app.
Potentially a good idea. In case of quest that matches requirements of StreetComplete quest (you can see them for example in https://github.com/westnordost/StreetComplete/blob/master/CONTRIBUTING.md#suggesting-new-quests ) new issues about specific proposals are welcomed! I remember looking through Osmose reports and failing to find a good candidates, I looked again at Keep Right and failed to find anything that would be a good quest and is not already supported.
I also would not consider it as the very important thing, but I will give it higher priority than before UX test (from "maybe would be nice to have, I am not going to work on it at all" to "I will spend some time on this")
I had time for StreetComplete but no access to my PC so I ended writing a bit about such server (see #1485)
Yes (I even have script that is listing popularity of each quest, I a now checking is it giving accurate statistics). Technically central server is not mandatory for synchronization but... More in #1485
@westnordost Do you have access to more precise total download count than "10 000+" displayed on Google Play to public?
As of early July 18 576 users made at least one edit - and I am curious how many downloaded app and failed to save even a single edit.
89% would be a famously good conversion rate. Though, not included are the number of users that installed the app from F-Droid, because there are no statistics at all. I also wouldn't know how to at least get an estimate how many percent of users are from F-Droid.
Could that be recorded as changeset metadata?
As the Play version and the F-Droid version are different builds of the app, that sounds to me in scope of the changeset metadata.
This could be done either by appending this to the
@westnordost : i agree that, this extra tag is mainly for statistics/curiosity... It can also be suspected as tracking personnal data. Maybe users do not want to share which store they are using (after all, users just want to contribute to OSM and do not leak personnal data).
Here is an attempt to find a way to use this tag:
I can not find any other useful beyond for it...