UX testing results #1470
Comments
|
Wow, that is a lot of input! Thank you for that, @matkoniecz |
westnordost
changed the title
|
Great thing/testing.
I guess this is because you've tested with new users. This would need more long-term tests to see how easily "boring" it may get when they see the same quest again and again.
Possibly unsolvable. We do not have that much screen space to put it all in there. Somewhere thre must be an "overflow" menu. But one could use hints/pointers to point there.
I'd trust Google. They know what to do. And the back button is universal on Android.
So the issue is the loading bar is stucking? In this case, I think Firefox for Android had/has a nice effect: Just make the bar a gradient that is also moving. So there is always something moving ("loading" indicator).
Just a placeholder text (maybe an example?) when nothing is written? |
I think that this wouldn't really be necessary to make StreetComplete more fun. Something simple like a leaderboard with world, national, and local "best StreetCompleters" would give users a sense of community and importance, but not to the extent of bribing users to enter incorrect information for points. |
|
I also thought about "mappy days" - solving quest, any quest, on given day makes that day successful. In general, certainly not all suggestion from users will be implemented (especially as some were contradictory). I mentioned only ones shared by many or in some other way valuable. Many things expressed only by a single person were not mentioned at all. |
|
Hi @matkoniecz, you've made an impressive work here ! About item 6 (gamification) discussed by you and @BjornRasmussen : Other thing in my mind:
|
|
@Binnette it's possible to get the scores of each player by analysing osm changeset tags. Every changeset created by SC leaves tags indicating what quest was answered. By looking at the changesets, it would be possible for an external system to create a StreetComplete leaderboard without any changes to the app. |
Potentially a good idea. In case of quest that matches requirements of StreetComplete quest (you can see them for example in https://github.com/westnordost/StreetComplete/blob/master/CONTRIBUTING.md#suggesting-new-quests ) new issues about specific proposals are welcomed! I remember looking through Osmose reports and failing to find a good candidates, I looked again at Keep Right and failed to find anything that would be a good quest and is not already supported. |
I also would not consider it as the very important thing, but I will give it higher priority than before UX test (from "maybe would be nice to have, I am not going to work on it at all" to "I will spend some time on this")
I had time for StreetComplete but no access to my PC so I ended writing a bit about such server (see #1485)
Yes (I even have script that is listing popularity of each quest, I a now checking is it giving accurate statistics). Technically central server is not mandatory for synchronization but... More in #1485 |
|
@westnordost Do you have access to more precise total download count than "10 000+" displayed on Google Play to public? As of early July 18 576 users made at least one edit - and I am curious how many downloaded app and failed to save even a single edit. |
|
In total, 20780 users installed the app. The app was uninstalled 12770 times. There was a huge peak in 2017, since mid 2018 the install and uninstall events evened out to roughly 250-500 installs and 150-350 uninstalls per month. |
|
89% would be a famously good conversion rate. Though, not included are the number of users that installed the app from F-Droid, because there are no statistics at all. I also wouldn't know how to at least get an estimate how many percent of users are from F-Droid. |
Could that be recorded as changeset metadata? As the Play version and the F-Droid version are different builds of the app, that sounds to me in scope of the changeset metadata. This could be done either by appending this to the |
|
Hmm, it could. Then, rather as an extra tag in the changeset so that it all counts as the same application. But how would such information be useful beyond satisfy our curiosity? |
|
@westnordost : i agree that, this extra tag is mainly for statistics/curiosity... It can also be suspected as tracking personnal data. Maybe users do not want to share which store they are using (after all, users just want to contribute to OSM and do not leak personnal data). Here is an attempt to find a way to use this tag: I can not find any other useful beyond for it... |
|
It'd be good to cross out or somehow mark those points that have already been addressed (in other tickets), so it is easier to keep an overview here. |
I added (missing issue for the main problem) where no issue is open and it would be useful to make one. I can write them, but I think that I prefer to finish open PRs/fix already open issues as it seems to be more useful activity. I also linked some issues that were matching. Main issue that is tricky to handle this way is "registration page at OSM website is horrible, on mobile it is turbo-ultra-horrible and one of main barriers to OSM growth", especially as it appears to not be an opinion shared by maintainers of the OSM website. |
|
Did you open a bug for the layout issues on the registration page? Your issue is just about the header image. |
|
See for example openstreetmap/openstreetmap-website#2287 that got rejected, maybe because I poorly explained what is the problem |
|
2287 was about something else, tbh. I'd create a new ticket or rename the open one to reflect the more obvious bugs in the layout. |
|
Main problem with registration form is not layouts bugs but massive length. There is a good reason why registration on a typical site is quite simple (at least in cases where organization want people to register). Especially people using "Login with X" are not expecting to see massive registration form. Yes, many of fields are optional, but it is not clear to people registering. One of main complaints was about selecting password anyway after using "register with X" |
Don't know if that helps, but did you know you can look for all your comments? |
|
It may be ready for closing as I opened issues for remaining problems where sole mention of problem was here, except
where I would open an issue, but it seems to be rejected already I am unsure whatever it would be useful to open again an issue on the OSM website once again. Something on topic "mobile registration is cumbersome, even in case 'log in with FB' or 'log in with Gmail'" but maybe more persuading than what I opened. |
|
Ok then I'll close this |
|
Results of a new UX testing round were posted at #4264 |
Recently I made a small UX testing run. In other words I found people willing to use this application as I looked at their phone. Hopefully it is OK to post here what I learned.
Conclusions based on testing (note that I focused on what should be changed/improved, I unfortunately mostly failed to note what was going well):
(1) people declared that they like the application, design and how it works. One person had a strong negative impression but it was caused by an OSM registration form broken in many ways.
(2) UX testing already resulted in multiple improvements. Many small issues were identified. Some of them are easy to fix and are already fixed. Fixed issues include improvements to a Polish translation, pull requests #1437, #1441, #1442, #1444, #1446, #1444, #1442, #1437, #1441, #1448, #1451, #1491, #1484, #1478, #1476, #1471. Some further pull requests are open: #1445, #1450, #1452, #1460, #1462, #1455 + some not yet included improvements of Polish translation.
(3) two of testers had contact with earlier version and considered current one as a massive improvement
(4) (missing issue for the main problem) for people new to OSM registration on mobile is a step barrier. Especially as registration page has broken layout on mobile, registration with Gmail account failed completely (this is now fixed), registering with Gmail account still asks about password and mail. Registration page was describes as "it looks like something done by students for a late project, not a website of a serious project".
I opened openstreetmap/operations#311 (now fixed), openstreetmap/openstreetmap-website#2288 (kept open), openstreetmap/openstreetmap-website#2287 (rejected, but I feel that it is worth rediscussing, but only after fixing one that was not refused), openstreetmap/openstreetmap-website#2285 (rejected, but alternative was proposed in a comment)
(5) #1552 (?) Missing tutorial/explanation was identified as a major issue. It turns out that for new users it is not clear how app works.
were all NOT obvious and NOT clear. Nearly all people were clearly confused.
(6) some form of gamification was widely requested, proposed, suggested or implied as needed. One memorable comment was "it is a more boring Pokemon Go". It was not mentioned by single older person and strongly pushed as important by the youngest one.
Youngest proposed that stars should unlock something in app. For example by allowing to spend in-app currency to build a house, or to build your own map "or at least achievements"
#1485 is the first step in that direction
(7) overall, based on UX test it is clear that at this moment application would benefit more from tweaking what exists (including also work like #1329, #1438) than from adding new quests. This was a bit surprising to me as I in past focused on and planned to continue work mostly on adding more quests
(8) #1554 problem shared by every single person was gluing map to a current location. Every single person attempted to pan map and failed to do it. Every single person failed to notice about ungluing. Every single person considered option used to unglue as hidden and not in the expected place. All were expecting that clicking on that button will locate them, not that it will trigger ungluing. Two suggested to detect attempts to pan map freely and to show message in such case.
(9) (missing issue for the main problem) problem shared by all three who encountered quest about no longer existing feature - it was not clear to any of them that expected path was to select "can't say", select note, create note.
This can be fixed by either adding special entry in "can't say' or by educating users (#1485 would be the first step). First was rejected in past (#594), though maybe it changed? Rejected #1097 is also related
(10) Both who encountered lit quest and all three who triggered surface quest were really unsure which answer should be used, especially for borderline ones.
(11) Both people who wanted to go back from note creation menu had problem to do it. One of them proposed adding explicit back button in case where there are other buttons present. Note that Google recommends not doing this - see https://material.io/design/navigation/understanding-navigation.html#reverse-navigation linked from https://developer.android.com/design (though I would not trust advice about navigation from page with broken navigation)
(12) (missing issue for the main problem, but may be fixed now) Quest loading indicator is generally not recognized as one. I admit that I personally looking at app interface I am unable to say whatever it is downloading quests, finished downloading or waits for new quota from Overpass API. Especially rolling back of red bar is a bit confusing.
This may be fixed now - rolling back was caused by combination of slow queries and timeouting too soon, both sure fixed. Bar is now also larger. It may be enough and it is not one of very big problems.
(13) All people who tried to create note were unsure which language for note text was preferable. Unfortunately I have no good idea how to explain this one without pushing too much text into note creation dialog. (EDIT: now one version of fix is in #1471, thanks to @rugk for an idea)
(14) issues #1456, #1457, #1463, #1485 were created for more complicated problems that are more complicated to fix (and maybe are not worth fixing or fix will not make situation better, so I started from issue rather than coding immediately)
(15) on Xiaomi Mi A2 lite in evening night style enabled itself, but only partially. Map style remained old, text style remained old, but background was dark, resulting in poorly readable text (not reported for now as I was unable to discover how night mode is enabling itself in auto mode despite looking at #1306) - reported as #1400 by a separate person
UX testing info:
UX testing done on biased, small and unrepresentative sample.
But it is good enough to detect low hanging fruit and resulted in a very useful feedback.
To be more specific - I tested on group of 7 people. All were my friends or my family. All from Kraków, Poland and the same cultural group, almost all of them are young adults.
One person is an active OSM contributor, second had an unused OSM account. I also asked for additional feedback on some OSM forums (talk-us, talk-au, slack-us). I also checked comments at Google Play in Polish and English. I based some things on comments at HN in OSM-related submissions that were linked in slack-us.
Despite that group was biased, small, unrepresentative sample multiple clearly problematic issues were identified. Fixing ones that have clear and obvious solution and releasing new version is at this point the best use of energy. It will be more useful than testing further and confirming that identified problems are actually problems.
So I plan to do at least tiny step to fix each of issues identified by more than one person. Then wait for a release that will include all or nearly all UX improvements, and test again with next group of people. I plan also to ask for feedback in more communities - talk mailing list, reddit/r/openstreetmap, HN, maybe also other...
My work on UX testing was sponsored by a NGI Zero Discovery grant
Similar issues that are articles rather than issues:
Google Play Statistics 2019 #1416
#866 also had some interesting statistics
The text was updated successfully, but these errors were encountered: