Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UX testing results #1470

Open
matkoniecz opened this issue Jul 5, 2019 · 14 comments

Comments

@matkoniecz
Copy link
Contributor

commented Jul 5, 2019

Recently I made a small UX testing run. In other words I found people willing to use this application as I looked at their phone. Hopefully it is OK to post here what I learned.

Conclusions based on testing (note that I focused on what should be changed/improved, I unfortunately mostly failed to note what was going well):

(1) people declared that they like the application, design and how it works. One person had a strong negative impression but it was caused by an OSM registration form broken in many ways.

(2) UX testing already resulted in multiple improvements. Many small issues were identified. Some of them are easy to fix and are already fixed. Fixed issues include improvements to a Polish translation, pull requests #1437, #1441, #1442, #1444, #1446, #1444, #1442, #1437, #1441, #1448, #1451, #1491, #1484, #1478, #1476, #1471. Some further pull requests are open: #1445, #1450, #1452, #1460, #1462, #1455 + some not yet included improvements of Polish translation.

(3) two of testers had contact with earlier version and considered current one as a massive improvement

(4) for people new to OSM registration on mobile is a step barrier. Especially as registration page has broken layout on mobile, registration with Gmail account failed completely (this is now fixed), registering with Gmail account still asks about password and mail. Registration page was describes as "it looks like something done by students for a late project, not a website of a serious project".

I opened openstreetmap/operations#311 (now fixed), openstreetmap/openstreetmap-website#2288 (kept open), openstreetmap/openstreetmap-website#2287 (rejected, but I feel that it is worth rediscussing, but only after fixing one that was not refused), openstreetmap/openstreetmap-website#2285 (rejected, but alternative was proposed in a comment)

(5) Missing tutorial/explanation was identified as a major issue. It turns out that for new users it is not clear how app works.

  • necessity to click on quest markers to proceed
  • need to zoom out to see quests in nearly all cases (#871)
  • waiting for quests to appear at start (#178, maybe it is feasible to use multiple overpass servers as proposed in #1145

were all NOT obvious and NOT clear. Nearly all people were clearly confused.

(6) some form of gamification was widely requested, proposed, suggested or implied as needed. One memorable comment was "it is a more boring Pokemon Go". It was not mentioned by single older person and strongly pushed as important by the youngest one.

Youngest proposed that stars should unlock something in app. For example by allowing to spend in-app currency to build a house, or to build your own map "or at least achievements"

(7) overall, based on UX test it is clear that at this moment application would benefit more from tweaking what exists (including also work like #1329, #1438) than from adding new quests. This was a bit surprising to me as I in past focused on and planned to continue work mostly on adding more quests

(8) problem shared by every single person was gluing map to a current location. Every single person attempted to pan map and failed to do it. Every single person failed to notice about ungluing. Every single person considered option used to unglue as hidden and not in the expected place. All were expecting that clicking on that button will locate them, not that it will trigger ungluing. Two suggested to detect attempts to pan map freely and to show message in such case.

(9) problem shared by all three who encountered quest about no longer existing feature - it was not clear to any of them that expected path was to select "can't say", select note, create note.

(10) Both who encountered lit quest and all three who triggered surface quest were really unsure which answer should be used, especially for borderline ones.

(11) Both people who wanted to go back from note creation menu had problem to do it. One of them proposed adding explicit back button in case where there are other buttons present. Note that Google recommends not doing this - see https://material.io/design/navigation/understanding-navigation.html#reverse-navigation linked from https://developer.android.com/design (though I would not trust advice about navigation from page with broken navigation)

(12) Quest loading indicator is generally not recognized as one. I admit that I personally looking at app interface I am unable to say whatever it is downloading quests, finished downloading or waits for new quota from Overpass API. Especially rolling back of red bar is a bit confusing.

(13) All people who tried to create note were unsure which language for note text was preferable. Unfortunately I have no good idea how to explain this one without pushing too much text into note creation dialog. (EDIT: now one version of fix is in #1471, thanks to @rugk for an idea)

(14) issues #1456, #1457, #1463, #1485 were created for more complicated problems that are more complicated to fix (and maybe are not worth fixing or fix will not make situation better, so I started from issue rather than coding immediately)

(15) not reported as issue for now - on Xiaomi Mi A2 lite in evening night style enabled itself, but only partially. Map style remained old, text style remained old, but background was dark, resulting in poorly readable text (not reported for now as I was unable to discover how night mode is enabling itself in auto mode despite looking at #1306)


UX testing info:

UX testing done on biased, small and unrepresentative sample.

But it is good enough to detect low hanging fruit and resulted in a very useful feedback.

To be more specific - I tested on group of 7 people. All were my friends or my family. All from Kraków, Poland and the same cultural group, almost all of them are young adults.

One person is an active OSM contributor, second had an unused OSM account. I also asked for additional feedback on some OSM forums (talk-us, talk-au, slack-us). I also checked comments at Google Play in Polish and English. I based some things on comments at HN in OSM-related submissions that were linked in slack-us.

Despite that group was biased, small, unrepresentative sample multiple clearly problematic issues were identified. Fixing ones that have clear and obvious solution and releasing new version is at this point the best use of energy. It will be more useful than testing further and confirming that identified problems are actually problems.

So I plan to do at least tiny step to fix each of issues identified by more than one person. Then wait for a release that will include all or nearly all UX improvements, and test again with next group of people. I plan also to ask for feedback in more communities - talk mailing list, reddit/r/openstreetmap, HN, maybe also other...


My work on UX testing was sponsored by a NGI Zero Discovery grant


Similar issues that are articles rather than issues:

Google Play Statistics 2019 #1416
#866 also had some interesting statistics

@westnordost

This comment has been minimized.

Copy link
Owner

commented Jul 5, 2019

Wow, that is a lot of input! Thank you for that, @matkoniecz
I will sticky this one, I imagine this can be a source of new tickets and discussion for some time!

@westnordost westnordost pinned this issue Jul 5, 2019

@westnordost westnordost changed the title report from looking how other people, especially newbies, use StreetComplete (UX testing) UX testing results Jul 5, 2019

@rugk

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2019

Great thing/testing.
Some ideas about some points following:

(7) overall, based on UX test it is clear that at this moment application would benefit more from tweaking what exists

I guess this is because you've tested with new users. This would need more long-term tests to see how easily "boring" it may get when they see the same quest again and again.
I think for those of us, who are more "experienced", it's always awesome to see a new quest type from time to time, to discover something new. Just as that games (or Pokemon Go) add new features/things from time to time, too…

(9)

Possibly unsolvable. We do not have that much screen space to put it all in there. Somewhere thre must be an "overflow" menu. But one could use hints/pointers to point there.
Or maybe use more common patterns users already know (like the three dots menus, e.g. in Firefox:
image
image )

(11)

I'd trust Google. They know what to do. And the back button is universal on Android.

(12)

So the issue is the loading bar is stucking? In this case, I think Firefox for Android had/has a nice effect: Just make the bar a gradient that is also moving. So there is always something moving ("loading" indicator).

(13)

Just a placeholder text (maybe an example?) when nothing is written?

@BjornRasmussen

This comment has been minimized.

Copy link

commented Jul 5, 2019

(6) some form of gamification was widely requested, proposed, suggested or implied as needed. One memorable comment was "it is a more boring Pokemon Go". It was not mentioned by single older person and strongly pushed as important by the youngest one.

Youngest proposed that stars should unlock something in app. For example by allowing to spend in-app currency to build a house, or to build your own map "or at least achievements"

I think that this wouldn't really be necessary to make StreetComplete more fun. Something simple like a leaderboard with world, national, and local "best StreetCompleters" would give users a sense of community and importance, but not to the extent of bribing users to enter incorrect information for points.

@matkoniecz

This comment has been minimized.

Copy link
Contributor Author

commented Jul 5, 2019

I also thought about "mappy days" - solving quest, any quest, on given day makes that day successful.

In general, certainly not all suggestion from users will be implemented (especially as some were contradictory). I mentioned only ones shared by many or in some other way valuable. Many things expressed only by a single person were not mentioned at all.

@Binnette

This comment has been minimized.

Copy link
Contributor

commented Jul 14, 2019

Hi @matkoniecz, you've made an impressive work here !

About item 6 (gamification) discussed by you and @BjornRasmussen :
If we want to "gamify" StreetComplete, add a leaderboard,... the first thing we need to do is to store 'players' points on a remote server. Because if you uninstall/reinstall StreetComplete, your score will be reseted to 0. So we will need a server, a database, an API and a lot of other stuffs, so we will spend time on developping StreetComplete "server" and less time on StreetComplete "client" on mobile. So I'm not sure, it is a great idea as I think we need to focus on mobile App. So "gamification" is not a priority for me.


Other thing in my mind:

  • OSM have a lot of validators, such as Osmose, KeepRight,... Can we consider adding quests from those validators issues. I think it can be a way to solve those issues when surveying with StreetComplete.
@BjornRasmussen

This comment has been minimized.

Copy link

commented Jul 14, 2019

@Binnette it's possible to get the scores of each player by analysing osm changeset tags. Every changeset created by SC leaves tags indicating what quest was answered. By looking at the changesets, it would be possible for an external system to create a StreetComplete leaderboard without any changes to the app.

@matkoniecz

This comment has been minimized.

Copy link
Contributor Author

commented Jul 14, 2019

@Binnette

Can we consider adding quests from those validators issues.

Potentially a good idea. In case of quest that matches requirements of StreetComplete quest (you can see them for example in https://github.com/westnordost/StreetComplete/blob/master/CONTRIBUTING.md#suggesting-new-quests ) new issues about specific proposals are welcomed! I remember looking through Osmose reports and failing to find a good candidates, I looked again at Keep Right and failed to find anything that would be a good quest and is not already supported.

@matkoniecz

This comment has been minimized.

Copy link
Contributor Author

commented Jul 17, 2019

So "gamification" is not a priority for me.

I also would not consider it as the very important thing, but I will give it higher priority than before UX test (from "maybe would be nice to have, I am not going to work on it at all" to "I will spend some time on this")

the first thing we need to do is to store 'players' points on a remote server

I had time for StreetComplete but no access to my PC so I ended writing a bit about such server (see #1485)

By looking at the changesets, it would be possible for an external system to create a StreetComplete leaderboard without any changes to the app.

Yes (I even have script that is listing popularity of each quest, I a now checking is it giving accurate statistics). Technically central server is not mandatory for synchronization but... More in #1485

@matkoniecz

This comment has been minimized.

Copy link
Contributor Author

commented Jul 23, 2019

@westnordost Do you have access to more precise total download count than "10 000+" displayed on Google Play to public?

As of early July 18 576 users made at least one edit - and I am curious how many downloaded app and failed to save even a single edit.

@westnordost

This comment has been minimized.

Copy link
Owner

commented Jul 23, 2019

In total, 20780 users installed the app. The app was uninstalled 12770 times. There was a huge peak in 2017, since mid 2018 the install and uninstall events evened out to roughly 250-500 installs and 150-350 uninstalls per month.

@westnordost

This comment has been minimized.

Copy link
Owner

commented Jul 23, 2019

89% would be a famously good conversion rate. Though, not included are the number of users that installed the app from F-Droid, because there are no statistics at all. I also wouldn't know how to at least get an estimate how many percent of users are from F-Droid.

@JeanFred

This comment has been minimized.

Copy link

commented Jul 25, 2019

I also wouldn't know how to at least get an estimate how many percent of users are from F-Droid.

Could that be recorded as changeset metadata?

As the Play version and the F-Droid version are different builds of the app, that sounds to me in scope of the changeset metadata.

This could be done either by appending this to the created_by tag − eg StreetComplete 12.1 (Google Play) & StreetComplete 12.1 (F-Droid) ; or in a different field like Potlatch2 does it (I guess the build field?)

@westnordost

This comment has been minimized.

Copy link
Owner

commented Jul 25, 2019

Hmm, it could. Then, rather as an extra tag in the changeset so that it all counts as the same application. But how would such information be useful beyond satisfy our curiosity?

@Binnette

This comment has been minimized.

Copy link
Contributor

commented Jul 25, 2019

@westnordost : i agree that, this extra tag is mainly for statistics/curiosity... It can also be suspected as tracking personnal data. Maybe users do not want to share which store they are using (after all, users just want to contribute to OSM and do not leak personnal data).

Here is an attempt to find a way to use this tag:
Maybe this tag can be useful to monitor deployment issue on stores (Play/F-Droid). For example, if every F-Droid users still use version X and Play users have version X+1. So may be we can more easily see in our "statistics" that there is a problem with F-Droid deployment.... Hum, I'm not convinced.

I can not find any other useful beyond for it...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.