Provide access to user research

[tobie]

At the F2F meeting in London, it was brought up that it's very difficult for the TSC or the AC to make decisions without access to existing but non-public user research.

@edent and @LJWatson volunteered to draft an opinion on the topic to submit to the AC for further discussion.

[KenjiBaheux]

@edent and @LJWatson I've talked with various folks to identify potentially relevant hypothesis, what we already have, what we could do for future work, what we can measure and share, etc. But without your input (opinion), it's unclear if any of this would be useful. Any update?

[LJWatson]

The goal is to be confident that new things are being created/developed in response to a user need, and that there is reasonable evidence of this.

I understand that Google does do user research, but not in a way that it is able to share publicly. My first suggestion is therefore that Google look at the way it conducts and collects user feedback, so that it is able to share the outcomes of that research without risking the anonymity of participants etc.

[edent]

Here's what I'd expect to see - using an example.

• We want to create amp-midi to let people put cool music on their site.
• We spoke to 50 website designers who currently use background music on their sites to see what they currently do, why they do it, and what challenges they face.
• We ran an online survey across the world to see what end users want from a sound on the web.

Users in LatAm loved music on pages, while those in Nordic countries would refuse to visit a page that autoplayed music.

People aged 18-25 had the most net favourable scores, but 30% of them reported a bad experience with an auto playing sound.

Developers expressed annoyance at the different audio formats (ogg, mp3, aac) and 64% of them wanted a single standard. 90% said they were worried about file size on performance speeds.

When presented with the idea of amp-midi developers responded with the following verbatim quotes.

---

That's the sort of thing I expect to see. You can fiddle with the numbers, the outreach, and the questions you ask. But that's the core of most user research. Talk to a wide range of users, listen to their needs.

[KenjiBaheux]

Thanks!

That's a bit different than what I had in mind (more on why, below).
That said, it's useful input.

I've been focusing on the user experiences enabled by AMP and related to features I work on, i.e. Signed Exchange, Portals. Unlike the <amp-midi> example, I think the questions for these user experiences would be framed in terms of "for users" and "for publishers", e.g. does experience X help users achieve their goals faster? does experience X result in more traffic for publishers than the statu quo?

I also noted that your input was mostly (solely?) focused on feedback as opposed to metrics. I guess we probably over-index on metrics ;) So, ideally, we would also survey users and publishers on top of the metrics. I'll float that back to the various folks involved.

[edent]

I guess we probably over-index on metrics ;)

[KenjiBaheux]

The same conclusion could be made about surveys ;) "It sounds like you selected easily satisfied participants, and/or gave them biased and easy questions rather than trying to make the experience good". Ultimately, metrics or surveys, it's still measuring something.

[LJWatson]

Worth mentioning the difference between user experience and user need: the former is how well something works for its target audience once it's available, the latter is whether there is a validated need for the thing in the first place. They're both important, but it's the latter we need to get better at (right across the industry).

[KenjiBaheux]

Agreed. Solving a user need / pain point comes first. We don't build things because we think it's cool, well except some of the April fools ( https://www.dandandin.net/the-real-april-fools-from-the-google-japanese-input-team/) ;)

…

[KenjiBaheux]

Just to remind everyone, I'm the messenger/mediator here, don't shoot me :)

I understand what an ideal picture* could look like but it's not easy nor always possible for various reasons. So, while these might not tick all the boxes, you may have seen these 2 blog posts that are relevant to the topic:

• https://developers.googleblog.com/2019/08/the-speed-benefit-of-amp-prerendering.html
• https://webmasters.googleblog.com/2019/07/helping-publishers-and-users-get-more.html

The first one talks about the speed benefits of AMP pre-rendering with graphs/data. The article doesn't (re)explain the user benefits of speed improvements / instant experience. I imagine that the plethora of research on that topic was deemed self-sufficient.

The second one talks about user and publisher benefits ("The speed and ease of this experience makes it more likely for users to visit a publisher's site, while still allowing users to continue their browsing session.") but without sharing concrete data. Not quite sure why... But, the good news is that the article also mentions that publishers will soon have insights about their traffic data from AMP in Google Images in the performance report.

Constructive feedback appreciated.

*: at one extreme this could be "share the raw data, user feedback to allow for independent analysis".

[tobie]

Worth noting the work happening around process for intent to implement (I2I), too. See notably:

• Update I2I template to include reviews amphtml#24072
• https://docs.google.com/document/d/1bF5tHbm-ObyjKLfoiIEAI5V-YvMsA3Bchsux5u4cKZM/edit

[LJWatson]

Thanks @KenjiBaheux.

You're right, the first of those posts probably doesn't need to provide evidence of user need. The need for improved performance is well documented and well understood.

The second post starts off with a reasonable statement about the ability to scan multiple ideas being an important part of making a decision, something we've known since long before the web was a thing. It comes a bit unraveled after that though.

Was there user research that qualified the claim that visual search was too slow, and validated the claim that it was slow enough to prevent task completion for some people?

Was there user research to validate the swipe to view feature as the chosen solution (or that caused other proposed solutions to be rejected)?

Was the chosen solution tested with users (including those with disabilities) before it was launched?

Was any of this done at a scale robust enough to be representative of the consumer audience?

Hope these comments help (and that I didn't shoot the messenger).

[LJWatson]

Thanks @KenjiBaheux .

You're right, the first of those posts probably doesn't need to provide evidence of user need. The need for improved performance is well documented and well understood.

The second post starts off with a reasonable statement about the ability to scan multiple ideas being an important part of making a decision, something we've known since long before the web was a thing. It comes a bit unraveled after that though.

Was there user research that qualified the claim that visual search was too slow, and validated the claim that it was slow enough to prevent task completion for some people?

Was there user research to validate the swipe to view feature as the chosen solution (or that caused other proposed solutions to be rejected)?

Was the chosen solution tested with users (including those with disabilities) before it was launched?

Was any of this done at a scale robust enough to be representative of thhe intended audience?

Hope this helps (and that i didn't shoot the messenger).

[KenjiBaheux]

Thanks @LJWatson I've shared this feedback. I hope it will inform follow-ups announcements and future work.