Telemetry for Bookmarks

[bifleming]

As a product owner, I want to track the usage of bookmarks in Fenix so that I can understand how our users leverage the feature, and detect anomalies or issues in user flow and usage.

Hypothesis

We believe this feature is useful to users, and successful when

• % of users bookmarking increases
• % of users opening website via bookmark increases

What questions will you answer with this data?

• How do people arrive at bookmark list (via e.g. library etc.) @topotropic - is coming from Library the only way to get to the bookmark list? (removing this as there will be only one way of getting to this list)
• Do people open, save, add, delete, move, share, copy bookmarks
• Do people create, delete folders
• Do people search for bookmarks in bookmark library list
• After bookmark saved, how often to users immediately edit it

Probes

• Folder created, deleted
• Bookmark opened, saved, added, deleted, moved, shared, copied
• Search used in bookmark library list
• Clicked edit label after bookmark was created

Dependencies (Added by PM and Eng)

• Glean SDK (Event ping)

Acceptance Criteria (Added by PM)

• Event pings can be queried via re:dash
• We are sending telemetry events for the actions listed in the requirements
• We have documented the telemetry
• We have asked a data steward to review the telemetry
• NOT an AC: Data science to create dashboard or further graphs (this will be a separate issue, this issue is only about hooking up the events described and that they can be queried in re-dash)

┆Issue is synchronized with this Jira Task

[topotropic]

* @topotropic - is coming from Library the only way to get to the bookmark list?

Yes. That's a great issue by the way (never thought I would write that) – maybe add something about sync, or is this a separate issue?

Looking forward to learn more about the features. Thanks!

[georgf]

As a product owner, I want to track the usage of bookmarks in Fenix so that I can understand how our users leverage the feature, and detect anomalies or issues in user flow and usage.

The part about "detect anomalies or issues" is interesting.
Does that mean you'd expect to analyse them and look for odd patterns manually through graphs or similar?

What questions will you answer with this data?

...
* After bookmark saved, how often to users immediately edit it

This is interesting to look at. I assume this would require:

• looking at edit actions per individual user
• filtering by timespan between the save event and the edit event
  Can amplitude do that?
  In redash that should be possible.

[bbinto]

The part about "detect anomalies or issues" is interesting.
Does that mean you'd expect to analyse them and look for odd patterns manually through graphs or similar?

I'm relying on Amplitude here

This is interesting to look at. I assume this would require:

* looking at edit actions per individual user

* filtering by timespan between the save event and the edit event
  Can amplitude do that?
  In redash that should be possible.

I'm relying on Amplitude here

[felixlawrence]

I am reviewing this from a data science perspective, asking whether this proposal is going to make it easy for me and other DSes to do our jobs and meet expectations once it's implemented. I'm not a data steward.

The probes seem well defined to me - I would change "Clicked edit label after bookmark was created" to "Clicked edit label", so that when the telemetry is being implemented there is no ambiguity about whether a click on the edit label is only recorded if it's immediately after the bookmark was created.

Most of the questions seem well defined too.

The only problem I noticed is that "How do people arrive" is going to be hard to work out. I don't think Firefox Desktop or Glean have a telemetry pattern for following users paths through the products. Whether we want such a pattern, what % of users we need this data from, and how it fits with our lean data principles is worth discussing at some point; that could be a big discussion.

One systematic way to work out "how do people arrive" would be to do something like sending navigation events for every visited part of the app - a DS can then sort these by timestamp to determine the user's path through the app. Any non-instrumented parts would introduce invisible gaps in the path. If we want to go down that path then we should put some thought into this system rather than immediately pursuing with the approach with which I am most familiar!

The ad-hoc, one-off solution would be to do something specific to the bookmark list: exhaustively work out all the paths to the bookmark list then send an event on loading the bookmark list specifying which path was used.

Separately, I'm not sure that regularly auto-refreshing dashboards are appropriate here - to me this smells more like a one-off report (to be re-run whenever the subject is revisited), or telemetry that gets used in an experiment. But I haven't seen the roadmap for bookmarks.

[bbinto]

Thanks @felixlawrence

The only problem I noticed is that "How do people arrive" is going to be hard to work out. I don't think Firefox Desktop or Glean have a telemetry pattern for following users paths through the products.

I would be relying on Amplitude for this, however comment #974 (comment) makes this obsolete (removing it from the description now)

[felixlawrence]

r+ then!

[colintheshots]

I think the above list of probes drastically undercounts the available UX features to interact with for bookmarks. Should I add to the list of probes as I feel is appropriate?

[bbinto]

@colintheshots please add them

[bsurd]

There is telemetry for most of the bookmark management options except for when deleting a folder.

@colintheshots Could you look into this?

[vesta0]

@colintheshots we can add telemetry for deleting a folder post-mvp if you haven't done it already.

[colintheshots]

@vesta0 Yes, we can add it post-MVP. This is still labeled a release blocker though.

[vesta0]

I didn't want to break it apart if it was already in progress but based on feedback from @colintheshots and @bsurd I think we can close this issue and track the remaining work here: #3174