Feature: Ensure we have opt-out metrics for all of our projects

[SgtPooki]

eta: 2023-01-31

Follow #125 for a list of the metrics that should be implemented within each project.

Tasks

Beta Give feedback

1. create metrics collection/consent library for our use #129
   P0 dif/easy effort/days kind/enhancement status/in-progress +5
   
2. analysis: Investigate which projects are react or not ipfs-shipyard/ignite-metrics#32
   +0
   
3. Release 0.19 kubo#9502 - separate webui metrics for kubo
4. Feat: getting basic metrics in ipfs-companion ipfs-companion#1115
   Epic +1
   
5. Add pageview metrics to countly public-gateway-checker#306
   P1 exp/novice status/in-progress +3
   
6. feat: add basic metrics to StarMaps pln-planning-tools/Starmap#173
   P0 effort/hours enhancement post-MVP status/ready +5
   
7. feat: ignite-metrics - Add telemetry without UI ipld/explore.ipld.io#106
   +0
8. feat: ignite-metrics - Add telemetry without UI ipfs-shipyard/ipfs-dag-builder-vis#38
   +0
   
9. feat: ignite-metrics - Add telemetry without UI laurentsenta/pl-diagnose#10
   +0
   
10. feat: ignite-metrics - Add telemetry without UI ipfs-shipyard/pinning-service-compliance#272
    +0
    
11. feat: ignite-metrics - Add telemetry without UI multiformats/cid-utils-website#56
    +0
    
12. feat: ignite-metrics - Add telemetry without UI ipfs-shipyard/ipfs-check#28
    +0
    
13. Switch to opt-out metrics ipfs-desktop#2370
    P1 kind/enhancement kind/maintenance topic/dependencies +4
14. Switch to opt-out metrics ipfs-webui#2085
    effort/hours kind/enhancement kind/maintenance released +4
    
15. feat: Metrics update ipfs-webui#2074
    P0 effort/hours exp/beginner kind/enhancement status/ready +5
    
16. feat: finalize metrics modal and icon public-gateway-checker#340
    P1 effort/hours exp/novice good first issue kind/enhancement status/ready topic/design-ux +7
    

[BigLep]

Great to see this initiative.

For web-ui, will we able to differentiate between https://webui.ipfs.io/ and the webui that is bundled in Kubo?

[SgtPooki]

Great to see this initiative.

For web-ui, will we able to differentiate between https://webui.ipfs.io/ and the webui that is bundled in Kubo?

I hadn't planned to do that work with this initiative but I'll add an item under webui. See ipfs/ipfs-webui#2074

[BigLep]

@SgtPooki @tinytb : a few thoughts before this rolls out to more places:

1. I wanted to make sure some of the questions that could come to mind for users were seen: feat: add countly metrics public-gateway-checker#309 (comment)
2. There were some stylistic suggestions in feat: add countly metrics public-gateway-checker#309 (comment) that likely make sense to incorporate for all properties
3. For companion, capturing clicks into Companion makes sense and is good. The other metric that would be great is the number of ipfs:// URLs that it handles (and ideally breakdown between gateway domain: localhost, ipfs.io, Cloudflare, etc). The point is that someone could not be "engaging" with the plugin directly in a given week but still be an "active user" because they're getting value out of the plugin that week because the extension is resolving ipfs:// URLs.

[aschmahmann]

@BigLep @tinytb @SgtPooki any chance we could get https://check.ipfs.network/ (https://github.com/ipfs-shipyard/ipfs-check) added to the list of things we should have metrics for? Metrics here could be frontend, backend or both but generally serving the same point of showing how these resources are being used.

[SgtPooki]

@BigLep

1. I wanted to make sure some of the questions that could come to mind for users were seen: feat: add countly metrics public-gateway-checker#309 (comment)

Answered in ipfs/public-gateway-checker#309 (comment) but #125 has the most up to date information. Regarding metrics breakdown. We will also be moving to more of a modal view (see #340). These updates will be in effect before adding to other projects (except companion, but companion will have a breakdown)

2. There were some stylistic suggestions in feat: add countly metrics public-gateway-checker#309 (comment) that likely make sense to incorporate for all properties

Absolutely.

3. For companion, capturing clicks into Companion makes sense and is good. The other metric that would be great is the number of ipfs:// URLs that it handles (and ideally breakdown between gateway domain: localhost, ipfs.io, Cloudflare, etc). The point is that someone could not be "engaging" with the plugin directly in a given week but still be an "active user" because they're getting value out of the plugin that week because the extension is resolving ipfs:// URLs.

This is a great callout but may not be in for initial launch. We need to tread carefully with metrics we send about usage not related specifically to the app, and ensure things are in place to prevent tracking pageUrls with(or without) CIDs, so that's my main focus for initial opt-out metrics release with companion.

I'll create a section in the metrics collection notion doc (link will be provided in #125 shortly) to address this.

The most difficult problem is getting opt-out metrics in place and ensuring users are aware of our changes. Once that is done, modifying the type of metrics we send can be done fairly easily; though it will still require updating documentation about metrics sent for that project.

[SgtPooki]

@aschmahmann @tinytb

any chance we could get https://check.ipfs.network/ (https://github.com/ipfs-shipyard/ipfs-check) added to the list of things we should have metrics for? Metrics here could be frontend, backend or both but generally serving the same point of showing how these resources are being used.

Do you have a countly account? We don't technically own ipfs-check (though maybe we should) and so if no one on our team will be viewing the metrics and generating reports it won't benefit us much.

I will add it to the list but my primary focus is getting metrics for the following ignite team projects:

• Companion
• public-gateway-checker
• starmaps

And then ensuring desktop & webui are switched to opt-out.

[BigLep]

@SgtPooki : I'll read/watch #125 as it sounds like that's where more of the action is on specifics.

I do think we should add IPFS Check (and pl-diagnose) to the list and just use Ignite's account. You could say that the "owners" should do it, but they don't really have team ownership. Getting the metrics in place heps give clarity on whether we should invest more in them, and if we invest it likely makes sense for it to happen in Ignite (accompanied with increasedresourcing or adjusting other priorities).

They can be lower on the list, but given diagnostic tools are the kind of things we believe users across implementations find valuable (and that they reduce the burden on core maintainers) taking the 1-2 days to measure usage I think is worth it.

[SgtPooki]

note that telemetry documentation is merged and #125 closed. Please open an issue on ignite-metrics if there or questions/comments!

[SgtPooki]

re-arranged priority of projects according to ipfs-shipyard/ignite-metrics#32 (comment), and removed ipfs-share-files

cc @tinytb

[SgtPooki]

In order to get metrics applied to our non-vital projects, I'm going to move forward with enabling the metrics and creating a backlog task for adding the UI treatment (toggle UI) to those projects. This will help us prioritize getting the numbers we need, while still allowing users to opt out (instructions should be provided in the backlog tacks for adding the toggles).

[SgtPooki]

PR is out for adding telemetry to starmap project: pln-planning-tools/Starmap#272, but dependent upon some updates to ignite-metrics

[SgtPooki]

Status update:

A Dashboard with existing metrics can be found at https://pl-strflt.notion.site/IPFS-Ignite-Metrics-Dashboard-dee2ada6225e4e01a172673496851210

There is a callouts section on the dashboard page to indicate the current state, but here is a recap:

1. We are now collecting telemetry for all of our projects except ipfs-companion
2. ipfs-companion metrics will be added once the following is done:
   • feat: add telemetry to companion ipfs-companion#1117 - Need approval from @lidel
   • An update to the privacy policy (PR TBD)
   • A release (ETA: 2023-01-31)
3. ipfs-desktop and ipfs-webui will have switched to opt-out metrics once the following is done:
   • feat: different countly keys for kubo and webui.ipfs.io deployments ipfs-webui#2081 merged
   • A release is done (ETA: 2023-01-27)

This comment will also be added into the description

[SgtPooki]

📣 All IPFS Ignite team projects have opt-out metrics added, and are released.

We’re still waiting on updates to ipfs-companion in the chrome store, but ipfs-companion has a new release: https://github.com/ipfs/ipfs-companion/releases/tag/v2.21.0 and is currently available via firefox.

ipfs-webui has been released - https://github.com/ipfs/ipfs-webui/releases/tag/v2.22.0
ipfs-desktop has been released - https://github.com/ipfs/ipfs-desktop/releases/tag/v0.26.0 with the new webui version.

we're waiting on Kubo to merge update of latest webui version: ipfs/kubo#9597

What metrics don't we have yet?

• ipfs-desktop metrics from users who haven't updated to the latest version
• ipfs-companion metrics from users who haven't updated to the latest version
  • all chrome users
• ipfs-webui-kubo metrics for users loading ipfs-webui from kubo

When will we have all of our metrics?

• users update to latest ipfs-desktop version
• feat: ipfs-webui 2.22.0 kubo#9597 is merged
• kubo releases after merging feat: ipfs-webui 2.22.0 kubo#9597
• ipfs-companion is approved and released in chrome store (v2.21.0 at https://chrome.google.com/webstore/detail/ipfs-companion/nibjojkomfdiaoajekhjakgkdhaomnch)
• all ipfs-companion users have updated to latest version (method of this varies by user settings and browser)

cc @tinytb @BigLep

[SgtPooki]

status update:

• ipfs-companion 2.21.0 was released in chrome store yesterday: 2023-01-29
• kubo v0.19 release is ~5 weeks out

[SgtPooki]

marking this as blocked because the only remaining action item is to confirm that the ipfs-webui-kubo metrics are showing properly once a kubo release is made: ipfs/kubo#9502

[SgtPooki]

we're getting metrics from kubo webui now: https://countly.ipfs.tech/dashboard#/63c596762a7760344a6b2cfd/. Something is up with metrics in countly though.. when I just logged in they prompted me for 2FA, so i'm assuming the authentication for the robot client needs something as well

[SgtPooki]

kubo metrics have been pretty stable, and is a higher usecase than webui directly. closing this since all of our projects have metrics.