[Run][Plugin Manager] Add telemetry

[mykhailopylyp]

Link to parent issue #5273.

Add measurements https://github.com/microsoft/PowerToys/wiki/PowerToys-Run-Plugin-spec#6-measurements

[mykhailopylyp]

@enricogior @ivan100sic

1. The enable / disable state. Validate if a plugin should be off by default based on population.
   Send event on enable/disable(Plugin name, Is Enabling)
   Then the number of people who disabled the plugin would be equal to the number of disabling minus the number of enabling.
   So we can determine percentage of active users who do not use the plugin.

2. Action word for plug. Validate if we should shift the Action word
   Send event on action keyword change(new action keyword, plugin name).
   So we can group by action keyword and plugin name and find percentage among active users.

3. Include in global result list for plug. Validate if we should shift the Action word from the global list
   Same as for 1.

4. Action word for plug. Validate if we should shift the default Action word.
   Duplicate of 2.

5. Is a direct shortcut used for a plugin. Validate if we should make certain plugins have direct shortcuts. If so, we should add in one more bit of telemetry, what shortcut was used to see if there are commonalities
   Will not be implemented in 0.33.

It would be useful to know how many active PowerToys Run users used Plugin settings to determine how successful the feature is. Can it be determined from the above events?

Also, we should think about thresholds to take an action in advance. For example, if 80% of active users disabled a plugin then we should make it disabled by default.

Any suggestions on how we can change events to get more precise results? Maybe you have an idea what else we can verify?

[ivan100sic]

Then the number of people who disabled the plugin would be equal to the number of disabling minus the number of enabling

You would miss all the people who just leave the plugin on by default.

Perhaps a better approach would be to send the entire plugin settings each time PT Run is used?

[enricogior]

@mykhailopylyp
we need to simplify things as much as possible to make easy to understand what type of data the telemetry provides.
When considering a telemetry event, we should start asking if that event will provide enough information to make us decide to change something.
Knowing how many times someone is changing any option in the settings isn't really useful, what we need to know is for each settings the percentage of people that are using the default value and how many have changed the default value.

We should not sent events while the user is still changing settings, we should only send an event after the settings app has been closed and in that even we will send the list of all settings for each plugin.

[mykhailopylyp]

Perhaps a better approach would be to send the entire plugin settings each time PT Run is used?

It is a decent solution. I like that we only consider users of PT Run.

[enricogior]

@mykhailopylyp @ivan100sic
we should send the telemetry event for the PT settings when the Settings app is closed and some settings have been changed.
When the user uses PT Run we sent the event for that, not for the settings.

[mykhailopylyp]

we should send the telemetry event for the PT settings when the Settings app is closed and some settings have been changed.
When the user uses PT Run we sent the event for that, not for the settings.

Do we have a way to determine if a user uses PowerToys Run? If we send plugin settings from the settings app would it be possible to get plugin usage data only for PT Run users?

[enricogior]

This turned out to be a larger work item than expected because we don't have a good infrastructure to report Settings events.
We need to take a step back and make sure we adopt the correct way of sending telemetry when users change any option.
Should we send individual events for each option or the full list of options for each module when one or more options for that module are changed?
We also need to make sure that all modules use the same approach.

[crutkas]

report all settings, this will then allow us only look at last reported.