Add feature activation events

[jhugman]

Fixes EXP-3982.

This also refactors exposure events and malformed feature config events to use the newly formed callback interface. While this isn't strictly necessary for this PR, it seems a natural companion refactor to this activation event work.

Pull Request checklist

• Breaking changes: This PR follows our breaking change policy
  • This PR follows the breaking change policy:
    • This PR has no breaking API changes, or
    • There are corresponding PRs for our consumer applications that resolve the breaking changes and have been approved
• Quality: This PR builds and tests run cleanly
  • Note:
    • For changes that need extra cross-platform testing, consider adding [ci full] to the PR title.
    • If this pull request includes a breaking change, consider cutting a new release after merging.
• Tests: This PR includes thorough tests or an explanation of why it does not
• Changelog: This PR includes a changelog entry in CHANGELOG.md or an explanation of why it does not need one
  • Any breaking changes to Swift or Kotlin binding APIs are noted explicitly
• Dependencies: This PR follows our dependency management guidelines
  • Any new dependencies are accompanied by a summary of the due dilligence applied in selecting them.

Branch builds: add [firefox-android: branch-name] to the PR title.

[codecov-commenter]

Codecov Report

Attention: 58 lines in your changes are missing coverage. Please review.

Comparison is base (03e8b27) 26.14% compared to head (f546763) 26.12%.

Files	Patch %	Lines
components/nimbus/src/stateful/nimbus_client.rs	0.00%	37 Missing ⚠️
components/nimbus/src/metrics.rs	0.00%	21 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5908      +/-   ##
==========================================
- Coverage   26.14%   26.12%   -0.03%     
==========================================
  Files         374      374              
  Lines       47238    47296      +58     
==========================================
+ Hits        12349    12354       +5     
- Misses      34889    34942      +53     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jhugman]

Request for data collection review form

1. What questions will you answer with this data?

The activation event is as a proxy for the exposure event. When has a feature under experiment been shown to a user?

2. Why does Mozilla need to answer these questions?

Mozilla needs to answer this question in order to judge the success or not of an experiment: it will power the automated analysis of experiments.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We are currently using exposure events, but this requires manual instrumentation by feature developers. This is often not done before the experiment is launched, and by then it is too late.

The activation event is the default that analysis will fall back upon.

4. Can current instrumentation answer these questions?

In most cases, where the feature has been instrumented with an exposure event, then yes. However, this is for the cases where the feature has not been instrumented at all, or not been instrumented correctly.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Description	Data Collection Category	Tracking Bug #
Feature Activation	Technical data	https://mozilla-hub.atlassian.net/browse/EXP-3950

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.
   The documentation can be found in the proposal document linked in the ticket above (internal only), the metrics.yaml on this PR, and the Glean dictionary.

7. How long will this data be collected? Choose one of the following:

I want to permanently monitor this data.

• jhugman@mozilla.com
• project-nimbus@mozilla.com

8. What populations will you measure?

All populations that use the Nimbus SDK. Currently, all mobile apps made by Mozilla, but ultimately for webapps that use Nimbus too.

9. If this data collection is default on, what is the opt-out mechanism for users?

Nimbus has internal tooling for opting out of experimentation. This data is not collected if the feature is not under experimentation.

10. Please provide a general description of how you will analyze this data.

We'll analyze it in Jetstream and it will be used to create dashboards for experiment health as well as additional data.

11. Where do you intend to share the results of your analysis?

They will be shared internally to Mozilla through the Experimenter console as well as ad-hoc through queries against the raw Glean data.

12. Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

No

data-review? @eliserichards, @jeddai

[eliserichards]

Request for data collection review form

1. What questions will you answer with this data?

The activation event is as a proxy for the exposure event. When has a feature under experiment been shown to a user?

2. Why does Mozilla need to answer these questions?

Mozilla needs to answer this question in order to judge the success or not of an experiment: it will power the automated analysis of experiments.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We are currently using exposure events, but this requires manual instrumentation by feature developers. This is often not done before the experiment is launched, and by then it is too late.

The activation event is the default that analysis will fall back upon.

4. Can current instrumentation answer these questions?

In most cases, where the feature has been instrumented with an exposure event, then yes. However, this is for the cases where the feature has not been instrumented at all, or not been instrumented correctly.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the [Firefox data collection categories](https://wiki.mozilla.org/Data_Collection) found on the Mozilla wiki.

Measurement Description Data Collection Category Tracking Bug #
Feature Activation Technical data https://mozilla-hub.atlassian.net/browse/EXP-3950

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.
   The documentation can be found in the proposal document linked in the ticket above (internal only), the [metrics.yaml](https://github.com/mozilla/application-services/pull/5857/files#diff-ca3094e555a194d8c839be789f65c5434e9d04ad84ec363a3072cd89225d9a83) on this PR, and the [Glean dictionary](https://dictionary.telemetry.mozilla.org/).

7. How long will this data be collected?  Choose one of the following:

I want to permanently monitor this data.

* [jhugman@mozilla.com](mailto:jhugman@mozilla.com)

* [project-nimbus@mozilla.com](mailto:project-nimbus@mozilla.com)


8. What populations will you measure?

All populations that use the Nimbus SDK. Currently, all mobile apps made by Mozilla, but ultimately for webapps that use Nimbus too.

9. If this data collection is default on, what is the opt-out mechanism for users?

Nimbus has internal tooling for opting out of experimentation. This data is not collected if the feature is not under experimentation.

10. Please provide a general description of how you will analyze this data.

We'll analyze it in Jetstream and it will be used to create dashboards for experiment health as well as additional data.

11. Where do you intend to share the results of your analysis?

They will be shared internally to Mozilla through the Experimenter console as well as ad-hoc through queries against the raw Glean data.

12. Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

No

data-review? @eliserichards, @jeddai

Data Review Form (to be filled by Data Stewards)

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

• Yes, through metrics.yaml and the Glean Dictionary, as well as the proposal document linked in EXP-3982.

2. Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism) Provide details as to the control mechanism available.

• Yes, through the "Send Usage Data" preference in the application settings, as well as Nimbus internal tooling.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

• Yes, this data will be permanently collected. James Hugman (jhugman@mozilla.com) will be monitoring as well as the Nimbus team (project-nimbus@mozilla.com).

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

• Category 1 - technical data.

5. Is the data collection request for default-on or default-off?

• Default on.

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

• No.

7. Is the data collection covered by the existing Firefox privacy notice? If unsure: escalate to legal if:

• Yes.

8. Does the data collection use a third-party collection tool? If yes, escalate to legal.

• No.

Result

data-review+

[jeddai]

r+wc

This looks good! 🎉

[jeddai]

To prevent a similar issue to the enrollment_status metric overload, can we initially add this as disabled and enable it via glean server knobs in nightly?

Suggested change

	expires: never
	expires: never
	disabled: true