Add anonymous analytics to Granted Approvals

[chrnorm]

Background

As developers of Granted Approvals we have very little visibility into how the project is being used. Typically we find out about deployments of Granted Approvals because a community member joins our Slack channel (usually when they have challenges getting started). Community members with successful deployments tend to be quieter because everything is working well.

This RFD discusses adding anonymous product analytics to Granted Approvals to help us improve the project. The questions which we want to answer through these include:

Question	Why
What Access Providers are in use?	Helps us to understand the impact of changes to providers; for widely-used alpha-version providers we will provide scripts to automatically update Access Rules when changes are made to Access Providers. Helps inform the direction of the project with the kinds of new Access Providers that are developed.
What percentage of Access Rules require manual approval?	Helps us prioritize manual approval policy improvements (such as 2-person approval for access) versus automated approval policy improvements (such as associating Linear/Jira tickets with access requests, or integrating with PagerDuty and other on-call platforms)
How quickly are Access Requests being approved?	One of the core values of Granted Approvals is that the access request workflow must be extremely fast. This includes the approval aspect. Our goal is that the median approval time is <10min but we have no way to measure this. Helps drive improvements around the request workflow, such as our notifications to approvers.
What version of Granted Approvals are teams running?	Helps us prioritize backporting patches to previous versions.

Given the nature of the project, collecting usage information is sensitive. We do not plan on collecting any identifiable information. This includes email addresses, AWS account IDs, Access Rule names/descriptions, deployment region, and identifiable deployment characteristics like a sign-in URL which may contain a company’s name. When collecting analytics we will not be logging IP addresses. We plan on implementing server-side analytics only, and are not planning on adding any client-side analytics JavaScript to the Granted Approvals web dashboards.

In terms of implementation, we plan on displaying a prompt to users informing them that anonymous analytics are collected. A deployment parameter will allow anonymous analytics to be enabled and disabled.

We plan on shipping a Go library github.com/common-fate/telem which will handle the dispatching of analytics events. The library will include a README with example events for each event type, as well as the source code itself being available for audit. We plan on measuring the performance of this library and measuring overall latencies to ensure that the library does not impact performance of Granted Approvals API endpoints.

We are in the process of drafting the Privacy Policy for the analytics data and will append it to this RFD once complete.

The initial analytics data that we plan on collecting are shown below.

{
  "event": "rule.created",
  "deployment": {
    "id": "DEP_ID",
    "release": "v0.9.0",
    "userCount": 100,
    "groupCount": 10
  },
  "data": {
    "usesSelectableOptions": true,
    "usesDynamicOptions": true,
    "updatedBy": "USR_ANALYTICS_ID",
    "provider": "commonfate/aws-sso@v2",
    "id": "RUL_ANALYTICS_ID",
    "maxDuration": 3600,
    "requiresApproval": true
  }
}

{
  "event": "rule.updated",
  "deployment": {
    "id": "DEP_ID",
    "release": "v0.9.0",
    "userCount": 100,
    "groupCount": 10
  },
  "data": {
    "usesSelectableOptions": true,
    "usesDynamicOptions": true,
    "updatedBy": "USR_ANALYTICS_ID",
    "provider": "commonfate/aws-sso@v2",
    "id": "RUL_ANALYTICS_ID",
    "maxDuration": 3600,
    "requiresApproval": true
  }
}

{
  "event": "request.created",
  "deployment": {
    "id": "DEP_ID",
    "release": "v0.9.0",
    "userCount": 100,
    "groupCount": 10
  },
  "data": {
    "requestedBy": "USR_ANALYTICS_ID",
    "provider": "commonfate/aws-sso@v2",
    "rule": "RUL_ANALYTICS_ID",
    "duration": 3600,
    "timingMode": "ASAP",
    "hasReason": true
  }
}

{
  "event": "request.review",
  "deployment": {
    "id": "DEP_ID",
    "release": "v0.9.0",
    "userCount": 100,
    "groupCount": 10
  },
  "data": {
    "requestedBy": "USR_ANALYTICS_ID",
    "reviewedBy": "USR_ANALYTICS_ID",
    "provider": "commonfate/aws-sso@v2",
    "rule": "RUL_ANALYTICS_ID",
    "duration": 3600,
    "timingMode": "ASAP",
    "hasReason": true,
    "review": "DENY"
  }
}

Where:

• DEP_ID is an anonymous identifier for the deployment, in the format dep_KSUID, where KSUID is a KSUID. This is used out of convenience as we make extensive use of KSUIDs in the Granted Approvals codebase for all resource identifiers.
• USR_ANALYTICS_ID is an anonymous identifier for the user, in the format usr_HASH, where HASH is the SHA256 hash of a user KSUID. Hashing the user IDs avoids dispatching resource IDs which are used internally by a Granted Approvals deployment.
• RUL_ANALYTICS_ID is an anonymous identifier for the rule, in the format rul_HASH, following the same approach as usr_HASH above.
• usesSelectableOptions indicates whether the Access Rule allows options to be selected
• usesDynamicOptions indicates whether the Access Rule uses Dynamic Fields (for example, our AWS OU feature uses this)
• requiresApproval indicates whether the Access Rule requires manual approval
• hasReason indicates whether a reason was provided when access was requested

We plan on using the following third-party subprocessors to process data:

• CloudFlare Workers
• Rudderstack
• Mixpanel

If the usage of third-party subprocessors is unacceptable to the community we will self-host the analytics stack ourselves.

We also plan to publish key metrics back to the community and share insights to openly discuss our roadmap. These metrics will be published in aggregate form only across all deployments.

Discussion

If there are no objections to this approach we plan to include an initial implementation of anonymous analytics in the next Granted Approvals release. We also don’t think this should be a one-off discussion and that a lack of objections at the moment gives us a free pass to collect anonymous analytics either. We aim for this RFD to be used as a discussion area on this topic and to continually seek feedback on this. Overall our aim is to be as ethical and open as possible, and to collect the information we feel we need to make Granted Approvals an amazing tool.

[Zordrak]

I fully support this, but main thoughts:

• Any analytics useful to Common Fate would probably be useful to the customer as well, and making it form the initial aspect of a proper observability solution would be very interesting. It would tempting to fully incorporate it into AWS CloudWatch and dashboard it there, being able to extract reports from it at will.
• Given the likelihood of production deployments being in highly secured possible-airgapped environments, it might be sensible to implement the reports as something that can be generated/downloaded ad-hoc by the user so that you can request an extract from a client where the service itself is incapable of phoning home.