⚠️ Disclaimer: This project is open sourced to solve problems that are critical to some users, and the functions provided may not be natively provided by GitHub Copilot. Therefore the contents, opinions and views expressed in this project are solely mine do not necessarly refect the views of my employer, These are my personal notes based on myunderstanding of the code and trial deployments to GitHub Copilot. If anything is wrong with this article, please let me know through the issues. l appreciate your help in correcting my understanding.
✅Risk Warning: This project uses the standard Copilot REST API to obtain data, aggregate data, and visualize it, without any potential risks.
Copilot Usage Advanced Dashboard 教程
| Version | Update Notes | Date |
|---|---|---|
| 1.0 | Document creation | 20241217 |
| 1.1 | Some updates | 20241218 |
| 1.1 | Add new templates | 20241221 |
| 1.2 | Support Copilot Standalone, thanks sombaner's great feedback | 20241224 |
| 1.3 | Compatible with metrics API | 20250208 |
| 1.4 | 1. Distinguish between insert and copy events of chat 2. Add model filter variables 3. Compatible with organizations that do not have teams 4. Fixed some bugs, for upgrades to older versions before 20250220, please refer to Old version (<=20250220) upgrade steps |
20250222 |
| 1.5 | Add daily usage history for each user, old version upgrade guide refer to this issue | 20250404 |
| 1.6 | refactor timezone handling in main.py & Docker run ENV paras | 20250410 |
| 1.7 | Add Elasticsearch authentication | 20250411 |
Copilot Usage Advanced Dashboard is a single data panel display that almost fully utilizes data from Copilot APIs, The APIs used are:
- List teams of an organization
- Get a summary of Copilot metrics for a team
- Get Copilot seat information and settings for an organization
- List all Copilot seat assignments for an organization
representing Copilot usage in multi organizations & teams from different dimensions. The features are summarized as follows:
- Data is persisted in Elasticsearch and visualized in Grafana, not just the past 28 days. So you can freely choose the time period you want to visualize, such as the past year or a specific month.
- You can freely adjust the style and theme of the chart, everything is a mature feature of Grafana.
- All stored data includes Organization and Team fields, which is convenient for data filtering through variable filters.
- Generate a unique hash key for each piece of data, and update the stored data every time the latest data is obtained.
- Visualizations in Grafana dashboards can be adjusted or deleted according to actual needs.
- Based on Grafana's built-in alerting function, you can set alert rules for some inappropriate usage behaviors, such as sending alerts to users who have been inactive for a long time.
- It can be easily integrated with third-party systems, whether it is extracting data from Elasticsearch to other data visualization platforms for data visualization, or adding other data sources in the Copilot Usage Advanced Dashboard for joint data visualization.
Designed 2 dashboards, both can exist at the same time in Grafana.
Copilot Metrics Viewer compatible dashboard
If you are familiar with the copilot-metrics-viewer project, then please try this dashboard and use it in subsequent deployments.
-
username：
demouser -
password：
demouser
New designed dashboard
-
username：
demouser -
password：
demouser
Supports four filtering varibales, namely
- Organzation
- Team
- Language
- Editor
The choice of variables is dynamically associated with the data display
First, based on List teams of an onganization, get all the teams under the Organization, and then based on Get a summary of Copilot usage for a team, sum and calculate the data of all teams under the Organization to get complete Organization-level data.
- Acceptance Rate Average =
sum(total_acceptances_count) / sum(total_suggestions_count) - Cumulative Number of Acceptence (Count) =
sum(total_acceptances_count) - Cumulative Number of Suggestions (Count) =
sum(total_suggestions_count) - Cumulative Number of Lines of Code Accepted =
sum(total_lines_accepted) - Acceptance Rate (%) =
total_acceptances_count / total_suggestions_count - Total Active Users =
total_active_users - Total Suggestions & Acceptances Count =
total_suggestions_count&total_acceptances_count - Total Lines Suggested & Accepted =
total_lines_suggested&total_lines_accepted
Based on the breakdown data in Get a summary of Copilot usage for a team, the data is aggregated by Teams to obtain data comparisons of different Teams.
- Number of Teams =
unique_count(team_slug) - Top Teams by Accepted Prompts =
sum(acceptances_count).groupby(team_slug) - Top Teams by Acceptance Rate =
sum(acceptances_count).groupby(team_slug) / sum(suggestions_count).groupby(team_slug) - Team Breakdown =
sum(*).groupby(team_slug)
Based on the breakdown data in Get a summary of Copilot usage for a team, the data is aggregated according to Languages to obtain data comparisons for different Languages.
- Number of Languages=
unique_count(language) - Top Languages by Accepted Prompts =
sum(acceptances_count).groupby(language) - Top Languages by Acceptance Rate =
sum(acceptances_count).groupby(language) / sum(suggestions_count).groupby(language) - Languages Breakdown =
sum(*).groupby(language)
Based on the breakdown data in Get a summary of Copilot usage for a team, the data is aggregated by Editors to obtain data comparisons for different Editors.
- Number of Editors =
unique_count(editor) - Top Editors by Accepted Prompts =
sum(acceptances_count).groupby(editor) - Top Editors by Acceptance Rate =
sum(acceptances_count).groupby(editor) / sum(suggestions_count).groupby(editor) - Editors Breakdown =
sum(*).groupby(editor)
Based on the data from Get a summary of Copilot usage for a team, we can get the usage of Copilot Chat.
- Acceptance Rate Average =
sum(total_chat_acceptances) / sum(total_chat_turns) - Cumulative Number of Acceptances =
sum(total_chat_acceptances) - Cumulative Number of Turns =
sum(total_chat_turns) - Total Acceptances | Total Turns Count =
total_chat_acceptances|total_chat_turns - Total Active Copilot Chat Users =
total_active_chat_users
Based on the data analysis of Get Copilot seat information and settings for an organization and List all Copilot seat assignments for an organization, the seat allocation and usage are presented in a unified manner.
- Copilot Plan Type =
count(seats).groupby(plan_type) - Total =
seat_breakdown.total - Active in this Cycle =
seat_breakdown.active_this_cycle - Assigned But Never Used =
last_activity_at.isnan() - Inactive in this Cycle =
seat_breakdown.inactive_this_cycle - Ranking of Inactive Users ( ≥ 2 days ) =
today - last_activity_at - All assigned seats =
*
Based on the breakdown data in Get a summary of Copilot usage for a team, we analyze the data from two dimensions: Languages and Editors. We can clearly see what combination of Languages and Editors can achieve the best Copilot usage effect.
- Active Users Count (Group by Language) =
active_users.groupby(language) - Accept Rate by Count (%) =
sum(acceptances_count).groupby(language) / sum(suggestions_count).groupby(language) - Accept Rate by Lines (%) =
sum(lines_accepted).groupby(language) / sum(lines_suggested).groupby(language) - Active Users Count (Group by Editor) =
active_users.groupby(editor) - Accept Rate by Count (%) =
sum(acceptances_count).groupby(editor) / sum(suggestions_count).groupby(editor) - Accept Rate by Lines (%) =
sum(lines_accepted).groupby(editor) / sum(lines_suggested).groupby(editor)
- You can view the distribution of seats, Enterprise or Business? and overall activation trends. And for users who don't use Copilot, they are ranked based on the length of inactivity and list users who have never activated.
- Ranking Language and Teams based on usage
You can analyze the total number of recommendations and adoption rate trends based on Count Lines and Chats
You can analyze the effect of Copilot in different languages and different editor combinations.
When setting up a deployment you will need to set the following variables for your pipeline manually:
| Variable | Description |
|---|---|
| AZURE_CLIENT_ID | The Client ID of the identity you want to use to deploy the application. |
| AZURE_ENV_NAME | The name of the Azure environment you want to deploy to, such as copilot-usage-advanced-dashboard-dev. |
| AZURE_LOCATION | The Azure location you want to deploy to, such as eastus, westus, etc. |
| AZURE_RESOURCE_GROUP | The name of the resource group you want to deploy to. |
| AZURE_SUBSCRIPTION_ID | The GUID for the subscription you want to deploy to. |
| AZURE_USER_PRINCIPAL_ID | The Object ID of a user you want to grant access to to the Azure Key Vault. |
| AZURE_TENANT_ID | The Azure Tenant ID of the identity you want to use to deploy the application. |
| GH_ORGANIZATION_SLUGS | This is your GitHub Organization name. This can be a comma-separated list of orgs if you want to index multiple orgs. |
| GH_PAT | This is your GitHub Personal Access Token. Mark this variable as secret in your pipeline. |
| AZURE_AUTHENTICATION_ENABLED | Enable Entra ID Single-Sign On (SSO) authentication. |
| AZURE_AUTHENTICATION_CLIENT_ID | The Client ID of the Azure AD application. |
| AZURE_AUTHENTICATION_OPEN_ID_ISSUER | The OpenID Connect issuer URL for Azure AD. |
If you are using Azure DevOps, make sure you change the name of the service connection to the name of your service connection. You will need to change line
30 and 45 of the azure-dev.yml file located in the .azdo/pipelines folder.
To create a service connection you can use the azd pipeline config --provider azdo command from the terminal. You can read more here: https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/pipeline-azure-pipelines.
You will need to install the "Install azd" extension from the marketplace in your Azure DevOps organization if you haven't already done so.
You will need to manually create the DevOps variables yourself in the Azure DevOps GUI.
You will create a pipeline using the azure-dev.yml file located in the .github/workflows folder. You will need to manually create the GitHub variables yourself in the GitHub GUI.
if you are using Azure Container Apps, please refer to the Azure Container Apps deployment document.
Deployment adds required role assignments, which require User Access Administrator RBAC role (more permissions than Contributor) or Owner RBAC role.
To deploy as Contributor RBAC role, pass doRoleAssignments parameter set to false or set AZURE_ROLE_ASSIGNMENTS environment variable when deploying with azd.
Required role assignments:
- Key Vault Secrets Officer on the KeyVault assigned to User Assigned Identity
- AcrPull on the Azure Container Registry assigned to User Assigned Identity
- Storage File Data SMB Share Contributor on the Storage Account assigned to User Assigned Identity
If you are not using Azure, you can use Linux with Docker, please refer to the Linux with Docker deployment document.
For cloud native deployment on Kubernetes, please refer to the Kubernetes deployment document.
At this point, return to the Grafana page and refresh. You should be able to see the data.
or
