⚠️ Disclaimer: This project is open sourced to solve problems that are critical to some users, and the functions provided may not be natively provided by GitHub Copilot. Therefore the contents, opinions and views expressed in this project are solely mine do not necessarly refect the views of my employer, These are my personal notes based on myunderstanding of the code and trial deployments to GitHub Copilot. If anything is wrong with this article, please let me know through the issues. l appreciate your help in correcting my understanding.
✅Risk Warning: This project uses the standard Copilot REST API to obtain data, aggregate data, and visualize it, without any potential risks.
Copilot Usage Advanced Dashboard 教程
| Version | Update Notes | Date |
|---|---|---|
| 1.0 | Document creation | 20241217 |
| 1.1 | Some updates | 20241218 |
| 1.1 | Add new templates | 20241221 |
| 1.2 | Support Copilot Standalone, thanks sombaner's great feedback | 20241224 |
| 1.3 | Compatible with metrics API | 20250208 |
| 1.4 | 1. Distinguish between insert and copy events of chat 2. Add model filter variables 3. Compatible with organizations that do not have teams 4. Fixed some bugs, for upgrades to older versions before 20250220, please refer to Old version (<=20250220) upgrade steps |
20250222 |
| 1.5 | Add daily usage history for each user, old version upgrade guide refer to this issue | 20250404 |
| 1.6 | refactor timezone handling in main.py & Docker run ENV paras | 20250410 |
| 1.7 | Add Elasticsearch authentication | 20250411 |
Copilot Usage Advanced Dashboard is a single data panel display that almost fully utilizes data from Copilot APIs, The APIs used are:
- List teams of an organization
- Get a summary of Copilot metrics for a team
- Get Copilot seat information and settings for an organization
- List all Copilot seat assignments for an organization
representing Copilot usage in multi organizations & teams from different dimensions. The features are summarized as follows:
- Data is persisted in Elasticsearch and visualized in Grafana, not just the past 28 days. So you can freely choose the time period you want to visualize, such as the past year or a specific month.
- You can freely adjust the style and theme of the chart, everything is a mature feature of Grafana.
- All stored data includes Organization and Team fields, which is convenient for data filtering through variable filters.
- Generate a unique hash key for each piece of data, and update the stored data every time the latest data is obtained.
- Visualizations in Grafana dashboards can be adjusted or deleted according to actual needs.
- Based on Grafana's built-in alerting function, you can set alert rules for some inappropriate usage behaviors, such as sending alerts to users who have been inactive for a long time.
- It can be easily integrated with third-party systems, whether it is extracting data from Elasticsearch to other data visualization platforms for data visualization, or adding other data sources in the Copilot Usage Advanced Dashboard for joint data visualization.
Designed 2 dashboards, both can exist at the same time in Grafana.
Copilot Metrics Viewer compatible dashboard
If you are familiar with the copilot-metrics-viewer project, then please try this dashboard and use it in subsequent deployments.
-
username：
demouser -
password：
demouser
New designed dashboard
-
username：
demouser -
password：
demouser
Supports four filtering varibales, namely
- Organzation
- Team
- Language
- Editor
The choice of variables is dynamically associated with the data display
First, based on List teams of an onganization, get all the teams under the Organization, and then based on Get a summary of Copilot usage for a team, sum and calculate the data of all teams under the Organization to get complete Organization-level data.
- Acceptance Rate Average =
sum(total_acceptances_count) / sum(total_suggestions_count) - Cumulative Number of Acceptence (Count) =
sum(total_acceptances_count) - Cumulative Number of Suggestions (Count) =
sum(total_suggestions_count) - Cumulative Number of Lines of Code Accepted =
sum(total_lines_accepted) - Acceptance Rate (%) =
total_acceptances_count / total_suggestions_count - Total Active Users =
total_active_users - Total Suggestions & Acceptances Count =
total_suggestions_count&total_acceptances_count - Total Lines Suggested & Accepted =
total_lines_suggested&total_lines_accepted
Based on the breakdown data in Get a summary of Copilot usage for a team, the data is aggregated by Teams to obtain data comparisons of different Teams.
- Number of Teams =
unique_count(team_slug) - Top Teams by Accepted Prompts =
sum(acceptances_count).groupby(team_slug) - Top Teams by Acceptance Rate =
sum(acceptances_count).groupby(team_slug) / sum(suggestions_count).groupby(team_slug) - Team Breakdown =
sum(*).groupby(team_slug)
Based on the breakdown data in Get a summary of Copilot usage for a team, the data is aggregated according to Languages to obtain data comparisons for different Languages.
- Number of Languages=
unique_count(language) - Top Languages by Accepted Prompts =
sum(acceptances_count).groupby(language) - Top Languages by Acceptance Rate =
sum(acceptances_count).groupby(language) / sum(suggestions_count).groupby(language) - Languages Breakdown =
sum(*).groupby(language)
Based on the breakdown data in Get a summary of Copilot usage for a team, the data is aggregated by Editors to obtain data comparisons for different Editors.
- Number of Editors =
unique_count(editor) - Top Editors by Accepted Prompts =
sum(acceptances_count).groupby(editor) - Top Editors by Acceptance Rate =
sum(acceptances_count).groupby(editor) / sum(suggestions_count).groupby(editor) - Editors Breakdown =
sum(*).groupby(editor)
Based on the data from Get a summary of Copilot usage for a team, we can get the usage of Copilot Chat.
- Acceptance Rate Average =
sum(total_chat_acceptances) / sum(total_chat_turns) - Cumulative Number of Acceptances =
sum(total_chat_acceptances) - Cumulative Number of Turns =
sum(total_chat_turns) - Total Acceptances | Total Turns Count =
total_chat_acceptances|total_chat_turns - Total Active Copilot Chat Users =
total_active_chat_users
Based on the data analysis of Get Copilot seat information and settings for an organization and List all Copilot seat assignments for an organization, the seat allocation and usage are presented in a unified manner.
- Copilot Plan Type =
count(seats).groupby(plan_type) - Total =
seat_breakdown.total - Active in this Cycle =
seat_breakdown.active_this_cycle - Assigned But Never Used =
last_activity_at.isnan() - Inactive in this Cycle =
seat_breakdown.inactive_this_cycle - Ranking of Inactive Users ( ≥ 2 days ) =
today - last_activity_at - All assigned seats =
*
Based on the breakdown data in Get a summary of Copilot usage for a team, we analyze the data from two dimensions: Languages and Editors. We can clearly see what combination of Languages and Editors can achieve the best Copilot usage effect.
- Active Users Count (Group by Language) =
active_users.groupby(language) - Accept Rate by Count (%) =
sum(acceptances_count).groupby(language) / sum(suggestions_count).groupby(language) - Accept Rate by Lines (%) =
sum(lines_accepted).groupby(language) / sum(lines_suggested).groupby(language) - Active Users Count (Group by Editor) =
active_users.groupby(editor) - Accept Rate by Count (%) =
sum(acceptances_count).groupby(editor) / sum(suggestions_count).groupby(editor) - Accept Rate by Lines (%) =
sum(lines_accepted).groupby(editor) / sum(lines_suggested).groupby(editor)
- You can view the distribution of seats, Enterprise or Business? and overall activation trends. And for users who don't use Copilot, they are ranked based on the length of inactivity and list users who have never activated.
- Ranking Language and Teams based on usage
You can analyze the total number of recommendations and adoption rate trends based on Count Lines and Chats
You can analyze the effect of Copilot in different languages and different editor combinations.
Dependent technology stack:
- Azure Container Apps
- Elasticsearch
- Grafana
- Python3
GITHUB_PAT:- Your GitHub account needs to have Owner permissions for Organizations.
- Create a personal access token (classic) of your account with the
manage_billing:copilot,read:enterprise,read:orgscope. Then please replace<YOUR_GITHUB_PAT>with the actual PAT. - If you encounter PAT permission error, please Allow access via fine-grained personal access tokens in Organization's Settings - Personal access tokens.
ORGANIZATION_SLUGS: The Slugs of all Organizations that you want to monitor, which can be one or multiple separated by,(English symbol). If you are using Copilot Standalone, use your Standalone Slug here, prefixed withstandalone:, for examplestandalone:YOUR_STANDALONE_SLUG. Please replace<YOUR_ORGANIZATION_SLUGS>with the actual value. For example, the following types of values are supported:myOrg1myOrg1,myOrg2standalone:myStandaloneSlugmyOrg1,standalone:myStandaloneSlug
LOG_PATH: Log storage location, not recommended to modify. If modified, you need to modify the-vdata volume mapping simultaneously.EXECUTION_INTERVAL: Update interval, the default is to update the program every1hours.ELASTICSEARCH_URL: The URL of your Elasticsearch, the default ishttp://localhost:9200, if you have modified the port, please modify it here.TZ: Timezone, the default isGMT, if you want to change it to your local timezone, please refer to tz database. For example, if you are in Toronto, please change it toAmerica/Toronto.
docker run -itd \
--net=host \
--restart=always \
--name cpuad \
-e GITHUB_PAT="<YOUR_GITHUB_PAT>" \
-e ORGANIZATION_SLUGS="<YOUR_ORGANIZATION_SLUGS>" \
-e LOG_PATH="logs" \
-e EXECUTION_INTERVAL=1 \
-e ELASTICSEARCH_URL="http://localhost:9200" \
-e TZ="GMT" \ # change to your local timezone if needed
-v /srv/cpuad-updater-logs:/app/logs \
satomic/cpuad-updaterIf your Elasticsearch instance requires authentication, pass include the ELASTICSEARCH_USER and ELASTICSEARCH_PASS environment variables.
-e ELASTICSEARCH_USER="elastic"
-e ELASTICSEARCH_PASS="mypassword"- Confirm that you are in the correct path
cd /srv/copilot-usage-advanced-dashboard - Install Dependencies
python3 -m pip install -r requirements.txt
- Setting Environment Variables. If you are using Copilot Standalone, use your Standalone Slug here, prefixed with
standalone:, for examplestandalone:YOUR_STANDALONE_SLUG.export GITHUB_PAT="<YOUR_GITHUB_PAT>" export ORGANIZATION_SLUGS="<YOUR_ORGANIZATION_SLUGS>"
- run
python3 main.py
- output logs
2024-12-17 05:32:22,292 - [INFO] - Data saved to logs/2024-12-17/nekoaru_level3-team1_copilot_usage_2024-12-17.json 2024-12-17 05:32:22,292 - [INFO] - Fetched Copilot usage for team: level3-team1 2024-12-17 05:32:22,293 - [INFO] - Data saved to logs/2024-12-17/nekoaru_all_teams_copilot_usage_2024-12-17.json 2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for organization: nekoaru 2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level1-team1 2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level1-team1 2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level2-team1 2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level2-team1 2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level2-team2 2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level2-team2 2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level3-team1 2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level3-team1 2024-12-17 05:32:22,293 - [INFO] - Sleeping for 6 hours before next execution... 2024-12-17 05:32:22,293 - [INFO] - Heartbeat: still running...
At this moment, in the VM, you should be able to see 3 containers running (if you have deployed them from scratch based on docker), as follows:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1edffd12a522 satomic/cpuad-updater:20241221 "python3 main.py" 23 hours ago Up 10 hours cpuad
b19e467d48f1 grafana/grafana:11.4.0 "/run.sh" 25 hours ago Up 10 hours grafana
ee35b2a340f1 docker.elastic.co/elasticsearch/elasticsearch:8.17.0 "/bin/tini -- /usr/l…" 3 days ago Up 10 hours 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp esAt this point, return to the Grafana page and refresh. You should be able to see the data.
or
-
Run the following command to set the GitHub credentials in the Azure Developer CLI.
azd env set GITHUB_PAT ... azd env set GITHUB_ORGANIZATION_SLUGS ...
-
Optional* Run the following commands to set the Grafana credentials. Note that not setting this values results in the deployment script generating credentials.
azd env set GRAFANA_USERNAME ... azd env set GRAFANA_PASSWORD ...
-
Run the following command to deploy the application.
azd up
-
After the deployment is complete, you can access the application using the URL provided in the output.
-
The username & password for the Grafana dashboard can be found in the Key Vault. Note that these are not secure credentials and should be changed.