The data issue of the Acceptance rate Average

[nickhou1983]

Dear Team.

We have found issues with the data.

You can see the screenshot, The acceptance rate average is 33.65%, In the line chart below, the acceptance rate in the past 28 days has not exceeded 30%.

[karpikpl]

@DevOps-zhuang - do you know what may cause this?

[karpikpl]

Hey @nickhou1983
the average is calculated using:

acceptanceRateAverage.value = cumulativeNumberAcceptances.value / cumulativeNumberSuggestions.value * 100;

where

• cumulativeNumberAcceptances is a sum of all total_acceptances_count - "The total number of Copilot code completion suggestions accepted by users."
• cumulativeNumberSuggestions is a sum of all total_suggestions_count - "The total number of Copilot code completion suggestions shown to users."

which in your case is: 92254/274183*100 = 33.65

Acceptance chart however is calculated by:

const rate = m.total_lines_suggested !== 0 ? (m.total_lines_accepted / m.total_lines_suggested) * 100 : 0;

where:

• total_lines_accepted - "The total number of lines of code completions accepted by users."
• total_lines_suggested - "The total number of lines of code completions suggested by Copilot."

based on https://docs.github.com/en/enterprise-cloud@latest/rest/copilot/copilot-usage?apiVersion=2022-11-28

I think we should add ℹ️ which explains each of the metrics displayed.

[hkelectric-cherylto]

I am also encountering an issue with the Abnormal Data:

Total Lines Accepted vs. Total Acceptances Count: It is illogical that the total lines accepted are fewer than the total acceptances count. However, what I actually observe is that the total lines accepted are always 0, and in some cases, they are less than the total acceptances count.

Example JSON:

JSON

{
"day": "2024-09-25",
"total_suggestions_count": 6,
"total_acceptances_count": 3,
"total_lines_suggested": 19,
"total_lines_accepted": 0,
"total_active_users": 2,
"total_chat_acceptances": 0,
"total_chat_turns": 0,
"total_active_chat_users": 0,
"breakdown": [
{
"language": "c#",
"editor": "visual_studio",
"suggestions_count": 6,
"acceptances_count": 3,
"lines_suggested": 19,
"lines_accepted": 0,
"active_users": 2
}
]
}

does anyone know what is the potential cause for the above issue?

[karpikpl]

I'll try to get this info, however this might get better chances of accepting in the GH product forums.

[gpaul-idexx]

I think "Acceptance Rate Average" is mislabeled. It's not the average of daily acceptance rates, it's the Acceptance Rate over the 28 day period. Therefore, labeling it "Acceptance Rate" is sufficiently accurate/clear (with the existing subtitle "Over the last 28 days")

[karpikpl]

@gpaul-idexx would this help?

[gpaul-idexx]

@karpikpl Personally, I think the content of the first graph should be changed to be based on acceptances, not lines (and leave the label of the chart as is). This gives the user an understanding of the relevant impact of copilot. The 2nd chart which is currently labeled Total Suggestions Count | Total Acceptances Count (whose content should stay the same IMO) would give the user understanding of the absolute impact.

The 3rd chart gives you absolutes about lines.

IMO the content of the current first chart is not that interesting (% of lines accepted). I think number of lines stats make more sense in absolute numbers of lines, rather than ratios. I'm not sure what the rate of lines accepted means to me - that seems more like a way for Github folks to measure Copilot, and less of a way for me to measure the value I get from Copilot.

So, my suggestion is to change the label at the source of this ticket from "Acceptance Rate Average" -> "Acceptance Rate", and to change the content of the Acceptance Rate graph to be based on total_acceptances_count / total_suggestions_count instead of total_lines_accepted / total_lines_suggested

[karpikpl]

@DevOps-zhuang what do you think about including this in your PR?

[DevOps-zhuang]

@DevOps-zhuang what do you think about including this in your PR?

I will look into it, and it seems some issue are already fixed in my PR, plese check my latest test website by visiting https://copilot-viewer.azurewebsites.net/

and, there is another hidden bug, which introduces unexisting editor/language, I am still struggling with it, not sure it is data issue/logic issues.

[DevOps-zhuang]

@karpikpl Personally, I think the content of the first graph should be changed to be based on acceptances, not lines (and leave the label of the chart as is). This gives the user an understanding of the relevant impact of copilot. The 2nd chart which is currently labeled Total Suggestions Count | Total Acceptances Count (whose content should stay the same IMO) would give the user understanding of the absolute impact.

The 3rd chart gives you absolutes about lines.

IMO the content of the current first chart is not that interesting (% of lines accepted). I think number of lines stats make more sense in absolute numbers of lines, rather than ratios. I'm not sure what the rate of lines accepted means to me - that seems more like a way for Github folks to measure Copilot, and less of a way for me to measure the value I get from Copilot.

So, my suggestion is to change the label at the source of this ticket from "Acceptance Rate Average" -> "Acceptance Rate", and to change the content of the Acceptance Rate graph to be based on total_acceptances_count / total_suggestions_count instead of total_lines_accepted / total_lines_suggested

thanks for your suggestion, and I mainly follow your suggestion, made change as below. could you please check it?

for breakdown page, like the language breakdown and editors breakdown.