-
Notifications
You must be signed in to change notification settings - Fork 12
Copilot Adoption Tool
This page is the specification for the Copilot Adoption area of the admin site (/copilot-adoption). It is not a how-to — it describes what the tool is for, what every figure on it means, and the rules any new feature has to satisfy.
Looking for how to use it? See the Copilot Adoption User Guide — walkthroughs, worked scenarios, a glossary, and what each warning means.
Use this page whenever the tool is changed. Every pull request that adds or alters a figure, chart, list or export in this area should be reviewed against the feature review checklist at the bottom, and this page should be updated in the same change. If a proposed feature does not fit the purpose below, that is a finding worth raising rather than a reason to quietly widen the scope.
The tool answers two spending questions and nothing else:
- Who is paying for a Microsoft 365 Copilot licence they are not using?
- Who would benefit from a licence they do not have?
Everything on the page exists to support one of those two questions, or to justify the numbers that answer them.
The reason the tool exists at all is that Microsoft's own Copilot usage reporting cannot answer either question well:
| Gap in Microsoft's reporting | What this tool does instead |
|---|---|
| "Active user" is a yes/no flag — one interaction in 28 days counts the same as fifty | Grades engagement 0–100 from frequency, depth and breadth, and bands it |
| Covers licensed users only | Also counts unlicensed Copilot Chat use, which is the strongest evidence of unmet demand there is |
| Unusable when the tenant conceals user information | Audit-derived figures are unaffected; the tool says which source each figure came from |
| No organisational metadata to act on | Joins department, job title, country, manager and account status, and exports the lot to CSV |
| Cannot tell "never started" from "tried it and stopped" | Splits those into Never used and Dormant, because they need opposite responses |
Two readers, in this order:
- The person who owns the Copilot budget — needs the headline figures, the funnel and the department breakdown, and needs to be able to defend every one of them in a renewal conversation.
- The person who runs enablement — needs the per-user lists, the recommended action for each person, and the CSV to hand to a department lead.
- Not a general Copilot usage report. Prompt content, sentiment, key phrases, agents and credit estimates belong in the Reports → Copilot area, not here. If a proposed chart does not change a licence decision or an enablement decision, it does not belong on this page.
- Not a per-user surveillance tool. Every figure is there to decide whether a licence is well placed, not to rank individuals against each other. There is deliberately no "top users" leaderboard.
- Not a replacement for Microsoft's compliance/audit reporting.
These are the rules that make the tool trustworthy. They are the ones most worth checking a new feature against.
Any figure that makes a claim carries an information ("i") button giving:
- What it claims, in one sentence;
- How it is worked out, in words;
- The formula, shown verbatim so it can be checked rather than trusted;
- The source import, and any caveat that comes with it.
This is not decoration. Every number here ends up in a licence negotiation or a conversation with a department lead, and the first two questions asked about all of them are "what exactly does that mean?" and "how do you know?". A methodology tab alone is not sufficient — by the time the reader has the question they are looking at the number, not at the tab.
Exception: figures that are self-defining need no tip. "Adoption by country" is a count of users by their country attribute; there is no hidden judgement in it. "Habitual users" is the opposite — it is a threshold on a weighted score, and is meaningless without its definition.
Rule of thumb: if the tool applied a threshold, a weight, a normalisation or a classification, it must explain itself.
Where an explanation is the same for a whole group of rows, it is stated once for the group, not repeated on every row. A "recommended action" column that prints the identical paragraph for every user in a band is a hundred copies of one sentence dressed up as a hundred findings.
The useful form of that information is the size of each job — "coach 76 people" is a plan. The per-user list carries a short tag; the meaning sits above the table and on the Enablement plan card.
The CSV is the exception: it keeps the full sentence on every row, because a spreadsheet gets sorted, filtered and split up, and cannot rely on a legend being nearby.
0% across six licences and 0% across six hundred are the same percentage and completely different decisions. Segment tables always show licences, active, habitual and never-used alongside the rate.
- Cowork adoption is only shown when Cowork activity was actually detected — on a tenant that never enabled it, "0%" reads as a failure rather than "not applicable here".
- Segments below the minimum licence count are omitted rather than shown as noise.
- Every missing or degraded import produces an explicit warning on the result, not a silently wrong number.
- The data sources section names which import supplied each figure and the snapshot date it came from.
Each chart backed by a query exposes its SQL through the </> SQL popover. These numbers get quoted in licence negotiations, so an admin has to be able to verify them independently.
Filtering, sorting and paging happen server-side against a single cached analysis, and the CSV export takes the identical parameters. A CSV that quietly disagrees with the chart above it is worse than no CSV.
"Exactly what is on screen" means data fidelity, not layout isomorphism. The Excel workbook is not a screenshot of the page: it re-lays the same numbers into sheets suited to a spreadsheet, and adds native Excel charts so the recipient can pivot and re-chart them. What it must never do is contain a number the page would not produce for the same filters and period — which is why, for example, the workbook's band percentages divide by the same analysed denominator the on-screen donut uses.
Every weight and threshold lives in CopilotAdoptionOptions, is unit-testable, and is returned to the client with the results so a figure can always be traced back to the rule that produced it. The UI must quote the options it was given, never hard-code a duplicate of them.
A regression to watch for: because the options object is serialised, every property on it needs its
[JsonProperty]camelCase name. Without it the whole methodology tab rendersNaN%and blanks — which is exactly what happened before this was fixed.
This page's whole audience is people in meetings: it gets projected, printed, and pasted into decks in greyscale. Around one man in twelve cannot reliably separate the red/green ends of the band palette. So every chart that uses colour to carry meaning must carry it a second way as well — a letter, a number, a label, or a separating hairline.
Applied: the intensity bubble chart draws the band initial (C/E/D/T) inside each bubble and repeats the pairing in a legend; the concentration bar numbers its cohorts and separates adjacent slices with a hairline; the band donut and every table label their slices in text.
Any aggregate that names a population ("76 people need a first habit") must be clickable through to that exact population, with the same members. A number the reader has to reconstruct by hand from filters on another tab is an invitation to reconstruct it wrongly — and then to quote the wrong one.
The drill-through must filter on the same key the aggregate was grouped by. The enablement plan groups by action code, so its drill-through filters by action code; filtering by band instead would look right and land on a different set of people.
Zero data is a normal state on a freshly installed tenant, not an error. A page that answers it with a grid of "no data" cards reads as broken software, tells the reader nothing about which of several quite different causes applies, and gives them nothing to do next. Every screen that can legitimately have nothing to show needs a written answer to "why is this empty, and what do I do about it?"
Each licensed user gets a graded score, because "did they use Copilot?" is almost never a yes/no question — someone who opened it twice and someone who lives in it produce the same "active user" count and need opposite responses.
| Component | Default weight | Measures | Default target |
|---|---|---|---|
| Frequency | 50% | Distinct days used | 60% of available working days (5-day week) |
| Depth | 30% | Interactions per active day | 5 per day |
| Breadth | 20% | Distinct Copilot surfaces used | 3 |
frequency = min(1, activeDays / expectedActiveDays)
depth = min(1, (interactions / activeDays) / depthTarget)
breadth = min(1, appsUsed / breadthTarget)
score = (frequency x 0.5 + depth x 0.3 + breadth x 0.2) / (0.5 + 0.3 + 0.2) x 100
Design decisions worth preserving:
- Working days, not calendar days. Against calendar days, someone using Copilot every single working day caps out at ~71% and looks like a partial adopter.
- Depth is per active day. Otherwise an intermittent-but-intensive user is penalised twice for the same low frequency.
- Every component is capped at 1 before weighting. Nothing above target buys extra credit, and no single component can carry a user on its own.
- The weighted sum is divided by the total of the weights. That is what keeps the result on a 0–100 scale whatever the weights are tuned to — and it is why the UI must quote each weight's share of the total rather than the raw weight. Quoting the raw weight is only correct while they happen to add up to 1.
-
"Last N days" means exactly N distinct calendar dates, ending today. The window starts at midnight N−1 days ago, not N — starting N days back spans N+1 dates, so a user active every single day would record 29 active days in a "28-day" window while
expectedActiveDaysis derived from 28. The numerator and the denominator have to measure the same window, or the score is inflated for precisely the most engaged users and the dates shown on screen are a day wider than their own label. One definition,CopilotAdoptionScoring.WindowStartUtc, sits besideTargetActiveDaysso the two cannot drift apart.
| Band | Rule | Meaning |
|---|---|---|
| Champion | score ≥ 75 | Among the deepest users |
| Established | score ≥ 50 | Habit formed — this is the "habitual users" line |
| Developing | score ≥ 25 | Habit forming |
| Trialling | score < 25, some activity | Occasional use only |
| Dormant | no activity this period, but some in the last 365 days | Tried it and stopped |
| Never used | no activity at all in the history window | Never started |
Users with no activity in the period are never scored, because a zero score would put two completely different problems in the same bucket. Dormant + Never used = reclaimable licences.
The unweighted counterpart to the bands, and the number a sceptical reader trusts first: how many days a month does this person actually open Copilot?
Deliberately not labelled "habit" on screen. "Habitual users" in the headline is the weighted engagement score — frequency plus depth plus breadth — whereas this card is frequency and nothing else. Using one word for both would put two different methodologies behind the same label, which is exactly the ambiguity the tool exists to remove. The two are meant to be read against each other: a large "Daily" tile with a low habit rate means people open Copilot constantly and do very little with it.
| Bucket | Default range |
|---|---|
| Infrequent | 1–5 active days a month |
| Moderate | 6–10 |
| Frequent | 11–19 |
| Daily | 20+ |
Because the reporting period is adjustable, active days are normalised — and rounded to whole days — before bucketing:
daysPerMonth = round(activeDays x 28 / windowDays)
Without the normalisation, "11+ active days" would mean a near-daily user over a 28-day period and a once-a-fortnight user over a 180-day one — the same tile silently changing meaning when the reader changed the period drop-down. The rounding is what makes the printed captions ("1–5 active days a month") describe the comparison exactly rather than approximately. Any activity at all rounds up to at least one day, so a single interaction in a 180-day window is Infrequent rather than unbucketed.
Habit percentages are a share of active users, not of all licences: someone who never opened Copilot is not an "infrequent" user, they are a reclaimable licence, and merging the two hides the more expensive problem.
Every licensed user gets exactly one action, so the counts sum to the whole licensed population.
Each label names the step to take, not the state the user is in. Earlier drafts used "Coach" and "Grow", which read as synonyms unless the reader had memorised the band thresholds — two rows needing genuinely different interventions looked like the same instruction. CopilotAdoptionTests.EveryActionLabel_IsDistinctAndNamesTheStepToTake pins this: labels must be unique, multi-word, and never a substring of one another.
| Action | Applies to | Meaning |
|---|---|---|
| Reclaim or onboard | Never used | Licence has produced nothing; confirm it is needed before renewal |
| Win back | Dormant | Ask what stopped; refresh or reassign |
| Build a first habit | Trialling | One repeatable habit in the app they already live in |
| Add a second app | Developing/Established with low breadth | Habit confined to one surface — cheapest remaining gain |
| Deepen to daily use | Developing with adequate breadth | Scenario-based session aimed at their actual job |
| No action needed | Established | Paying for itself |
| Recruit as advocate | Champion | Run a peer session for their department |
Every row of the enablement plan is a drill-through. Clicking one opens the Licensed users tab pre-filtered to exactly the people that number counted, and its CSV export contains exactly that list. The filter is on the action code, not on the band, because an action such as "Add a second app" spans two bands — filtering by band would land the reader on a different set of people from the number they just clicked, which is worse than having no link at all.
Unlicensed users are scored 0–100 so that evidence beats inference:
| Signal | Default weight | Default target | Why |
|---|---|---|---|
| Already using Copilot Chat with no licence | 35 | 20 interactions | The only signal that proves demand for Copilot itself |
| Teams collaboration | 25 | 60 messages + meetings | Heavy collaborator |
| Email volume | 20 | 80 sent + read | Heavy correspondent |
| Document work | 20 | 40 files viewed/edited | Heavy document worker |
score = copilot x 35 + collab x 25 + email x 20 + documents x 20 (each component capped at 1)
recommended when score >= 50
The cap matters: without it a single extremely noisy mailbox would clear the threshold on email alone. Disabled accounts are excluded from candidates — but kept in the licensed-user list, because a disabled account still holding a Copilot licence is the clearest reclaim there is.
| Source | Covers | Caveats the UI must state |
|---|---|---|
Copilot audit log (copilot_chats, from Audit.General) |
Every user, licensed or not; per-app; matches the selected period exactly | Needs the Copilot audit import. Activity API events arrive 2–3 days in arrears |
| Microsoft Copilot usage report | Licensed users only | Unusable when the tenant conceals user information. Its window is Microsoft's, not the one selected |
| Microsoft 365 usage reports (Teams / Outlook / SharePoint / OneDrive) | Workload activity, used to find heavy users without a licence | A dated snapshot, not a period total. Published a few days in arrears |
| User metadata (licences, department, manager, account status) | The licensed population and all segmentation | Without it there is no denominator and no segmentation |
Licence classification is listed in full on the page, including the SKUs that were excluded, because Microsoft ships Copilot-branded SKUs that are not a Microsoft 365 Copilot licence (Copilot Studio, Copilot for Sales) and ships new licence SKUs regularly. The licensed population must be checkable rather than taken on trust.
The Overview is ordered by the questions a budget owner asks, in the order they ask them, and split into four numbered sections so a long page still reads as an argument rather than a wall of cards. An earlier draft ran seventeen cards in the order they happened to be built, which buried "adoption by department" — the who do I talk to card — near the bottom.
1. Where you stand — the headline position.
| Element | Answers |
|---|---|
| Headline figures | Licences, adoption rate, habitual users, reclaimable licences, average/median engagement, unlicensed use, recommended candidates |
| Gauges | Adoption rate and habit rate against the judgement scale, side by side — the gap between them is the finding |
| Adoption funnel | Where in licensed → ever used → active → habitual → champion the biggest drop is |
The last funnel step is not a loss. Each stage prints the count that did not reach it from the stage immediately above. For the middle steps that is a genuine drop-off and is shown in red. For the final step it is not: everyone who reaches Habitual has, by this tool's own definition, formed a habit — §3.2 calls that "habit formed" and the action catalogue marks it "no action needed / paying for itself". Champion is a top tier, not a target for the whole population, so flagging those users in red as "lost" would contradict the tool's own advice and point enablement budget at the people already succeeding. That step therefore reads "N not yet Champions" in neutral grey and should be read as the size of the advocate pool. Every figure on the right of the funnel carries a hover explanation.
2. What to do next — the work this creates.
| Element | Answers |
|---|---|
| Enablement plan | How many people need each action — the size of each job. Every row drills through to the people in it |
| Adoption by department | The running order for an enablement plan |
| Where the unmet demand is | Departments that could be rebalanced rather than bought for |
3. How Copilot is being used — the evidence behind those recommendations.
| Element | Answers |
|---|---|
| How often people open Copilot | Unweighted frequency: days a month, no scoring |
| Engagement mix donut | The whole licensed population in six mutually exclusive bands |
| Where Copilot is used treemap | Which surfaces carry the traffic, and which are unused |
| The shape of adoption | Where the typical user differs from the best ones, on frequency/depth/breadth |
| Usage frequency and intensity bubble chart | Which departments are frequent-but-shallow vs deep-but-occasional |
| Usage concentration | Whether adoption is broad or carried by a top decile |
| Who is doing the Copilot work | Licensed against unlicensed share of total volume |
| What Copilot is working on | Which kinds of tenant content Copilot grounded its answers in |
4. Trend and wider reach — direction of travel.
| Element | Answers |
|---|---|
| Weekly active users trend | Whether an enablement programme is working, across all three populations |
| Weekly Copilot volume trend | Interaction volume, licensed against unlicensed — headcount can flatten while volume climbs |
| Licensed and unlicensed side by side | Departments with idle licences and heavy unlicensed use — a reallocation, not a purchase |
| Adoption by country | The same measures for organisations that run enablement regionally |
First-run state. When the analysis completes and finds zero Copilot licences, the Overview is replaced by a designed empty state rather than seventeen cards each saying "no data" — which reads as a broken page rather than an unfinished import. It names the three usual causes in the order they are worth checking (licence import not run yet; a Copilot SKU this build does not recognise; the tenant genuinely holds no licences) and, when unlicensed Copilot activity was found, says so, because that combination almost always means the licence import is incomplete.
Per-user list with server-side search, band / recommended-action / department filters, "Cowork users only", "Disabled accounts only", sorting and CSV export. Carries a short action tag per row with the meanings stated once above the table. The action filter is what the enablement-plan drill-through sets, and it is a normal, visible, reversible filter — arriving here from a click never leaves the reader on a filtered list with no way to see how they got there.
Unlicensed Copilot Chat as a population in its own right — headline users/interactions/habitual count, its own habit strip using identical rules to the licensed one, usage by app, and usage by department. This is the one Copilot population Microsoft's own reporting cannot see at all.
The Copilot agent estate: active/known/custom counts, most-used and most-versatile agent, an inventory-health donut, a treemap of interactions per agent, agent usage by department, and the full inventory table with a Keep / Review / Retire / New verdict and a stated reason for each agent.
Ranked unlicensed candidates with their four component scores and a per-user justification safe to paste into a licence request. Unlike the licensed list, the justification genuinely differs per row — no two candidates reach the same score by the same route — so it stays a per-row column.
The full methodology: engagement score with a worked example, bands and habit buckets, action selection, the opportunity score, agent verdicts and usage concentration, data sources with their snapshot dates, and the complete SKU classification table.
A Excel report button in the page header exports the whole report as a single .xlsx: twelve sheets covering every figure, table and chart on the page, with native, live Excel charts bound to the cells rather than pasted-in pictures.
This is not "download the data" — the CSV exports already do that per list. It exists for a specific job: comparing before and after an enablement programme. A screenshot of a dashboard cannot be compared with another screenshot six months later — the numbers cannot be subtracted, and nobody can tell what period or thresholds either was run with. The workbook records both.
| Sheet | Contents |
|---|---|
| Report | Run metadata, period covered, which imports supplied data, every threshold used, and any warnings |
| Headline figures | The population figures and rates, with a chart |
| Adoption funnel | Each stage with its conversion from the stage above |
| Engagement | Band mix (doughnut), habit buckets (column), score profile, usage concentration (pie) |
| Weekly trend | Six months of weekly data, with an active-users line chart and a stacked-area volume chart |
| Departments and apps | Adoption by department and country, frequency/intensity, licensed vs unlicensed, usage by app, resource types |
| Agents | Estate figures, health doughnut, and the full inventory with verdict and reason |
| Unlicensed usage | Headline figures, habit buckets, usage by app and department |
| Enablement plan | Each action with how many people need it |
| Licensed users | Per-user list, auto-filtered and frozen |
| Licence opportunities | Ranked candidates with their justifications |
| How this is calculated | Every formula written out, plus the SKU classification table |
Design decisions worth preserving:
- The Report sheet records the thresholds, not just the numbers. Two snapshots are only comparable if both were scored by the same rules — the tuning is adjustable, so without this "adoption went up" could turn out to mean "the bar moved". The workbook's methodology sheet says so explicitly.
- Charts are native and bound to ranges, so a reader can re-plot, re-theme, or copy them into a board deck. Pictures would make the file a dead end.
-
Written with zero new dependencies. The
.xlsxis produced byCommon.Entities.Xlsx, a small hand-rolled OpenXML writer overSystem.IO.Compression. EPPlus 5+ is commercially licensed, and any third-party library would have meant binding-redirect changes across several projects — see the NuGet rules in the engineering notes. - Built from the same cached analysis that renders the page, so the workbook can never quietly disagree with the screen it was downloaded from.
Because the writer is hand-rolled, CopilotAdoptionWorkbookTests generates a real workbook from synthetic data on every build and asserts that it is a valid OPC package, that every part is well-formed XML, that no relationship is broken, that the charts reference cell ranges, that Greek text and ampersands survive, and that numbers are written invariantly even under a de-DE culture. A malformed part reaches a customer as "Excel found unreadable content" with no indication of the cause, so this is not optional coverage.
microsoft/AI-in-One-Dashboard is a Power BI template (.pbit) over pre-processed rollup files produced from Purview audit + Entra/MAC exports by PAX (Portable Audit eXporter). It is the closest comparable tool, and several ideas here came from it. This table records what we cover, so a future change cannot silently regress a capability.
Worth knowing when comparing: Microsoft label it an experimental template and state that Purview audit logs are "not intended to serve as the sole source of truth for licensing or full-fidelity reporting". Our licensed population comes from the imported licence assignments rather than inferred from audit activity, and the SKU classification is listed on the page.
| AI-in-One page / capability | Here |
|---|---|
| M365 Copilot — usage trends (active users, actions, actions per user) | ✅ Weekly active users + weekly volume trend |
| M365 Copilot — leaderboard by organisation | ✅ Adoption by department + combined leaderboard |
| M365 Copilot — habit formation (Infrequent/Moderate/Frequent/Daily + donut) | ✅ Habit strip + engagement-mix donut |
| Copilot adoption thresholds / usage-rank cohorts | ✅ Usage concentration (percentile cohorts) |
| Unlicensed Copilot Chat — usage trends | ✅ Unlicensed series on both trend charts |
| Unlicensed Copilot Chat — leaderboard | ✅ Unlicensed usage by department + combined leaderboard |
| Unlicensed Copilot Chat — habit formation | ✅ Unlicensed habit strip, identical rules to licensed |
| Unlicensed Copilot Chat — usage by application | ✅ Unlicensed usage-by-app treemap |
| Copilot (Unlicensed) — licence prioritisation matrix | ✅ Exceeded — a per-user scored candidate list with a written justification, rather than a department quadrant |
| Copilot overall — combined trends (three populations) | ✅ Licensed / unlicensed / agent-user series on one chart |
| Copilot overall — combined leaderboard (licensed vs unlicensed, % using agents) | ✅ Licensed-and-unlicensed side-by-side table |
| Usage frequency and intensity scatter | ✅ Frequency-vs-intensity bubble chart |
| Top resource types by user activity | ✅ "What Copilot is working on" |
| Agents — usage trends | ✅ Agent-user trend series + agent usage by department |
| Agents — leaderboard (users, actions, actions per user, type) | ✅ Agent inventory table |
| Agents — health check (Keep / Review / Retire / New) | ✅ With a stated reason per agent |
| Agents — most popular / most versatile | ✅ |
| Agents — habit formation | 🟡 Partial — agent users are counted, not bucketed separately |
| Agents — creator, description, Agent 365 metadata | ❌ Not in our Copilot audit import |
| Agents — AI-generated "use cases" narrative | ❌ Deliberately not copied |
| Per-user leaderboard naming individual heavy users | ❌ Deliberately not copied |
| Export to Excel with charts | ✅ Not in AI-in-One — it is Power BI, so its equivalent is a .pbix needing Power BI Desktop and the PAX pipeline |
Visual vocabulary. AI-in-One is a Power BI report and gets its chart library free. This tool draws everything as dependency-free SVG, so each visual is a deliberate choice rather than a drag-and-drop. The set is: arc gauges (headline rates), a tapering funnel (not bars), doughnuts (engagement mix, agent health, habit split), a squarified treemap (usage by app and by agent), a radar (the frequency/depth/breadth profile, tenant vs its own Champions), a bubble scatter with median quadrant lines (frequency vs intensity), a stacked area (population composition over time), lines (weekly trends), gradient bars (categorical breakdowns), conditional-shaded matrices (leaderboards), and stacked share bars (usage concentration).
Two of those have no AI-in-One equivalent at all — the radar profile and the arc gauges — and the funnel, the concentration bar and the quadrant lines are ours as well.
What we deliberately do not copy
- The AI-generated narrative page. It needs an LLM call over tenant data and produces prose that reads authoritatively while being unverifiable — the exact opposite of design principle 2.1. Every assertion here has a stated formula behind it.
- A per-user "top users" leaderboard. Ranking named individuals against each other is a non-goal (§1). Every per-user list here exists to decide whether a licence is well placed, which is why the default sort is least-engaged-first.
Where this tool goes further: the graded engagement score and bands (AI-in-One uses raw active-day buckets only), the adoption funnel, per-user recommended actions and the aggregated enablement plan, per-user licence justifications, the SKU classification table, SQL popovers on every query, the methodology tab, and CSV export of both user lists.
Run through this for every change to the Copilot Adoption area.
Purpose
- Does this change help answer "who is wasting a licence" or "who needs a licence"? If not, does it belong in Reports → Copilot instead?
- Does it change a decision, or is it interesting-but-inert?
Explainability
- Does every new figure carry an "i" tip with what, how, the formula, and the source?
- Does the tip quote the actual options returned by the API rather than hard-coded numbers?
- Is any new threshold, weight or normalisation added to
CopilotAdoptionOptions— with a[JsonProperty]camelCase name — and to the methodology tab and this page? - Is any explanation that is identical across rows stated once rather than per row?
Honesty
- Are percentages shown with their absolute counts?
- Does the figure degrade to an explicit warning (not a silent zero) when its import is missing?
- Is a metric hidden entirely rather than shown as "0%" when the feature is not in use on the tenant?
- Does a normalised figure keep the same meaning across all period options (7 / 28 / 90 / 180 days)?
Consistency
- Is the scoring logic in
CopilotAdoptionScoring(one implementation, unit-tested), not duplicated in SQL or TypeScript? - Do the CSV export, the on-screen list and the summary all come from the same cached analysis?
- Do band and action colours match the shared palettes, so two views never appear to disagree?
- Does an aggregate average the same population its other dimensions describe? (The frequency/intensity bubble colours by the active users' mean, because both its axes are active-only; the department table uses the whole-population mean, because it is reporting on all the licences.)
- Does a chart's centre total, or a "these counts sum to…" claim, actually equal the sum of what is drawn? Aggregates built from the scored rows are capped by
MaxLicensedUsersScoredand are not necessarily the licence count.
Charts
- Does a bubble/area encoding scale by area, with no constant added to the radius? Adding a fixed minimum radius silently inflates the smallest categories.
- Is there axis headroom so the largest marker and its label are not clipped by the SVG viewBox?
- Does a treemap lay its rows along the shorter side of the remaining space? Using the longer side still fills the area exactly but produces unreadable slivers.
- Is the chart type chosen for the question, not for novelty? Doughnuts and stacked areas are for composition; bars and lines are for comparison; a radar is for shape. A radar with many axes, or with axes on different units, looks impressive and says nothing.
- Does the chart survive greyscale and colour-blindness — is every meaning carried by something other than hue as well? (Principle 2.8.)
- If a visual is added to the page, is it also added to the Excel export — and vice versa? The two are meant to carry the same report.
Navigation and empty states
- Does every aggregate that names a population drill through to exactly that population, filtered on the same key it was grouped by? (Principle 2.9.)
- Is the resulting filter visible and reversible, rather than a hidden state the reader cannot see or clear?
- Does this screen have a designed empty state that says why it is empty and what to do next? (Principle 2.10.)
- Are two different measures ever given the same name? "Habit rate" (weighted score) and "how often people open Copilot" (raw days) were one word apart from being indistinguishable.
- Does each action/label name the step to take rather than the state the reader is in, and is it distinct from every other label? Pinned by
EveryActionLabel_IsDistinctAndNamesTheStepToTake.
Excel export
- Does any new figure appear on the correct sheet, with its explanation?
- Do the workbook tests still pass? They generate a real file and validate the package, and they are the only thing standing between a malformed part and a customer seeing "Excel found unreadable content".
- Are new thresholds added to the Report sheet? A snapshot that does not record the rules it was scored by cannot honestly be compared with another one.
Scale — assume a tenant of ~200,000 users
- Any new query batched rather than per-user, and SARGable (no
LOWER()on an indexed column)? - Any new aggregation done in the existing in-memory pass rather than as another database round trip?
- If a new index is proposed, is there a measured before/after benchmark? (See the engineering notes on proving schema changes.)
Documentation
- Is this page updated in the same pull request?
- Do new user-visible behaviours get a plain-English release-note line?
Assessed against Microsoft's own 2026 tooling (the M365 admin-centre Copilot dashboard, the Copilot Control System measurement pillar, Viva Insights' Business Impact Report, and Microsoft's free Analytics Hub template library) and the commercial tools in this space.
The strategic shift to be aware of: "who is using Copilot" is now largely a solved, free, native capability. The frontier has moved to three things — credit/consumption cost management, defensible business value, and readiness grounded in real data exposure. Microsoft ships free Power BI templates for all three (ValueLens, Consumption Central, Cowork Credit Chargeback), and competitors compete on idle-licence reclamation in currency, oversharing readiness, and cross-vendor "shadow AI" spend.
The good news: most of the raw signal is already imported and simply not surfaced.
| # | Feature | Why it matters | Data |
|---|---|---|---|
| 1 | Copilot Credits cost analytics and chargeback | Cowork and agents put Copilot on a metered bill. "What are we spending, on whom" is the budget owner's first question. Roll up to licence/department/manager, add a showback view and a forecast against a prepaid pack |
copilot_chats.copilot_credit_estimate_total and the _json breakdown are already stored per interaction |
| 2 | Cost per active user | Licence price plus credits divided by active or habitual users — what each productive user actually costs once idle licences are amortised | Above + a per-licence cost input |
| 3 | Reclaimable spend in currency | Turns the reclaim count into the number a CFO acts on. Four buckets: purchased-but-unassigned, assigned-never-used, dormant, active | Needs Graph subscribedSkus for purchased-vs-assigned |
| 4 | Model mix / deep-reasoning share | Premium models carry far more credit weight. Flags expensive usage patterns |
copilot_ai_models already detects deep-reasoning models |
| 5 | Grounding rate | Share of interactions grounded in tenant content vs generic chat. Low grounding means a licence is buying what free Copilot Chat already gives — a downgrade recommendation the tool cannot currently make |
copilot_event_files, copilot_event_accessed_resources
|
| 9 | Data-exposure signal | Which interactions actually touched unlabelled or highly-sensitive content. Defensible precisely because it is what Copilot did touch, not a hypothetical scan |
CopilotEventAccessedResource.SensitivityLabelId is already stored |
- Actual credit consumption imported to reconcile against our estimate.
- Deterministic "hours saved" — an admin-configurable, action-weighted model (meeting summary = meeting length, drafting = n minutes) where every number traces to a count times a shown rate. Deliberately formula-based, which is a selling point against black-box narrative tooling.
- Before/after adopters vs matched non-adopters — a difference-in-differences view of workload behaviour, like Viva's Business Impact Report but from data we already hold. Needs licence-assignment dates.
- Time-to-first-value and time-to-habit by onboarding cohort.
- Agent depth — per-agent credit consumption and adoption curve on top of the existing Keep/Review/Retire inventory.
-
Shadow / third-party AI usage — the "paying twice for AI" signal. Our importer currently handles
CopilotInteractiononly;ConnectedAIAppInteraction/AIAppInteractionare separate ingestible operations. - Structured survey / NPS ingestion for sentiment — structured input only, never generative text (see below).
- AI-generated narrative summaries. Microsoft is adding a natural-language analyst agent. This breaches design principle 2.1 outright: prose that reads authoritatively and cannot be verified. Our template-based per-user justifications are deterministic and traceable, which is the better answer, not a lesser one.
- A named-user fluency leaderboard. Microsoft's Readiness Report ranks individuals to find "natural champions". Ranking named individuals against each other is a stated non-goal (§1). A department- or cohort-level wave-planning version is fine; a per-person ranking is not.
- Generic usage reporting. Prompts, trends and per-app breakdowns for their own sake belong in Reports → Copilot. Anything here must be framed around licence spend, cost drivers or value.
- Pre-aggregated rollups. Re-scoped after measurement — see "Measured at scale" below. The analysis derives everything from repeated joins over the raw Copilot facts. Rollups maintained at import time remain the answer for the widest windows and are the prerequisite for the credit analytics above being affordable, but they are no longer the first fix: the dominant cost turned out to be how the aggregate was written, not how much data it read, and that has been fixed without any schema change. This is still the largest piece of engineering on this list and needs migrations plus the before/after benchmarks the schema-change policy requires.
- Streaming the workbook rather than buffering it, and a per-worker concurrency limit on export.
- Caching only compact summary data and paging detail from a persisted analysis, rather than holding the whole scored object graph per window.
Benchmarked on a synthetic 200,000-user tenant — 60,000 Copilot licences, 34,000 of them with activity, 12,000,000 interactions over 365 days — against the real LicensedUsersSql, with SET STATISTICS IO/TIME, medians of three runs discarding the cold run. Never against a customer database.
The finding was not what the earlier audit assumed. The fact scan was cheap; the cost was asking for three COUNT(DISTINCT ...) in one GROUP BY. SQL Server streams a single distinct aggregate cheaply, but two or more force it to fan the input out through a spool and process each distinct separately — and that spool was most of the query. It also pushed the query past the 90-second command timeout, so on a large tenant the report degraded to a warning rather than merely being slow.
| Window | Logical reads | of which worktable | Elapsed | |
|---|---|---|---|---|
| 28 days (default) | before | 114,750,742 | 25,413,875 | 281 s |
| after | 772,272 | 232 | 73 s | |
| 365 days (widest) | before | 115,172,403 | 25,835,536 | 303 s |
| after | 818,851 | 792 | 135 s |
Both metrics improve at both selectivities and the spool disappears. Isolation runs confirmed the cause precisely: removing one of the three distinct counts changed almost nothing (36.6M reads), removing all three collapsed the query to 92k — and narrowing app_host from nvarchar(MAX), the intuitive suspect, was worth about 1%.
Correctness was gated before performance: old and new aggregates were run side by side over the same window and compared row for row — 33,999 rows each, zero differing. That comparison exists because the real risk is NULL handling: COUNT(DISTINCT CASE WHEN ... END) ignores NULLs implicitly, whereas the rewrite must exclude them explicitly.
LicensedUsersQuery_NeverAsksForSeveralDistinctCountsInOneGrouping pins the shape, because re-introducing it is an innocent-looking edit whose damage only appears on a tenant nobody tests against.
Still to measure: AgentUsageSql, UnlicensedUsageRowsSql and WeeklyAdoptionTrendSql use the same multiple-distinct shape. They group far more coarsely — per agent, per week — so the blow-up should be smaller, but they have not been measured and the same technique applies. The 365-day window, while no longer failing, is still slow at this scale and is the case rollups would address.
The tool can only be demonstrated — or reviewed — on a population that actually varies. Scattering N interactions randomly across N users, which is what volume/importer testing wants, gives every licensed user near-identical usage: the whole tenant lands in one band, and the funnel, the treemap, the profile radar and every "who should I target" panel correctly show nothing. That looks like a broken tool.
Tests.FakeDataGen therefore has an adoption scenario mode, offered (default yes) by Generate combined profiling data (O365 + Copilot):
Shape Copilot usage into adoption personas (all funnel stages, contrasting departments)? [Y/n]
It plants:
- Every funnel stage — Champion, Established, Developing, Trialling, Dormant and Never used.
- Several different shapes at similar scores — "frequent but shallow" (63), "deep but narrow" (58) and "broad but occasional" (49) all sit close together, so the profile radar and the per-component breakdown have something real to distinguish. A tool reporting one number would call these the same user; they need opposite interventions.
- Contrasting departments — each department is assigned a Leading / Progressing / Lagging maturity, so the department charts show variation rather than a flat line.
- Reclaimable licences — including licences still assigned to disabled accounts.
- An unlicensed population with real Copilot usage, which is what the licence-opportunity ranking is built from.
- One agent per health verdict — Keep, Review, Retire and New. These need first/last-use dates up to ~115 days back, so the scenario deliberately reaches past the requested spread; a 90-day run could otherwise never produce a retirement candidate.
Persona targets are expressed only in the three signals the analysis actually measures — distinct active dates, interactions per active date, and distinct app_host values — so the bands are produced by the real scoring code rather than asserted by the generator. At startup the generator re-scores every persona through CopilotAdoptionScoring and warns if any no longer lands in the band it claims, which is what stops a change to a tuning default from silently turning the demo tenant into something that no longer demonstrates what this page describes.
Volume and performance testing should answer no to the persona prompt, or use Generate fake Copilot activity, which keeps the original random scatter.
The analysis is the most expensive thing the web application does, and its failure mode is quiet: a query that exceeds the 90-second command timeout degrades to a warning on the page rather than an error, so a tenant can sit with a half-populated report for months and never raise a ticket. Telemetry exists to make that visible to an operator instead of only to whoever happens to open the page.
Every completed analysis emits one CopilotAdoptionAnalysis custom event to Application Insights, with per-step durations as measurements (numbers), not dimensions (strings) — so they can be percentiled and charted without casting.
| Where | What |
|---|---|
customDimensions |
WindowDays, WarningCount, Outcome (Complete / Degraded), TimedOut, SlowestStep
|
customMeasurements |
TotalMs, plus <Step>Ms for every step — LicensedUsersMs, WeeklyTrendMs, AgentEstateMs, LicenceOpportunitiesMs, UnlicensedPopulationMs, UsageByAppMs, ResourceTypesMs, ScoringMs
|
One event per analysis, not per request. The event is emitted inside the cache's Lazy factory, so a cache hit — which does no work — does not appear as though it did. Otherwise the p95 would be dominated by cached reads and the slow runs, which are the whole point of measuring, would vanish into the average.
Step names come from CopilotAdoptionSteps constants so a saved query or alert keeps working across releases. Nothing in the event is derived from tenant data: every value is a duration, a count or a compile-time constant.
Is any tenant's report degrading? This is the important one — it finds reports that look fine but are not.
customEvents
| where name == "CopilotAdoptionAnalysis"
| where tostring(customDimensions.Outcome) == "Degraded"
or tostring(customDimensions.TimedOut) == "true"
| project timestamp,
window = customDimensions.WindowDays,
warnings = customDimensions.WarningCount,
slowest = customDimensions.SlowestStep,
totalMs = customMeasurements.TotalMs
| order by timestamp descHow slow is the analysis, and which step owns it?
customEvents
| where name == "CopilotAdoptionAnalysis"
| summarize p50 = percentile(todouble(customMeasurements.TotalMs), 50),
p95 = percentile(todouble(customMeasurements.TotalMs), 95),
runs = count()
by window = tostring(customDimensions.WindowDays)customEvents
| where name == "CopilotAdoptionAnalysis"
| summarize runs = count() by slowest = tostring(customDimensions.SlowestStep)
| order by runs descApproaching the timeout? Worth alerting on before it starts failing — a step at 70s on a growing tenant will be a warning next quarter.
customEvents
| where name == "CopilotAdoptionAnalysis"
| where todouble(customMeasurements.TotalMs) > 60000
| project timestamp, slowest = customDimensions.SlowestStep,
totalMs = customMeasurements.TotalMsSuggested alert: any event with TimedOut == "true", and a warning-level rule on p95(TotalMs) > 60000. Both are generic enough to sit alongside the existing HealthCheck and ImporterHeartbeat rules.
Workbook generation failures are tracked as App Insights exceptions (TrackException) plus an error trace, because a failure there is a defect in this application rather than anything the caller did. The endpoint deliberately returns a plain message instead of the exception text — the default Web API behaviour would put the stack trace in a response body reachable by every admin user.
Telemetry never changes request behaviour: every tracking call in this feature is wrapped so that a logging failure cannot turn a working request into a broken one, or replace a useful error with a confusing one.
- Copilot analytics — what Copilot data is collected and when
- Activity & Usage Analytics
- Analytics data model
- Configuration reference
- Home
- What data is collected
- The web portal
- Licence activity
- Copilot data & stats
- Architecture & costs
- App registrations setup
- Install with the installer
- Manual installation
- Private endpoints (optional)
- Certificate authentication (optional)
- Enable CSP for AITracker
- Verify the deployment
- Legacy SPO web setup