Copilot Adoption Design Notes

This page is for contributors and maintainers of Copilot Adoption. It keeps the engineering background that programme leads and administrators don't need: how the tool compares with Microsoft's AI-in-One dashboard, known gaps and ideas, the performance measurements behind its design, the demo-data generator, and API changes. Section numbers follow the specification.

On this page: 5a. Capability coverage · 7. Known gaps and possible next steps · Performance measurements · 8. Demo and test data · API changes

---

5a. Capability coverage vs the AI-in-One Dashboard

microsoft/AI-in-One-Dashboard is a Power BI template (.pbit) over pre-processed rollup files produced from Purview audit + Entra/MAC exports by PAX (Portable Audit eXporter). It is the closest comparable tool, and several ideas here came from it. This table records what we cover, so a future change cannot silently regress a capability.

Worth knowing when comparing: Microsoft label it an experimental template and state that Purview audit logs are "not intended to serve as the sole source of truth for licensing or full-fidelity reporting". Our licensed population comes from the imported licence assignments rather than inferred from audit activity, and the SKU classification is listed on the page.

AI-in-One page / capability	Here
M365 Copilot — usage trends (active users, actions, actions per user)	✅ Weekly active users + weekly volume trend
M365 Copilot — leaderboard by organisation	✅ Adoption by department + combined leaderboard
M365 Copilot — habit formation (Infrequent/Moderate/Frequent/Daily + donut)	✅ Habit strip + engagement-mix donut
Copilot adoption thresholds / usage-rank cohorts	✅ Usage concentration (percentile cohorts)
Unlicensed Copilot Chat — usage trends	✅ Unlicensed series on both trend charts
Unlicensed Copilot Chat — leaderboard	✅ Unlicensed usage by department + combined leaderboard
Unlicensed Copilot Chat — habit formation	✅ Unlicensed habit strip, identical rules to licensed
Unlicensed Copilot Chat — usage by application	✅ Unlicensed usage-by-app treemap
Copilot (Unlicensed) — licence prioritisation matrix	✅ Exceeded — a per-user scored candidate list with a written justification, rather than a department quadrant
Copilot overall — combined trends (three populations)	✅ Licensed / unlicensed / agent-user series on one chart
Copilot overall — combined leaderboard (licensed vs unlicensed, % using agents)	✅ Licensed-and-unlicensed side-by-side table
Usage frequency and intensity scatter	✅ Frequency-vs-intensity bubble chart
Top resource types by user activity	✅ "What Copilot is working on"
Agents — usage trends	✅ Agent-user trend series + agent usage by department
Agents — leaderboard (users, actions, actions per user, type)	✅ Agent inventory table
Agents — health check (Keep / Review / Retire / New)	✅ With a stated reason per agent
Agents — most popular / most versatile	✅
Agents — habit formation	🟡 Partial — agent users are counted, not bucketed separately
Agents — creator, description, Agent 365 metadata	❌ Not in our Copilot audit import
Agents — AI-generated "use cases" narrative	❌ Deliberately not copied
General named-user fluency leaderboard	❌ Not implemented; the targeted See-PII-gated agent-use ranking is a separate licence/advocate view
Export to Excel with charts	✅ Not in AI-in-One — it is Power BI, so its equivalent is a .pbix needing Power BI Desktop and the PAX pipeline

Visual vocabulary. AI-in-One is a Power BI report and gets its chart library free. This tool draws everything as dependency-free SVG, so each visual is a deliberate choice rather than a drag-and-drop. The set is: arc gauges (headline rates), a tapering funnel (not bars), doughnuts (engagement mix, agent health, habit split), a squarified treemap (usage by app and by agent), a radar (the frequency/depth/breadth profile, tenant vs its own Champions), a bubble scatter with median quadrant lines (frequency vs intensity), a stacked area (population composition over time), lines (weekly trends), gradient bars (categorical breakdowns), conditional-shaded matrices (leaderboards), and stacked share bars (usage concentration).

Two of those have no AI-in-One equivalent at all — the radar profile and the arc gauges — and the funnel, the concentration bar and the quadrant lines are ours as well.

What we deliberately do not copy

• The AI-generated narrative page. It needs an LLM call over tenant data and produces prose that reads authoritatively while being unverifiable — the exact opposite of design principle 2.1. Every assertion here has a stated formula behind it.
• A general employee/fluency leaderboard. Do not confuse that with the Agents tab's authorised named usage ranking; its purpose is licence/advocate review, not performance measurement.

Where this tool goes further: the graded engagement score and bands (AI-in-One uses raw active-day buckets only), the adoption funnel, per-user recommended actions and the aggregated enablement plan, per-user licence justifications, the SKU classification table, SQL popovers on every query, the methodology tab, and CSV export of both user lists.

---

7. Known gaps and possible next steps

Release scope, not issue-title scope. The behavioural Transformation Paradox view (#649) is deferred, not shipped: it needs a distinct-insight decision before a prototype. Optional prompt taxonomy does not implement #652's O*NET-specific mapping or establish model accuracy. The 33 governed settings are editable (#683); #684's operational activation window remains open, not an available control. Settings scope distinguishes editable targets from fixed windows/caps.

Assessed against Microsoft's own 2026 tooling (the M365 admin-centre Copilot dashboard, the Copilot Control System measurement pillar, Viva Insights' Business Impact Report, and Microsoft's free Analytics Hub template library) and the commercial tools in this space.

The strategic shift to be aware of: "who is using Copilot" is now largely a solved, free, native capability. The frontier has moved to three things — credit/consumption cost management, defensible business value, and readiness grounded in real data exposure. Microsoft ships free Power BI templates for all three (ValueLens, Consumption Central, Cowork Credit Chargeback), and competitors compete on idle-licence reclamation in currency, oversharing readiness, and cross-vendor "shadow AI" spend.

The good news: most of the raw signal is already imported and simply not surfaced.

Status of the older ideas below. This is an ideas archive, not a promise that every item ships in this release. Model/plugin mix and labelled-resource share are surfaced in the DLP governance section. Actual attributed credits have their own Agent costs view; automatic reconciliation with audit estimates is not established. Ideas 1–3 are largely delivered, since Stable 1857, on separate pages: Copilot spend shows who consumed Copilot credits, by person and by organisation group, and Copilot investment models the licence cost per active holder and the licence cost freed by reclaiming idle seats, from administrator-saved assumptions. Putting currency on the Copilot Adoption page itself would still change its hours-only contract and needs a separate product decision.

Tier 1 — high value, data already captured

#	Feature	Why it matters	Data
1	Copilot Credits cost analytics and chargeback	Cowork and agents put Copilot on a metered bill. "What are we spending, on whom" is the budget owner's first question. Roll up to licence/department/manager, add a showback view and a forecast against a prepaid pack	copilot_chats.copilot_credit_estimate_total and the _json breakdown are already stored per interaction
2	Cost per active user	Licence price plus credits divided by active or habitual users — what each productive user actually costs once idle licences are amortised	Above + a per-licence cost input
3	Reclaimable spend in currency	Turns the reclaim count into the number a CFO acts on. Four buckets: purchased-but-unassigned, assigned-never-used, dormant, active	Needs Graph subscribedSkus for purchased-vs-assigned
4	Model mix / deep-reasoning share	Premium models carry far more credit weight. Flags expensive usage patterns	copilot_ai_models already detects deep-reasoning models
5	Grounding rate	Share of interactions grounded in tenant content vs generic chat. Low grounding means a licence is buying what free Copilot Chat already gives — a downgrade recommendation the tool cannot currently make	copilot_event_files, copilot_event_accessed_resources
9	Data-exposure signal	Which interactions actually touched unlabelled or highly-sensitive content. Defensible precisely because it is what Copilot did touch, not a hypothetical scan	CopilotEventAccessedResource.SensitivityLabelId is already stored

Tier 2 — valuable, heavier or needs a new import

• Actual credit consumption: available separately through Agent costs, with delegated consumption. A billed agent's identifier is not automatically matched to its audit identifier, so this is not an implemented estimate-versus-bill reconciliation.
• Deterministic hours models: current seat-holder/licence/Cowork models expose per-reader assumptions. Their time-credit coefficients are not additional fields in the 33-control tenant editor. Crediting meetings by reliably imported actual duration remains a separate idea; the fixed fallback is visibly modelled.
• Before/after adopters vs matched non-adopters — a possible difference-in-differences view, not a shipped causal analysis. Licence history records observed changes from the first completed post-upgrade refresh, not exact assignment dates before that baseline.
• Time-to-first-value and time-to-habit by onboarding cohort.
• Agent depth: the usage-based breadth, depth, builders, reach and growth measures are described in §3.9 and agent growth. Do not interpret them as billed credits or a proven billed/audit identity join.
• Shadow / third-party AI usage — the "paying twice for AI" signal. Our importer currently handles CopilotInteraction only; ConnectedAIAppInteraction / AIAppInteraction are separate ingestible operations.
• Survey comparison guidance — compare approved, aggregated employee-survey results with the adoption workbook outside the product; see Pairing usage with an employee survey. Survey ingestion and individual-response linkage are not part of this feature.

Deliberately rejected

• AI-generated narrative summaries. Microsoft is adding a natural-language analyst agent. This breaches design principle 2.1 outright: prose that reads authoritatively and cannot be verified. Our template-based per-user justifications are deterministic and traceable, which is the better answer, not a lesser one.
• A named-user fluency leaderboard. Microsoft's Readiness Report ranks individuals to find "natural champions". Ranking named individuals against each other is a stated non-goal (§1). A department- or cohort-level wave-planning version is fine; a per-person ranking is not.
• Generic usage reporting. Prompts, trends and per-app breakdowns for their own sake belong in the Copilot reports (Insights → Copilot → Usage and Prompt insights). Anything here must be framed around licence spend, cost drivers or value.

Still open from earlier reviews

• Pre-aggregated rollups. Re-scoped after measurement — see "Measured at scale" below. The analysis derives everything from repeated joins over the raw Copilot facts. Rollups maintained at import time remain the answer for the widest windows and are the prerequisite for the credit analytics above being affordable, but they are no longer the first fix: the dominant cost turned out to be how the aggregate was written, not how much data it read, and that has been fixed without any schema change. This is still the largest piece of engineering on this list and needs migrations plus the before/after benchmarks the schema-change policy requires.
• Streaming the workbook rather than buffering it, and a per-worker concurrency limit on export.
• Caching only compact summary data and paging detail from a persisted analysis, rather than holding the whole scored object graph per window.

---

Performance measurements

Historical engineering measurements on synthetic tenants, kept so that a change can be compared with them. They were never taken against a customer database, and they are not acceptance results for a particular SQL tier or a promise about a current build.

Measured at scale

Historical engineering benchmark on a synthetic 200,000-user tenant — 60,000 Copilot licences, 34,000 with activity, 12,000,000 interactions over 365 days — against LicensedUsersSql, with SET STATISTICS IO/TIME, medians of three runs discarding the cold run. Never against a customer database. The 365-day row is an engineering test, not a selectable portal period: the portal supports rolling 7/28/90/180 days and past ranges of 7–180 days.

The finding was not what the earlier audit assumed. The fact scan was cheap; the cost was asking for three COUNT(DISTINCT ...) in one GROUP BY. SQL Server streams a single distinct aggregate cheaply, but two or more force it to fan the input out through a spool and process each distinct separately — and that spool was most of the query. It also pushed the query past the 90-second command timeout, so on a large tenant the report degraded to a warning rather than merely being slow.

Window		Logical reads	of which worktable	Elapsed
28 days (default)	before	114,750,742	25,413,875	281 s
	after	772,272	232	73 s
365 days (widest)	before	115,172,403	25,835,536	303 s
	after	818,851	792	135 s

Both metrics improve at both selectivities and the spool disappears. Isolation runs confirmed the cause precisely: removing one of the three distinct counts changed almost nothing (36.6M reads), removing all three collapsed the query to 92k — and narrowing app_host from nvarchar(MAX), the intuitive suspect, was worth about 1%.

Correctness was gated before performance: old and new aggregates were run side by side over the same window and compared row for row — 33,999 rows each, zero differing. That comparison exists because the real risk is NULL handling: COUNT(DISTINCT CASE WHEN ... END) ignores NULLs implicitly, whereas the rewrite must exclude them explicitly.

LicensedUsersQuery_NeverAsksForSeveralDistinctCountsInOneGrouping pins the shape, because re-introducing it is an innocent-looking edit whose damage only appears on a tenant nobody tests against.

These query timings are not full API acceptance results or a guarantee for a particular SQL tier. In particular, the 135-second engineering result exceeds the usual 90-second per-query budget; it must not be described as a successful selectable 365-day report. Wider-history rollups and other query-shape work remain separate engineering decisions.

Query cost of individual measures

Seat holders' time saved

One more query in each analysis, named SeatHolderTimeSaved both as a step and as a query in the telemetry (§9). On a synthetic 200,000-user tenant, with 60,000 seat holders and 12 million Copilot interactions, it took about 8 s for a 28-day period and about 43 s for 180 days. That is inside the 90-second per-query limit, but it lengthens the first, uncached load of a long period on a large tenant.

Manager modelling

Manager modelling adds no query. The licensed-user query carries u.manager_id and the manager's account_enabled from the manager join it already makes. Measured on a synthetic 200,000-user tenant, logical reads and plan operators were identical with and without the two columns, at both 50,000 and 2,000 seats. The unlicensed usage already carries each user's id. The computation is one dictionary build and one O(n) pass: about 15 ms at the 50,000 + 50,000 row caps, once per analysis and once per filtered view. A filtered view resolves managers against the whole tenant's rows, so a report inside the filter whose manager is outside it is still resolved.

Agent breadth, depth, builders and reach

One grouped pass over the period's Copilot interactions (IX_copilot_chats_time_stamp_user_id range seek, one row per agent–person pair, department ids from users), plus the Copilot Studio authoring events for builders; no new index. Measured on a synthetic 200,000-user tenant: 28 days, 10,896 logical reads and ~0.4 s including reading 250,000 pairs; 90 days, 32,637 reads and ~1.1 s for 683,000 pairs — the same reads as the existing agent use by department query. The builders query read 1,572 / 3,971 pages (28 / 90 days) on 3 million audit events.

Microsoft's tenant prompt figures

One query in its own analysis step (MicrosoftReportFigures, step and query alike), seeking the table's unique index (report_type, report_period_days, report_date, app_name). The table holds one row per Copilot surface per refresh date and is cleaned with the other usage reports, so the cost does not grow with tenant size. The SQL is on the Analyst view's line.

Agent growth, year on year

Measured on a synthetic 200k-user tenant (9.6M Copilot interactions over 400 days, 3.1M of them with an agent, 90,000 Copilot users, 2,000 agents of mixed origin) against the real query, nine warm runs, cold run discarded, OPTION (RECOMPILE). The series is one range seek on IX_copilot_chats_time_stamp_user_id, collapsed once to (window, agent, person) in a temp table. Which agents count is decided in C# by the agent-origin classifier and sent to SQL Server as two JSON lists of agent ids, so the query text never changes with the size of the agent table. The issue's prototype asked for two COUNT(DISTINCT ...) in one grouping, which spools:

Query	Span	Logical reads	Elapsed (median)
As shipped, customer-built agents	28 days	10,156	0.75 s
	364 days	75,382	3.9 s
	392 days (the series)	77,953	4.0 s
As shipped, every agent counted	392 days	79,028	4.8 s
Prototype in #645 (every agent)	28 days	1,236,303	0.6 s
	364 days	9,114,809	5.3 s
	392 days	9,297,683	5.3 s
Billing evidence	392 days	14,836	0.1 s

Counting every agent, the shipped query and the prototype return the same figures for every window. At the span that ships, the shipped shape reads about 118 times less and is faster. At 28 days the prototype is about 0.15 s faster, because the temp table is a fixed cost, but it reads over 100 times more. A 50,000-agent list of agents of unknown origin adds about 0.15 s. The step runs beside the inventory rather than after it, and it is far inside the 90-second per-query budget. No schema change.

---

8. Demo and test data

The tool can only be demonstrated — or reviewed — on a population that actually varies. Scattering N interactions randomly across N users, which is what volume/importer testing wants, gives every licensed user near-identical usage: the whole tenant lands in one band, and the funnel, the treemap, the profile radar and every "who should I target" panel correctly show nothing. That looks like a broken tool.

Tests.FakeDataGen therefore has an adoption scenario mode, offered (default yes) by Generate combined profiling data (O365 + Copilot):

Shape Copilot usage into adoption personas (all funnel stages, contrasting departments)? [Y/n]

It plants:

• Every funnel stage — Champion, Established, Developing, Trialling, Dormant and Never used.
• Several different shapes at similar scores — "frequent but shallow" (63), "deep but narrow" (58) and "broad but occasional" (49) all sit close together, so the profile radar and the per-component breakdown have something real to distinguish. A tool reporting one number would call these the same user; they need opposite interventions.
• Contrasting departments — each department is assigned a Leading / Progressing / Lagging maturity, so the department charts show variation rather than a flat line.
• Reclaimable licences — including licences still assigned to disabled accounts.
• An unlicensed population with real Copilot usage, which is what the licence-opportunity ranking is built from.
• One agent per health verdict — Keep, Review, Retire and New. These need first/last-use dates up to ~115 days back, so the scenario deliberately reaches past the requested spread; a 90-day run could otherwise never produce a retirement candidate.

Persona targets are expressed only in the three signals the analysis actually measures — distinct active dates, interactions per active date, and distinct app_host values — so the bands are produced by the real scoring code rather than asserted by the generator. At startup the generator re-scores every persona through CopilotAdoptionScoring and warns if any no longer lands in the band it claims, which is what stops a change to a tuning default from silently turning the demo tenant into something that no longer demonstrates what this page describes.

Volume and performance testing should answer no to the persona prompt, or use Generate fake Copilot activity, which keeps the original random scatter.

---

API changes

Cowork usage-report properties removed

Builds from Stable 1833 to 1846 tried to import Microsoft's per-user Cowork usage report from a Graph function that doesn't exist (see Known issues and Cowork use comes from the Copilot audit log). When that import was removed, so was everything that only the report could have supplied.

The Copilot Adoption API's JSON no longer contains these 22 properties. None was added or renamed:

coworkAutomationRatioPct, coworkCreditsPerTask, coworkMinutesSavedPerTask, coworkReportActiveDays, coworkReportLastActivityDate, coworkReportRetainedUser, coworkReportRetainedUsers, coworkReportRetentionPct, coworkReportScheduledTasks, coworkReportTotalTasks, coworkReportUserInitiatedTasks, coworkReportUsers, coworkTasks, coworkTasksPerActiveUser, coworkTaskUsers, coworkUsageReportAvailable, coworkUsageReportDate, coworkUsageReportPeriodDays, observedCoworkTasks, observedTaskRateUsers, observedTasksPerPersonPerMonth and projectedCoworkUsers.

coworkAuditUsers stays, and always equals coworkUsers. An old workbook link that still carries a coworkMinutesSavedPerTask parameter has it ignored.

---

Previous: Copilot Adoption Sources and Outputs Reference. Back to the start: Copilot Adoption Specification.

Reference set: Specification → Metrics reference → Sources and outputs reference → Design notes