Skip to content

Copilot Adoption Troubleshooting

Sam Betts edited this page Nov 12, 2026 · 4 revisions

This page is for administrators and operators who keep Copilot Adoption running. It starts from the symptom you're seeing, explains how a report is produced, and documents the telemetry the analysis sends to Application Insights, with ready-made queries and alert rules.

On this page: Start with the symptom · How a report is produced · Monitoring and telemetry · Queries an operator actually wants · Exceptions

Start with the symptom

What you see Check first
Empty figures everywhere, or "No Microsoft 365 Copilot licences were found in this tenant" Whether the imports have stored anything: Setup → Check that the data is arriving
A warning at the top of the report What it means and what to do: Interpretation → Warnings you might see
A figure shows a dash, or a chart is blank Interpretation → When a figure is blank. A blank is unknown, never zero
"The Copilot Adoption score settings couldn't be read, so this report isn't available right now." (HTTP 503) Table storage connectivity and the runtime identity's access: Settings → How changes apply, and where they are stored
The leadership card says pending, unavailable or stale Leadership comparison → When the comparison is hidden, then its troubleshooting table
Someone is, or isn't, in the reclaimable figure when you didn't expect it Their reclaim tier and any reclaim exclusion
The report is slow, never finishes, or says its figures are incomplete The run's telemetry: find the run, then read what its last event proves
The first load of a period is much slower than the second Expected: each period is analysed once and then cached (below)
Since Stable build 1860 (PR #858): the loading card says Report needs attention, a stage says Failed, or there is a Retry failed analysis button Loading progress and retrying a failed analysis
Since Stable build 1860 (PR #858): "A different report version was returned. Check the report again to align its figures and tables." Occasionally expected: the report was replaced in the meantime. If it keeps happening on an App Service with more than one instance, check session affinity

How a report is produced

  1. Imports collect the evidence. The activity importer reads the Copilot audit records and the Graph directory, licence and usage-report feeds you enabled, each on its own schedule. Opening the page doesn't call Microsoft 365.
  2. The facts are stored. Azure SQL holds Copilot interactions and their metadata (not prompt text), directory and licence facts, usage reports and the optional sources. Operational state, such as the score settings and the leadership group, is in the AnalyticsState Azure Table.
  3. The web app analyses a period on demand. Server-side SQL aggregates the stored facts, and the analysis applies the score and privacy rules. A completed analysis is cached in the web process for about ten minutes (one minute if its figures are incomplete), keyed by period, seat classification and the effective score settings. A filtered view is derived from that analysis.
  4. Settings are read on every request. Each report or export rereads the score settings before using a cached analysis. If they can't be read within the ten-second storage deadline, the report refuses (HTTP 503) rather than falling back to defaults.
  5. The working comes with the answer. The page, the CSV exports and the Excel report all come from the same analysis.

So healthy imports, retained history and readable settings come before interpreting a chart. A cached analysis is neither live telemetry nor a saved historical baseline.

Loading progress and retrying a failed analysis

Since Stable build 1860 (PR #858). Builds before 1860 show only a spinner, Analysing Copilot adoption..., while the analysis runs.

The loading card. While the analysis runs, the page shows a card instead of a spinner:

  • How far it has got: N% of stages processed, and the time so far. The percentage counts the analysis stages that have finished, failed ones included. It is not a time estimate: one large query can hold it still for minutes.
  • What it is doing: checking for a cached report or starting the analysis; waiting for analysis capacity, when no query of this run has started yet (the Queued stage below); loading the analysis inputs; scoring and assembling the figures; or publishing the report.
  • Each stage, under What is loading and what has loaded: Waiting, Loading, Loaded, Failed, Not needed or Blocked by an earlier failure, with how long it took. A failed stage says why, with an error reference: the ErrorId of that run's QueryFailed or StepFailed event (below). To find it: customEvents | where name == "CopilotAdoptionLifecycle" and tostring(customDimensions.ErrorId) == "<reference>".
  • The page's two supporting requests, Filter options and Queries behind the figures. If one fails, Retry request repeats just that request.

When the report is ready, the card says Report ready, or Report needs attention if something failed.

Retry failed analysis. When a stage of the report on screen has failed, for example because a query timed out, the card offers Retry failed analysis. Anyone who can read the report can use it.

  • It runs the whole analysis again, not just the failed stage, so it can take several minutes, and it loads the database as the first run did.
  • It can be used once a minute for the same report. Until then the card says Retry available in Ns, and the server refuses an earlier retry of that report with HTTP 429: An analysis retry was started recently. Wait before retrying.
  • While it runs, the previous report stays on the page but can't be used: The previous report remains visible while a replacement is built. Downloads and printing are paused. Other readers keep getting the previous report until the replacement is ready.
  • The previous report is kept if the replacement fails, can't get a slot to run in, or comes back with incomplete figures where the previous report's figures were complete: The analysis retry failed. The previous report has been kept. or The replacement analysis did not complete safely. The previous report has been kept. A replacement can still have failed stages of its own.
  • Check again asks for the report again without rebuilding it. After a retry that failed, it shows the report that was kept, rather than repeating the retry's error. Only Retry failed analysis rebuilds, at most once a minute.

A different report version. The page checks that its supporting requests were answered from the same analysis as the figures on screen. If one wasn't, for example because the cached report was replaced in the meantime, the page checks again by itself once. If it still doesn't match, it says A different report version was returned. Check the report again to align its figures and tables., with Check again.

More than one web app instance: keep session affinity on. The progress, the retry and its once-a-minute limit belong to the web app instance that runs the analysis: like the analysis cache, they are kept in that instance's memory. On an App Service that runs more than one instance, keep Session affinity (ARR affinity) on, under the App Service's Settings → Configuration → General settings (Microsoft Learn), so that each browser keeps reaching the same instance. Without it, the page's requests reach different instances, each with its own analysis: each instance runs the analysis for itself, the progress can jump between them, the page says A different report version was returned more often, and the once-a-minute limit counts separately on each instance. The installer doesn't change this setting.

Monitoring and telemetry

The analysis is the most expensive thing the web application does, and its failure mode is quiet: a query that exceeds the 90-second command timeout degrades to a warning on the page rather than an error, so a tenant can sit with a half-populated report for months and never raise a ticket. Telemetry exists to make that visible to an operator instead of only to whoever happens to open the page.

It is sent to the solution's own Application Insights resource (the web app's AppInsightsConnectionString setting). The web app sends custom events, traces and exceptions only — it does not install the Application Insights HTTP modules, so there is no requests or dependencies telemetry to look for. There are two custom events:

Event When Use it for
CopilotAdoptionLifecycle Many per run, as each boundary is crossed, plus a heartbeat every 30 seconds Diagnosing a run that is slow, stuck, failing or was killed. This is the one to start with: it exists while the run is still going and survives a run that never finishes
CopilotAdoptionAnalysis One per completed run Trends, percentiles and alerts across runs. It is not emitted by a run that hangs, crashes or is recycled

The lifecycle events carry the run's RunId as a dimension. Every event of the run, and any exception it raised, also has it as operation_Id — and that is the only place the completion event carries it, so join CopilotAdoptionAnalysis on operation_Id. The id is also returned to the browser — runId in the 202 "still building" body, an X-CopilotAdoption-RunId header, and diagnostics.runId in the summary JSON — so a customer's browser trace (HAR) identifies the exact run. Nothing in either event is derived from tenant data: every value is a duration, a count, a compile-time name, a type name or a numeric error code.

The lifecycle stream

Stage Meaning
Started A run began for WindowDays (and a licence override if HasSeatOverride)
Queued / GateAcquired / GateBypassed Each web app instance runs at most two analyses at once, plus one more on the overflow slot after a 3-minute wait. The limit is held in the web app's own memory, so a scaled-out plan has that allowance per instance, and while the app restarts with overlapping recycling, the outgoing and incoming copies can briefly each run their own. Queued means this run waited for a slot; GateAcquired.DurationMs is how long. Heartbeats continue while it waits, with nothing in ActiveOperations. GateBypassed means this run did not get a slot within 3 minutes and went ahead on the single overflow slot. On its own that does not prove anything is hung — earlier queued runs may simply have taken each slot as it freed — so check whether the runs holding the slots are still heartbeating
GateTimedOut Terminal. As GateBypassed, but the overflow slot was taken too, so the run was dropped without touching the database rather than adding load. The page keeps polling, and its next poll queues a fresh run with a new RunId
QueueFull Terminal. Both slots were busy and eight runs were already queued, so the run was turned away at once, without queueing. The page's next poll tries again. Seen repeatedly, more distinct periods and licence overrides are being requested at once than the web app will queue
Abandoned Terminal. The run queued, and nobody had asked for its result for 60 seconds with no request still waiting on it, so it never touched the database. It can happen while the run is still queued (it re-checks every 15 seconds) or just after it reaches a slot or the overflow slot. Normal when someone clicks through several periods
StepStarted / StepCompleted / StepFailed One analysis step (licensed users, weekly trend, opportunities, Cowork readiness...)
QueryStarted / QueryCompleted / QueryFailed One SQL query inside a step, named in Query
ScoringStarted / ScoringCompleted The in-memory scoring of the loaded rows
ServiceReturned → CachePublished → CompletionTelemetryReturned The analysis finished and callers can already read it (the cache is written before either of the first two is sent) → publication confirmed → the completion event was handed to the channel
Failed Terminal. The run threw; an exception with the same RunId is in the exceptions table
Heartbeat Every 30 s while the run is alive, with ActiveOperations (what is running right now), working set, managed heap, Gen2 collections, free thread-pool workers and timer drift
HostStopping The web app shut down during the run; ShutdownReason says why

Reading a failed query. Failure events carry ExceptionType, the innermost exception's type, and also:

  • FailureKind, one of:
    • Timeout
    • Deadlock
    • Throttled
    • Unavailable
    • Connection
    • Permission
    • SchemaMismatch (the database was not upgraded)
    • SqlError
    • Cancelled
    • OutOfMemory
    • Other
  • SqlErrorNumber, for example -2 for a timeout, 1205 for a deadlock, 208 for an invalid object, or 10928/40501 for Azure SQL resource limits.
  • Win32ErrorCode.
  • ExceptionChain: the exception type names, outermost to innermost.

What the last event of a stuck run tells you:

Last evidence What it proves
No Started event and an immediate page A 10-minute web-process cache hit; nothing ran
QueryStarted without QueryCompleted That query had not returned: connection, SQL execution, transfer and materialisation are all inside this boundary. Query Store says whether SQL itself finished
QueryCompleted without StepCompleted SQL returned; the C# projection for that step did not finish
All steps complete, no ScoringCompleted Final in-memory scoring stalled or failed
Repeating Heartbeat The run is alive; read ActiveOperations to see what it is waiting on
HostStopping A graceful shutdown interrupted the run
No terminal event, and the next run has a new InstanceId Abrupt process loss (recycle, out-of-memory, crash)

Incomplete figures are reused for one minute. An analysis whose figures are incomplete (FiguresIncomplete, for example because a query timed out) is reused for one minute, so a reload soon after a passing database problem runs the analysis again. A complete analysis is reused for ten minutes.

The completion event

Every completed analysis also emits one CopilotAdoptionAnalysis event, with per-step durations as measurements (numbers), not dimensions (strings) — so they can be percentiled and charted without casting.

Where What
customDimensions WindowDays, WarningCount, Outcome (Complete / Degraded), TimedOut, SlowestStep, FiguresIncomplete and FailedSteps
customMeasurements TotalMs, plus <Step>Ms for every completed step, including SeatHolderTimeSavedMs (the seat-holder time-saved estimate). Failure counters include FailedQueryCount, TimedOutQueryCount and IncompleteReasonCount; the tenant-prompt step adds MicrosoftReportFiguresMs. Use the emitted step names rather than assuming a fixed list of measurements.

Outcome is Degraded only when the figures are known to be incomplete (FiguresIncomplete: a query failed, or data is still being backfilled) or a query or step failed. TimedOut means a query was classified as a timeout.

Step names come from CopilotAdoptionSteps and query names from CopilotAdoptionQueries, so a saved query or alert keeps working across releases.

Older builds (1838 and earlier). QueryFailed carries only ExceptionType. A SQL command timeout's innermost exception is Win32Exception (SqlClient wraps "The wait operation timed out" in a SqlException with error number -2), so read ExceptionType = Win32Exception with DurationMs ≈ 90,000 as a command timeout; SqlException covers every other SQL error. Don't alert on Outcome == "Degraded" there: it means only WarningCount > 0, and almost every real tenant has at least one caveat that isn't a failure (the Cowork eligibility note is added on every run that sees any Cowork use). TimedOut meant "some step took 90 s or more", and the completion event has no FiguresIncomplete, so check the page's banner or figuresIncomplete in the summary JSON. Use the QueryFailed query below instead.

Queries an operator actually wants

One row per run — start here, and find your RunId:

customEvents
| where timestamp > ago(1d) and name == "CopilotAdoptionLifecycle"
| extend RunId = tostring(customDimensions.RunId), Stage = tostring(customDimensions.Stage),
         WindowDays = toint(customDimensions.WindowDays), ElapsedMs = tolong(customMeasurements.ElapsedMs)
| summarize Started = min(timestamp), LastEvent = max(timestamp), MaxElapsedMs = max(ElapsedMs),
            Stages = make_set(Stage, 64), FailedQueries = countif(Stage == "QueryFailed"),
            FailedSteps = countif(Stage == "StepFailed"), Heartbeats = countif(Stage == "Heartbeat")
            by RunId, WindowDays, InstanceId = tostring(customDimensions.InstanceId)
| join kind=leftouter (
    customEvents
    | where timestamp > ago(1d) and name == "CopilotAdoptionAnalysis"
    | project RunId = operation_Id, FiguresIncomplete = tostring(customDimensions.FiguresIncomplete)
  ) on RunId
| extend Published = set_has_element(Stages, "ServiceReturned") or set_has_element(Stages, "CachePublished")
| extend Outcome = case(
    set_has_element(Stages, "Failed"), "Failed (exception)",
    set_has_element(Stages, "GateTimedOut"), "Dropped: no slot within the queue wait",
    set_has_element(Stages, "QueueFull"), "Turned away: queue full",
    set_has_element(Stages, "Abandoned"), "Abandoned before it touched the database",
    Published and (FailedQueries > 0 or FailedSteps > 0), "Completed - DEGRADED",
    Published and FiguresIncomplete == "true", "Completed - figures incomplete",
    set_has_element(Stages, "CompletionTelemetryReturned"), "Completed",
    Published, "Completed (completion event not confirmed)",
    set_has_element(Stages, "HostStopping"), "Interrupted (host stopping)",
    LastEvent < ago(2m), "Stopped with no terminal event (process lost?)",
    MaxElapsedMs > 600000, "Still running after 10+ min (the page has given up)",
    "Running")
| project Started, RunId, WindowDays, Outcome, TotalSec = round(MaxElapsedMs / 1000.0, 1),
          FailedQueries, FailedSteps, Heartbeats, InstanceId, LastEvent
| order by Started desc

"Completed" means the run finished without a failed query or step, which is not the same as complete figures: they can also be incomplete while the Copilot interaction backfill is catching up. The query picks that up from the completion event's FiguresIncomplete.

CachePublished is the moment the page could show the result. The cache is written just before ServiceReturned is sent, so either of the two means the result was available; the query treats them alike. CompletionTelemetryReturned only confirms the completion event was handed to the channel afterwards. A run with the first and not the second, long after it finished, produced its result, but its CopilotAdoptionAnalysis event is unconfirmed — look for one with the same operation_Id, since the event can arrive when its confirmation does not. A StepFailed with no QueryFailed is still a degraded run: the step's own C# code (a projection, or the scoring) failed after its queries had returned.

Everything one run did, in order:

customEvents
| where timestamp > ago(3d) and name == "CopilotAdoptionLifecycle"
| where tostring(customDimensions.RunId) == "<RunId>"
| project timestamp, Seq = tolong(customMeasurements.Sequence), Stage = tostring(customDimensions.Stage),
          Step = tostring(customDimensions.Step), Query = tostring(customDimensions.Query),
          ExceptionType = tostring(customDimensions.ExceptionType), FailureKind = tostring(customDimensions.FailureKind),
          SqlErrorNumber = tostring(customDimensions.SqlErrorNumber), DurationMs = tolong(customMeasurements.DurationMs),
          ElapsedMs = tolong(customMeasurements.ElapsedMs), Active = tostring(customDimensions.ActiveOperations),
          WorkingSetMB = round(todouble(customMeasurements.ProcessWorkingSetBytes) / 1048576, 0),
          Shutdown = tostring(customDimensions.ShutdownReason)
| order by Seq asc

Which queries fail or are close to the timeout? Seven days, for trends and for picking the slow ones:

customEvents
| where timestamp > ago(7d) and name == "CopilotAdoptionLifecycle"
| extend Stage = tostring(customDimensions.Stage)
| where Stage in ("QueryCompleted", "QueryFailed")
| extend Query = tostring(customDimensions.Query), WindowDays = toint(customDimensions.WindowDays),
         ExceptionType = tostring(customDimensions.ExceptionType), FailureKind = tostring(customDimensions.FailureKind),
         DurationMs = tolong(customMeasurements.DurationMs)
| extend Failure = case(Stage == "QueryCompleted", "",
    isnotempty(FailureKind), FailureKind,
    ExceptionType == "Win32Exception" and DurationMs >= 85000, "Timeout (inferred)",
    ExceptionType)
| summarize Runs = count(), Failures = countif(Stage == "QueryFailed"),
            p50 = percentile(DurationMs, 50), p95 = percentile(DurationMs, 95), MaxMs = max(DurationMs),
            FailureKinds = make_set_if(Failure, Stage == "QueryFailed", 8)
            by Query, WindowDays
| order by Failures desc, p95 desc

Why was SQL slow? The telemetry says which query took the time and how it failed, not what the database was waiting on. For that, take the failing query's UTC window to the Azure SQL database itself: Query Store (Query Performance Insight in the portal) for the top queries by duration and CPU in that window, their wait categories (sys.query_store_wait_stats), and the database's CPU, Data IO, Log IO and DTU metrics. A query that is slow only while the importer is running points at contention rather than at the query.

How slow is the analysis, and which step owns it?

customEvents
| where name == "CopilotAdoptionAnalysis"
| summarize p50 = percentile(todouble(customMeasurements.TotalMs), 50),
            p95 = percentile(todouble(customMeasurements.TotalMs), 95),
            runs = count()
        by window = tostring(customDimensions.WindowDays)
customEvents
| where timestamp > ago(7d) and name == "CopilotAdoptionAnalysis"
| mv-expand kind=array m = customMeasurements
| extend Step = tostring(m[0]), Ms = todouble(m[1])
| where Step endswith "Ms" and Step != "TotalMs"
| summarize p50 = percentile(Ms, 50), p95 = percentile(Ms, 95), MaxMs = max(Ms) by Step
| order by p95 desc

Were several periods analysed at once? Overlapping runs compete for the same database.

customEvents
| where timestamp > ago(1d) and name == "CopilotAdoptionLifecycle"
| summarize Start = min(timestamp), End = max(timestamp), WindowDays = take_any(toint(customDimensions.WindowDays))
            by RunId = tostring(customDimensions.RunId)
| mv-expand Minute = range(bin(Start, 1m), bin(End, 1m), 1m) to typeof(datetime)
| summarize ConcurrentRuns = dcount(RunId), Windows = make_set(WindowDays) by Minute
| where ConcurrentRuns > 1
| order by Minute desc

Web-process memory during analyses (each cached analysis holds tens of thousands of rows per list):

customEvents
| where timestamp > ago(7d) and name == "CopilotAdoptionLifecycle"
| where isnotnull(customMeasurements.ProcessWorkingSetBytes)
| summarize WorkingSetMB = max(todouble(customMeasurements.ProcessWorkingSetBytes)) / 1048576
            by bin(timestamp, 10m), cloud_RoleInstance
| render timechart

Suggested alerts. The Health Alerts procedure alerts on the number of rows a query returns, so each query below returns rows only when there is something to act on:

  • any QueryFailed in 24 hours:

    customEvents
    | where timestamp > ago(24h) and name == "CopilotAdoptionLifecycle"
    | where tostring(customDimensions.Stage) == "QueryFailed"
    | summarize Failures = count(), Queries = make_set(tostring(customDimensions.Query), 16)
    | where Failures > 0

    The last line matters: a summarize with no by returns a row even when nothing matched, which would fire the alert every time;

  • a run with no terminal event after 15 minutes — stuck, very slow, or lost with its process:

    customEvents
    | where timestamp > ago(1h) and name == "CopilotAdoptionLifecycle"
    | summarize Started = min(timestamp), Stages = make_set(tostring(customDimensions.Stage), 64)
                by RunId = tostring(customDimensions.RunId)
    | where Started < ago(15m)
    | where array_length(set_intersect(Stages, dynamic(["ServiceReturned", "CachePublished", "Failed", "HostStopping",
                                                        "Abandoned", "GateTimedOut", "QueueFull"]))) == 0
  • a warning-level rule on p95(TotalMs) above 60 seconds for any window:

    customEvents
    | where timestamp > ago(24h) and name == "CopilotAdoptionAnalysis"
    | summarize p95Ms = percentile(todouble(customMeasurements.TotalMs), 95), Runs = count()
                by WindowDays = tostring(customDimensions.WindowDays)
    | where p95Ms > 60000

They sit alongside the rules on the Health Alerts page.

Asking a customer for evidence? The lifecycle events are privacy-safe by design and can be shared as they are. Exception messages (innermostMessage, and the message of traces) can quote data values. Ask for exception types only, or have the customer review them before sending.

Exceptions

Workbook generation failures are tracked as App Insights exceptions (TrackException) plus an error trace, because a failure there is a defect in this application rather than anything the caller did. The endpoint deliberately returns a plain message instead of the exception text — the default Web API behaviour would put the stack trace in a response body reachable by every admin user.

Telemetry never changes request behaviour: every tracking call in this feature is wrapped so that a logging failure cannot turn a working request into a broken one, or replace a useful error with a confusing one.

Related pages


Previous: Copilot Adoption Leadership Comparison. Back to the start: Copilot Adoption Setup.

Administrators' path: Setup → Settings → Leadership comparison → Troubleshooting

Start here

Copilot analytics

Using the portal

Install & configure

Beyond Copilot

Operate

Upgrading from an older build

Reference

Project

Clone this wiki locally