Copilot Backfill Recovery

Copilot → Copilot Backfill Recovery

Released hardening and remaining Preview acceptance

1854 ships #757, #763 and #764 for Preview backfill recognition/recovery, cancellation, quota accounting and split-day progress. Cancellation is checked before permissions, between submissions/queries and at page boundaries; committed records stay. Waiting remote searches yield to the next normal cycle. Soft-budget expiry stops extra polls/submissions, not an in-flight SQL commit. These fixes do not close #682's live acceptance gate.

Saved job state repairs a failed shared-ledger mirror before another submission, including when a replacement job inherits that recovery state. This covers tested partial-write recovery, not the unproven remote-POST-before-any-durable-write crash window above.

Malformed/non-array Graph page envelopes fail visibly. Mixed valid/malformed records finish with an unrecognisedAuditData gap and mapping counters rather than clean completion. A continuation 401/403 is reported as graphAccessDenied; only rejected/expired 400/410 continuations use the existing first-page replay through normal de-duplication.

Partial-save recovery also covers interactions inside the live import's recent-event cache window. After verifying that the interaction has no Copilot details, backfill can re-stage its audit record to save the missing details. User-group and URL filters, within-batch de-duplication, and other workloads' normal duplicate checks still apply. Complete Copilot interactions are skipped, not rewritten.

Retryable slice failures retain their underlying exception in the private importer log. The portal and durable job keep sanitized error details, not raw database errors; use the private log to distinguish a failed save from an empty Microsoft response.

Keep Preview. Transport fixtures do not prove a real tenant's AuditLogsQuery.Read.All grant, operation filtering, retention or actual response shape. Those criteria and an accepted deployed live-import impact envelope remain outstanding.

An earlier production-schema synthetic LocalDB baseline imported 200,000 users/interactions in about 65 minutes; concurrent normal-persistence probes of 100 synthetic events reached 44.9 seconds. Resource-metadata network enrichment was disabled. That binary preceded the final cancellation/ledger and recent-cache recovery changes. These are historical local contention findings, not current-source scale proof, Azure SQL throughput promises or controlled before/after live-import evidence.

A separate two-million-interaction memory fixture exercises parsing, state and ID-only persistence; its throughput must not be substituted for SQL performance.

The source-tree Benchmarks\Invoke-CopilotAuditBackfillEvidence.ps1 runner creates a dedicated LocalDB instance and catalog and refuses catalog reuse. Normal Release persistence uses instance-global staging tables: separate catalogs alone do not isolate parallel normal-import experiments. The runner stops only its own instance, never drops/resets a catalog, and restores the original test configuration byte-for-byte. It defaults to LocalDB 15.0 (SQL Server 2019); -LocalDbVersion selects another installed version. This is a developer evidence tool, not a customer deployment step. Build the Release test assembly before running:

.\Benchmarks\Invoke-CopilotAuditBackfillEvidence.ps1 -RegressionOnly -ResultsDirectory <private-results>
.\Benchmarks\Invoke-CopilotAuditBackfillEvidence.ps1 -SqlRegressionOnly -ResultsDirectory <private-results>
.\Benchmarks\Invoke-CopilotAuditBackfillEvidence.ps1 -SqlScale -SqlDays 5 -ResultsDirectory <private-results>

SQL scale exercises the production database upgrader, Graph-record mapper, backfill orchestration, existing-event query and audit staging/merge persistence. Its source is synthetic, not Microsoft Graph, and its state/ledger is in memory, not Azure Table. Five days generate 200,000 fictional users and one million interactions, with one Unicode resource per user repeated across days and distributed synthetic event IDs. Resource-metadata network enrichment is disabled. This uniform fixture is not a model of every tenant's traffic distribution.

The harness records elapsed import time, used data/index pages per table, and paired 100-event normal-persistence probes: a warmed serial probe immediately before a concurrent probe while a backfill batch is active. It excludes the initial cold probe and retains each pair, rather than hiding growth-dependent contention in one median. Probes have no resource children; storage includes those probes and is used pages, not allocated database/log file size. Existing-event query timings cover 100, 1,000 and 5,000 IDs at 0%, 10% and 100% overlap, using five warmed measurements after discarding the first.

The current-source million-interaction run did not complete. The harness is available, but completed SQL throughput, storage-growth delta, paired writer timings and overlap-selectivity evidence are not established. It does not prove no live-import impact, a complete normal-import cycle, or deployed Azure SQL capacity. Keep results private and synthetic. Do not run the experiment against a deployed analytics database.

What must pass before Preview is removed

1. In an explicitly approved Global-service test tenant/runtime application, prove app-only AuditLogsQuery.Read.All access, the CopilotInteraction operation filter, retention and actual payload shapes. Broader audit access requires a separate administrator consent decision; the installer does not grant it automatically.
2. Run one bounded historical query through terminal status and privately compare authorized response shapes with the parser. Agree a query/page/time budget first; preserve raw responses privately.
3. Exercise cancellation/restart and an overlapping live/backfill interval in the same deployment. Synthetic parser, paging, recovery and resume tests do not replace live-service proof.
4. Measure and accept a deployed normal-import impact envelope. Backfill runs inline after live import with an eight-minute soft budget; an in-flight request or SQL batch can exceed it. Trial a small history window off peak and monitor cycle completion and SQL CPU/Data IO/Log IO.
5. Resolve or explicitly retain the remote-POST-before-durable-write crash-window and distributed multi-worker atomicity limitations. Saved-job/shared-ledger repair is not exactly-once remote submission accounting.

When a backfill finishes

State	What it means	What to do
Completed	Every slice was searched and imported.	Check the result.
Completed with gaps	It finished, but some days aren't complete. A failed day had a search that still failed after three attempts: Microsoft reported it as failed, it couldn't be submitted or its results couldn't be read, or it was still unfinished 24 hours after it was submitted. An incomplete day had an hour that still returned more records than one search can, so part of that hour is missing. Since Stable 1856 (PR #768): the missing part is within a quarter-hour rather than an hour, and the records that search did return are imported.	For failed days, start a new backfill with Earliest date on or before the oldest failed day. Incomplete days only happen in very large tenants, and searching again won't fill them.
Failed	It stopped because of something that affects every day; Last error says what. What it imported before stopping stays.	Fix the cause, then start a new backfill.
Cancelled	An administrator cancelled it. What it imported stays.	Nothing.

A new backfill can't target only the affected days. It searches every day from its earliest date up to the moment you start it, and skips what's already there, so choose the latest earliest date that covers the gap.

On Administration → Service health, the backfill has its own row under Component health, Copilot audit backfill. Queued, running, completed and cancelled backfills report Healthy. A failed backfill, or one that completed with gaps, reports Degraded, never Unhealthy, because the backfill is optional; that also makes Service health's overall status Degraded. It stays Degraded for seven days after the backfill finished, unless a later backfill replaces it. After that it reports Healthy again, with a note to review the last result on the backfill page.

Backfill errors

Last error	Does it stop the backfill?	What to do
The runtime app is missing AuditLogsQuery.Read.All.	Yes	Grant the permission and consent, give it an importer cycle, then start a new backfill.
Copilot audit import is switched off.	Yes	Switch the Copilot import back on, then start a new backfill.
Microsoft Graph refused the Audit Search request with 401 or 403.	Yes	Check that AuditLogsQuery.Read.All is still consented, then start a new backfill.
Microsoft Graph rejected the Audit Search query shape.	Yes	Graph answered 400 Bad Request, so the backfill needs changing for your tenant. Open an issue.
Microsoft Graph returned audit records, but this build did not recognise their Copilot auditData format.	Yes	Open an issue before you try again.
Microsoft Graph throttled Audit Search query submissions.	No	Nothing. It pauses and carries on.
Microsoft Graph Audit Search reported a failed query.	No	Nothing. The same message covers a search that couldn't be submitted, or was still unfinished 24 hours after it was submitted. Each search is tried up to three times in all, and a day that still fails is listed under Failed days.
A Microsoft Graph Audit Search slice was still truncated after splitting to hours.	No	Nothing. That day is listed under Incomplete days.
A Microsoft Graph Audit Search slice was still truncated after splitting to quarter-hours. The records it returned were imported; the rest of that window is missing. (since Stable 1856, PR #768)	No	Nothing. That day is listed under Incomplete days.
The backfill failed unexpectedly. Check the web job logs.	No	Despite the wording, the backfill carries on next cycle. If the error keeps coming back, look for the exception in Application Insights.

The page can also show an error of its own, at the top. Backfill state could not be read or saved in Azure Storage. means the web app couldn't reach the AnalyticsState table: check the Storage connection string, that the storage account's Table service is reachable, and the runtime service principal's Storage Table Data Contributor role. A Copilot audit backfill is already running. means one was started since the page last refreshed; only one runs at a time.

What a backfill writes, and what it doesn't

• The same as the live import. Backfilled interactions go through the live import's own save path. They get the same Copilot detail rows, agent classification, accessed resources, and meeting and file details, and appear in the Copilot reports on their original dates. The same user scope applies: people outside UserGroupsFilter aren't imported, and people removed by the user-scope purge aren't brought back while the scope stays as it is.
• Compiled profiling weeks aren't updated. The weekly profiling tables behind the Activity & Usage Analytics reports compile each week once, so weeks compiled before the backfill keep their old Copilot figures. Recompile those weeks afterwards: see Rebuilding profiling data → Option 1.
• No duplicates. Before writing, it checks whether each interaction is already in the database with its Copilot details (a copilot_chats row), skips those and counts them as Records already present. The save itself never creates duplicate rows either, so the overlap with the live import, which always includes the last week, is harmless, and so is running a backfill again.
• Complete interactions stay as they are. An interaction that's already there with its Copilot details is skipped whole, so the backfill doesn't fill in the fields captured since build 1807 on older rows.
• Partly saved interactions are completed. An interaction whose audit record was saved without its Copilot details, for example because a save was interrupted, doesn't count as already present: it's saved again and the missing details are written. So running a backfill again also repairs those, apart from roughly the last week, where the import's own duplicate check still counts them as done.
• Its searches are labelled. Each one is named M365 Analytics Insights Copilot backfill followed by its UTC time range, so you can recognise them if they show up in Microsoft Purview's audit search list.

Backfill privacy

• It stores the same data as the live import, from the same audit records. Copilot audit records contain no prompt or response text, and none is stored.
• Its own telemetry in Application Insights, the CopilotAuditBackfill custom events and the HealthCheck events for the CopilotAuditBackfill component, carries only counts and state or error codes: no audit payloads, sign-in names, tenant IDs, record IDs or search IDs. As with every Graph import, a failing request can still appear with its URL in the importer's general error logs.
• The backfill's record in AnalyticsState holds its window, progress and gaps, and the sign-in name of the administrator who started it. Microsoft Graph errors are kept as the HTTP status and Graph error code, without request URLs. A backfill's record is kept for 180 days after it last changed.
• AuditLogsQuery.Read.All can read your whole unified audit log, every workload, not only Copilot. The backfill only asks for Copilot interactions, and nothing else in the solution uses the permission, so you can remove it once you no longer need the backfill.

Checking a backfill's result

The page marks the backfill Preview and says: "This backfill uses a newer Microsoft Graph API whose results have not been verified on every tenant. After the first days complete, check that the imported counts match what you expect before relying on the backfilled history."

To check, count interactions per UTC day before and after:

SELECT CAST(time_stamp AS date) AS [day], COUNT(*) AS interactions
FROM dbo.copilot_chats
WHERE time_stamp >= DATEADD(day, -180, SYSUTCDATETIME())
GROUP BY CAST(time_stamp AS date)
ORDER BY [day];

Days that were missing should now have counts in line with the days around them. To compare a day with the source, run a Microsoft Purview audit search for that day's Copilot interactions; expect fewer here if UserGroupsFilter is set. If the counts look wrong, open an issue.

Its milestones are also recorded in Application Insights:

customEvents
| where name == "CopilotAuditBackfill"
| project timestamp, Outcome = tostring(customDimensions.Outcome), State = tostring(customDimensions.State),
          ErrorCode = tostring(customDimensions.ErrorCode),
          Imported = toint(customMeasurements.RecordsImported), Seen = toint(customMeasurements.RecordsSeen)
| order by timestamp desc

Outcome is started, throttled, submissionBudgetPaused, truncated, sliceFailed, sliceIncomplete or cycleError along the way, and the final state at the end.

---

Return to Copilot · All topics in this guide

Previous: Copilot Backfill · Next: Copilot Agent Evidence