Skip to content

User Scope and Purge

Sam Betts edited this page Nov 9, 2026 · 7 revisions

Web Portal → User Scope and Purge

User scope

How UserGroupsFilter resolved, and a purge of what was imported about people outside it. The filter itself, and what it does to each import, is described in Configuration reference → Restricting imports to Entra groups.

  • Filter — the configured patterns, whether they resolved and when, and how many people are in scope. If the groups couldn't be read it says why; a 403 from Microsoft Graph means the runtime app registration is missing Group.Read.All. Refresh reads the groups again straight away; otherwise the web app re-reads them about once an hour.
  • Groups — each matched group with its object ID, how many direct user members it has, and which patterns matched it. Patterns that matched no group are listed too.
  • People in the database — how many users are stored, and how many of them are inside and outside the scope.
Purging people outside the scope

The filter only affects what is imported from now on. Purge removes what was imported earlier about everyone outside the scope: people imported before the filter was set, and people who have since left its groups.

Deleted Their audit-log events and Copilot details; web sessions, page views, clicks and searches; page comments and likes; sent-email records; Teams ownership, membership and reactions; Microsoft 365, Copilot and Cowork usage rows; Copilot interaction-history statistics; licence assignments and licence history, Copilot Studio per-user credits and Copilot reclaim exclusions; their user-organisation values; and then their user record.
Anonymised On a Teams call with someone inside the scope, they become the anonymous Unknown User and their call feedback is deleted. A call with nobody inside the scope left on it is deleted. A Power App or flow they were given access to keeps its share record, without them as the recipient. People they managed are kept, with no manager, and replies to their page comments are kept, no longer linked to the comment.
Kept Everything about people inside the scope, tenant-wide figures such as the Copilot summaries and SharePoint site usage, and the records listed under What each erasure path keeps, such as page URLs, their address on email sent by someone inside the scope, and the change log of user-organisation CSV imports.

Since Stable 1859 (PR #815): the purge also deletes the weekly activity profiles compiled from their usage, their Copilot credit history (preview) readings, anything a retired feature left about them in a table an upgrade kept, and the search text, Copilot prompt phrases and their own email addresses once nothing else uses them. Replies to their Viva Engage messages are kept, no longer linked to them. The page then lists what is not removed: see What each erasure path keeps. The box you tick says the purge deletes the data described above about them, no longer everything stored. If a second purge still skips someone, the web app's log names the table that refers to them.

It can't be undone. If you might need the data again, take a database backup first, or note the time you start so that Azure SQL point-in-time restore can take you back.

When it won't start. A purge only ever works from a list of group members read from Microsoft Graph at that moment - complete, and with at least one person in it. It won't start:

  • when no filter is set, because everyone is in scope;
  • when the groups can't be read completely. The last list that did resolve is never used, because someone who has joined a group since would look as if they were outside it. Since Stable 1859 (PR #812), that includes a page of members that Microsoft Graph returns blank or without its list of members. The scope then shows as unavailable, because Microsoft Graph couldn't read the groups, and the purge is refused. Builds before 1859 take the members read up to that page as the whole group;
  • when the filter matches nobody, because that would remove everyone;
  • when there is nobody outside the scope, or another purge is already running.

You also have to tick a box confirming that you understand it can't be undone.

Since Stable 1856 (PR #768): the number you confirm is binding. The page sends it, with the filter it showed, when you start the purge, and the purge is refused (HTTP 409), with nothing removed, if more people are outside the scope by then, or the filter has changed. The page then shows the new numbers and asks you to tick the box again. Fewer people is fine. A running purge also never removes more people in all than the number you confirmed, even after a restart: if it finds more, it stops with The purge stopped because more people are outside the scope than the number confirmed when it started....

While it runs. The purge works in the background, in small batches, while the imports carry on, so you can leave the page. When you come back it shows the phase, how many people it has removed, and how many rows it has changed per table: a table name counts rows deleted, and a table.column name counts rows kept but anonymised. Only one purge runs at a time, even when the web app has several instances, and every instance shows its progress.

  • Stop takes effect after the current batch. What has been removed stays removed; start the purge again to finish it.
  • If the web app restarts - a deployment, a scale operation, platform maintenance - the purge starts again by itself when the web app does, and finishes. It works out afresh who is outside the scope and removes whatever is still stored about them, so nothing is missed. This needs Azure Storage: without it, a restart stops the purge instead (see Where it keeps its progress below). Since Stable 1856 (PR #768): if the members of the UserGroupsFilter groups have changed since you confirmed the purge, it stops instead, so that you can confirm again: The purge stopped when it started again after a restart, because the members of the UserGroupsFilter groups had changed since it was confirmed.... What it removed before stays removed.
  • Changing UserGroupsFilter restarts the web app, and a running purge then stops instead of carrying on: the people it was removing were chosen under the old filter, so some may be inside the new one. Start it again to purge against the new filter. If the filter changes after you confirm a purge but before it has worked out who to remove, it removes nobody.
  • If Azure Storage can't be reached, the page still shows the scope and the counts, but a purge can't start, because it couldn't record its progress. A purge that loses it part-way stops and says so; what it removed stays removed, and starting it again finishes the job.
  • People with new data. The imports keep running during a purge. They re-read the groups about once an hour, so for up to an hour they can still import data about someone who has just left a group - and if they haven't been able to read the groups since they started, they import everyone. Anyone outside the scope who gets new data after the purge has passed a table is kept, together with that data, and counted as skipped. Running the purge again removes them.

How long it takes. Mostly it depends on how many rows it deletes. On a developer machine, with 200,000 users and 2 million audit events, a purge of 20% of the people took about a minute, and a purge of 90% of them (a filter set to a pilot group) about ten minutes. It reads each per-person table once, a batch at a time, so the time grows in step with the size of the database. It also writes transaction log for every row it deletes - around half a kilobyte on that test data, more where rows carry larger payloads. On a smaller Azure SQL service tier that log rate, not the CPU, sets the pace: a purge that removes tens of millions of rows can run for many hours there, and the imports slow down while it runs. Consider scaling the database up for the duration.

Who can run it. Anyone with the Administration permission (Portal.Administration); it doesn't need See PII. Each purge records who started it.

One person, by a DBA. The optional stored procedure dbo.CleanDataByUser removes and anonymises the same data as this purge, for one user ID, in a single transaction. The installer doesn't deploy it: a DBA creates it with Clean.Data.By.User.StoredProc.sql, attached to Stable 1859. Since Stable 1859 (PR #815) it matches this purge. The copy in earlier builds fails part-way for a person with data in several areas, and leaves them half-erased, so if you installed it, run the 1859 script again: it's CREATE OR ALTER. Stop the Office 365 importer WebJob while the procedure runs, because an audit event saved meanwhile can be left behind. It refuses to purge the anonymous Unknown User (error 51002). What it keeps, and how it differs from this purge: What each erasure path keeps.

Logs. Each purge logs its progress to the web app's Application Insights as traces, and a failure as an exception too, with the purge's number and counts. It also records a UserScopePurge custom event when it starts and when it ends or stops, with who started it, the outcome and the counts - a record of every purge that lasts after the page has forgotten it. None of these name anyone the purge removed.

Where it keeps its progress. Nowhere in the analytics database, which holds reporting data only, so there is no database upgrade for this feature. While a purge runs, the list of people it is removing exists only in the purge's own database session, and the one-at-a-time rule is a lock held by that session; both disappear with it. The purge's record - progress, counts, who started it - is kept in the solution's runtime state table, AnalyticsState, in its Azure Storage account (partition UserScopePurge), where a finished purge's record stays for 90 days. A deployment with no Storage connection string keeps the record in the web app's memory instead, and the page says so: there a restart stops a running purge rather than resuming it, and with several web app instances only the one running the purge shows its progress.

What each erasure path keeps

The purge and dbo.CleanDataByUser remove and anonymise the same data, and both work only on the analytics database. Neither removes:

  • Figures about the whole tenant, such as the Copilot summaries and SharePoint site usage.
  • Their address on email that someone else sent them. That email belongs to its sender, so it still lists them as a recipient.
  • Other people's records that name them. Someone else's audit record can be about their mailbox or files. On a Teams call, a comment thread, a Viva Engage thread or a share that involves people who stay, those people's records are kept, with this person anonymised or unlinked as described under Anonymised above.
  • Page addresses (URLs). Neither path deletes a URL, even one that only they visited, so a URL that contains their name, such as the address of their OneDrive, can still name them.
  • Administrative records: who installed the solution, changed a portal setting, excluded someone from Copilot reclaim recommendations, or started a user-organisation import.
  • User-organisation imports: the change log, kept in Azure Storage, that records whom each CSV import changed, and a file that is still waiting to be applied.
  • Anything outside the analytics database, such as application logs, the importers' state in Azure Table storage, message-tracing captures, and database backups until they expire.

Where they differ:

The purge CleanDataByUser
Removes Everyone outside UserGroupsFilter, started by an administrator on this page One user ID, run by a DBA
Email addresses Deletes the purged people's own addresses once no email uses them. The addresses of other people they emailed stay, even when no email uses them any more Deletes the addresses on the person's own sent email, their own included, once no other email uses them
If it fails or is stopped part-way What it removed stays removed; start it again to finish Nothing is removed: it runs in one transaction, and rolls everything back

Neither stops later imports from collecting them again. After the purge, UserGroupsFilter keeps these people out of later imports, apart from the cases under People with new data above. But Copilot credit history (preview), if it's switched on, reads every directory user whatever the scope, so its next collection stores readings about them again. CleanDataByUser doesn't change who is imported: someone still in your directory, and inside UserGroupsFilter if you set one, is imported again by the next import cycles.


Return to Web Portal · All topics in this guide

Previous: Portal Administration

Start here

Copilot analytics

Using the portal

Install & configure

Beyond Copilot

Operate

Upgrading from an older build

Reference

Project

Clone this wiki locally