Message Tracing

Troubleshooting → Message Tracing

Message tracing

An optional diagnostic, off by default. Only the Office365ActivityImporter web job uses it.

Message tracing saves the API responses the importer receives that match a pattern you set, such as every response that contains a particular app ID, byte for byte, to a blob container in the solution's storage account. Use it when you need to see exactly what Microsoft returned, for example: does any audit record mention our app, and what exactly does it say?

It's for short investigations. The captures are verbatim and can contain personal data, so switch it on, capture what you need, then switch it off and delete them. See Message tracing and privacy.

Turning message tracing on and off

1. On the solution's App Service → Environment variables → App settings, add MessageTraceMatch with your pattern, for example *00000000-0000-0000-0000-000000000000* to capture every response that contains that app ID. The other settings are optional: see Configuration reference → Message tracing.
2. Save. Saving app settings restarts the web app and its web jobs, and the importer reads the settings as it starts. It then logs a warning that tracing is on: see How you can tell message tracing is on.
3. Let the importer run until the responses you want have come through. Captures appear in the container as they're saved.
4. To turn it off, delete MessageTraceMatch (or empty it) and save. Re-running the installer doesn't turn it off, because the installer keeps app settings it doesn't manage.

Writing message tracing patterns

• A pattern is matched against the whole response body, as raw JSON text. To find text anywhere in a response, put * at both ends, as in *contoso-app*. Without them, a pattern only matches a response that is exactly that text.
• * matches any run of characters, including none, and ? matches exactly one character. Matching ignores case.
• Separate several patterns with ;. A response is saved if any of them matches.
• Every pattern needs at least four characters that aren't * or ?, so a mistyped pattern can't capture everything. If any pattern breaks this rule, tracing stays off: the importer logs a warning, Service health shows it, and the imports carry on as normal.
• Match on something that appears as it is in the JSON, such as an ID, a user principal name or a distinctive name. JSON can escape some characters, such as quotes and backslashes, so a pattern that contains them may not match.

What message tracing captures, and what it never does

It inspects the JSON responses the Office365ActivityImporter web job receives from:

• the Office 365 Management Activity API: the audit records it imports;
• Microsoft Graph: users, groups, Teams, calls, Copilot file and meeting details, the Microsoft 365 usage reports and the other Graph calls the importer makes;
• the other Microsoft APIs it calls the same way, such as the Power Platform licensing API and Azure Cost Management for agent costs.

It never captures:

• anything from the AppInsightsImporter web job or the web app;
• responses that aren't JSON, such as the Copilot usage reports, which Microsoft Graph normally returns as CSV files;
• sign-in (token) responses, Azure Key Vault responses, or any response that contains an access or refresh token;
• Microsoft Graph change-notification subscriptions (/v1.0/subscriptions, /beta/subscriptions), since Stable 1856 (PR #768). Their responses echo the Teams calls webhook's clientState, which on builds before 1856 was the runtime app registration's client secret, so a capture made by such a build can contain it: see Teams and calls troubleshooting → After upgrading from a build before 1856;
• Copilot AI interaction history. Prompt and response text is never stored, by this or anything else in the solution.

A capture is the whole response, not just the part that matched. One Activity API response can hold thousands of audit records about many people.

Where the captures are saved

• Container: message-traces (change it with MessageTraceContainer), in the storage account named by the Storage connection string. It's private, and it's created the first time something is saved.
• Blob name: the UTC capture time, the source and a unique ID, in date folders, such as 2026/10/04/134502123-graph-<id>.json. The source is activity-api for the Management Activity API, graph for Microsoft Graph and azure-management for Azure Cost Management; for any other API it's the API's host name, with dots turned into hyphens.
• Content: the response body exactly as received, with its content type.
• Metadata (URL-encoded): capturedUtc, method, requestUrl, statusCode, contentType and matchedPattern. Query parameters that look secret, such as sig, code, token or key, are removed from the URL. The request's own body and headers aren't saved.

The importer reaches the storage account as it does for its runtime state: with the key or SAS in the Storage connection string, or as the runtime service principal if the account refuses that or the connection string has none. The service principal then needs Storage Blob Data Contributor, which the installer already assigns on the solution's storage account, or on its resource group before Stable 1859 (see Prerequisites → Azure RBAC roles).

Message tracing limits

Limit	Default	What happens
Response size: MessageTraceMaxBodyBytes	32 MB	A larger response is skipped. You can raise it to at most 256 MB; a higher value is treated as 256 MB.
Captures per hour: MessageTraceMaxPerHour	500	Once this many have been saved in a clock hour (UTC), further matches are dropped until the next hour. The count is per running importer. The importer logs a warning the first time this happens in each hour.
Upload queue (fixed)	100 captures or 256 MB waiting	If uploads can't keep up, further matches are dropped.

Tracing never makes an import fail because of a limit, a full queue or a failed upload. Captures are uploaded by a background worker. Skipped or dropped captures and failed uploads are counted, the import carries on, and the importer logs a warning about failed uploads at most every five minutes. If storage can't be used, Service health says so and the importer stops collecting new matches for 10 minutes. That covers a container that can't be opened, and storage refusing or failing a save after it opened: for example access is refused or shared-key access was turned off, the service principal lacks or lost Storage Blob Data Contributor, the container was deleted while tracing was on, a firewall, private endpoint or DNS problem blocks the storage account, or the service keeps failing. Matches during the pause count as dropped. After it, the next match opens the container again, which re-creates it if it was deleted and falls back to the runtime service principal (RBAC) if the key is refused. A save rejected only for itself (a 400) just counts as failed and changes nothing else. The one failure tracing can pass on isn't new: if the connection drops while a response is being read, that request fails with the same kind of error the import would have hit without tracing, and the importer's usual retry applies.

What it costs while it's on. Every JSON response is held in memory and checked against your patterns, so the importer uses somewhat more memory and CPU. When MessageTraceMatch isn't set, nothing is inspected.

How you can tell message tracing is on

• At start-up the importer logs a warning: "MESSAGE TRACING IS ENABLED: every API response that matches '…' is saved in full to blob container '…' in the solution's storage account. These responses can contain personal data. Remove the MessageTraceMatch app setting to turn it off."

• After every import cycle it logs "Message tracing summary: saved=…, dropped=…, failed=…, queued=…". The first three are totals since the importer started; queued is what's still waiting to upload.

• Administration → Service health shows a Message tracing component (Seguimiento de mensajes in Spanish):

  Status	When
  Degraded: tracing is enabled	For as long as tracing is on and saving works; the importer reports it again every cycle. It also makes the overall status Degraded. That's a deliberate reminder, not a fault.
  Degraded: configuration is invalid	MessageTraceMatch is set, but a pattern was refused, so tracing is off.
  Degraded: can't save to Azure Blob storage	At start-up: MessageTraceMatch is set, but there's no Storage connection string or the container name isn't valid, so tracing is off. While running: the container couldn't be opened, or storage refused or failed a save, so nothing is being saved; this replaces the enabled state until the container opens again.
  No row	Tracing is off. The importer reports "Message tracing is off." as Healthy, which replaces any earlier Degraded status, and Service health doesn't list it. An install that never uses tracing has no Message tracing row.

  In each Degraded case the importer's warning in the log says what's wrong. When storage is the problem, at start-up or later, Service health says: "Message tracing is requested but can't save to Azure Blob storage, so no responses are being saved; imports continue normally. Check the Storage connection string, the blob container name, network access to the storage account and the Storage Blob Data Contributor role."

To find these lines in Application Insights:

traces
| where message contains "message tracing" or message contains "MessageTrace"
| project timestamp, severityLevel, message
| order by timestamp desc

Message tracing and privacy

• Captures are verbatim API responses, so they can contain personal data: names, email addresses and IP addresses in audit records and user profiles and, if those imports are on, Teams channel messages and sent-email subjects, recipients and bodies. A request URL in the metadata can include a person's sign-in name. Treat the container like the analytics database.
• Keep it short and narrow: switch tracing on for as little time as you can, with patterns as specific as you can make them.
• Consider a lifecycle management rule that deletes the container's blobs after a few days.
• When you've finished, remove MessageTraceMatch and delete the container. Removing the setting doesn't delete what was saved.
• Anyone who can read the storage account's blobs, with its keys or a role such as Storage Blob Data Reader, can read the captures.
• Your patterns aren't secret: the start-up warning names them, so they appear in the web job log and Application Insights, and each capture's metadata records the one that matched. Don't use a pattern that is itself sensitive, such as a person's email address, unless that's acceptable there. The response bodies themselves never go to Application Insights.
• Remove personal data before you attach a capture to a GitHub issue.

---

Return to Troubleshooting · All topics in this guide

Previous: SQL Upgrade Troubleshooting