Monitoring

By default, the resources are created with minimal level performance tiers to avoid unexpected Azure consumption. It is necessary to monitor KPIs to ensure the PaaS components are sufficiently scaled.

Related: set up automated health alerts so you're warned about stalls, and see Monitoring system performance for diagnosing slow imports and database growth.

Example of an under-scaled database:

In this case we would recommend upscaling the database to the next performance tier.

Another KPI to monitor: the app service plan CPU & memory percentage.

If you see CPU at 90-100% constantly, you may need to upscale the app-service.

Confirm Import Cycles & Sections Are Finishing (custom events)

As well as free-text log messages (traces), the importer web-jobs emit custom telemetry events to Application Insights (the same instance created for web-tracking that holds the system logs below) whenever they finish a unit of work. These are the clearest signal that the pipeline is actually completing its work — far more reliable than pattern-matching log text:

Event name	Raised when	Raised by (operation_Name)
FinishedImportCycle	A full import loop has finished — one lap of the web-job before it waits ~10 mins and goes around again	Office365ActivityImporter, AppInsightsImporter
FinishedSectionImport	An individual section within a cycle has finished (e.g. audit events, Teams, usage reports, sent emails, user metadata, App Insights hits)	Office365ActivityImporter, AppInsightsImporter

Each event carries a context custom dimension with the operation/section name and how long it took, for example Teams import: 0 hours, 3 mins, and 20 seconds.

These land in the customEvents table (classic Application Insights). If you query a workspace-based resource from the Log Analytics workspace directly, use AppEvents with Name, OperationName and Properties instead of customEvents, name, operation_Name and customDimensions.

Every completed import cycle, newest first — proof the importers are actually looping and finishing:

customEvents
| where name == "FinishedImportCycle"
| extend Duration = tostring(customDimensions.context)
| project timestamp, operation_Name, Duration
| order by timestamp desc

Is the Microsoft 365 importer keeping up? A full cycle should complete at least once every 24 hours (see the note further down). This shows the gap between consecutive cycles, so a growing backlog is obvious at a glance:

customEvents
| where name == "FinishedImportCycle" and operation_Name == "Office365ActivityImporter"
| order by timestamp asc
| extend HoursSincePreviousCycle = datetime_diff('hour', timestamp, prev(timestamp))
| project timestamp, HoursSincePreviousCycle, Duration = tostring(customDimensions.context)
| order by timestamp desc

Which sections finished, and how long each took — spot a slow or stalled section (audit events, Teams, usage reports, sent emails, user metadata, hits, …):

customEvents
| where name == "FinishedSectionImport"
| extend Section = tostring(customDimensions.context)
| project timestamp, operation_Name, Section
| order by timestamp desc

Cycle completions over time (both importers) — a healthy system produces a steady drumbeat; gaps mean a cycle stalled:

customEvents
| where name == "FinishedImportCycle"
| summarize Cycles = count() by bin(timestamp, 1h), operation_Name
| render timechart

Tip: the context string embeds the duration as H hours, M mins, and S seconds. To chart section durations numerically (e.g. to catch a section trending slower over time), parse it out:

customEvents
| where name == "FinishedSectionImport"
| extend ctx = tostring(customDimensions.context)
| extend Section = tostring(split(ctx, ":")[0])
| extend DurationSec = toint(extract(@"(\d+) hours", 1, ctx)) * 3600
                     + toint(extract(@"(\d+) mins", 1, ctx)) * 60
                     + toint(extract(@"(\d+) seconds", 1, ctx))
| project timestamp, Section, DurationSec
| order by timestamp desc

Monitor System Messages in Application Insights

All system logging is also registered in the Application Insights instance created for web-tracking.

Example Log Analytics Queries

"Microsoft 365 importer" web-job messages (queries are multi-line):

traces
| where operation_Name == "Office365ActivityImporter"

See when "Microsoft 365 importer" has finished an import cycle (the wait message is logged between import loops):

traces
| where operation_Name == "Office365ActivityImporter" and message == "Waiting 10 mins..."

For a cleaner signal, prefer the FinishedImportCycle custom event described in Confirm Import Cycles & Sections Are Finishing — it fires exactly once per completed cycle and includes the cycle duration, so you don't have to rely on matching the wait-message text.

For call-logging specifically:

traces
| where operation_Name == "Office365ActivityImporter"
or operation_Name == "CallRecordWebhookController"

Important: a full import cycle should complete at least once every 24 hours. Any longer & the backlog will be growing quicker than the importer is importing. If this happens, you need to upscale the database & app-service-plan. To monitor this automatically, set up the alert described in Health Alerts.

"Application Insights importer" web-job messages:

traces
| where operation_Name == "AppInsightsImporter"

Graph API Webhook messages

traces
| where operation_Name == "CallRecordWebhookController"

Trace every call, failures and successes. A call's lifecycle spans two web-jobs: the webhook (CallRecordWebhookController) records each Graph notification and queues it, then the calls importer (Office365CallsImporter) reads it back off Service Bus and writes it to SQL. This returns every trace from both, newest first, with severityLevel so failures stand out (1 = Information, 2 = Warning, 3 = Error, 4 = Critical) and the Graph call ID pulled out wherever the message contains one:

traces
| where operation_Name in ("CallRecordWebhookController", "Office365CallsImporter")
| extend CallId = extract(@"ID '([^']+)'", 1, message)
| project timestamp, operation_Name, severityLevel, CallId, message
| order by timestamp desc

To see just the webhook side - every message it logs (test pings, valid/invalid change counts, each queued call and any errors), not only the successful ones - filter to the one operation:

traces
| where operation_Name == "CallRecordWebhookController"
| extend CallId = extract(@"ID '([^']+)'", 1, message)
| project timestamp, severityLevel, CallId, message
| order by timestamp desc

Just the failures across the whole calls pipeline (Error + Critical only):

traces
| where operation_Name in ("CallRecordWebhookController", "Office365CallsImporter")
| where severityLevel >= 3
| project timestamp, operation_Name, severityLevel, message
| order by timestamp desc

End-to-end: from webhook receipt to saved in the database. The webhook logs New call POSTed from Graph with ID '<id>' and the importer logs Added call ID '<id>' to database from ServiceBus for the same Graph call ID. Joining them shows which received calls actually made it into the database - a blank SavedAt means the call was received but never persisted (failed, abandoned or skipped):

let received =
    traces
    | where operation_Name == "CallRecordWebhookController"
    | where message startswith "New call POSTed from Graph with ID"
    | extend CallId = extract(@"ID '([^']+)'", 1, message)
    | project ReceivedAt = timestamp, CallId;
let saved =
    traces
    | where operation_Name == "Office365CallsImporter"
    | where message startswith "Added call ID"
    | extend CallId = extract(@"ID '([^']+)'", 1, message)
    | project SavedAt = timestamp, CallId;
received
| join kind=leftouter saved on CallId
| extend SecondsToProcess = datetime_diff('second', SavedAt, ReceivedAt)
| project ReceivedAt, CallId, SavedAt, SecondsToProcess
| order by ReceivedAt desc

Volume of calls recorded by the webhook over time:

traces
| where operation_Name == "CallRecordWebhookController"
| where message startswith "New call POSTed from Graph"
| summarize CallsRecorded = count() by bin(timestamp, 1h)
| render timechart

These queries use the classic Application Insights table names (traces, operation_Name, message, severityLevel). If you query a workspace-based resource from the Log Analytics workspace directly, use AppTraces, OperationName, Message and SeverityLevel instead.

Call records read from Service Bus

traces
| where message contains "ServiceBus"

General Exception Searching

If you want to see where an error may be logged but are unsure where, you can query for all exceptions being logged to get a start:

This gives you a good idea of what may be causing problems, although be careful; some exceptions are normal to see, depending on the circumstances.

For example, if audit data is loaded for external users then those users won’t be found when the web-job tries to load the user from your Azure AD and a "not found" exception will be logged. These errors need to be studied, but exception reporting can give a clue why a certain table isn’t being populated for example, if there’s a blocking issue.

Web-Job Log Files

If Application Insights does not show data for some reason, web-jobs also log files to the standard web-job logs on the app-service:

In Kudu, the app-service logs are available, including any output from either web-job.

Navigate to C:\home\data\jobs\continuous + name of web-job, to see the file-system log.

You can edit the file directly to see the contents or download to your local computer.

Recommended: Setup Health Alerts

Once everything is running, we highly recommend creating alerts for the system to monitor health & be able to respond to problems, should they happen.

See Health Alerts for the recommended set of Application Insights and Azure Monitor alert rules covering data-flow, infrastructure load and app-secret expiration.