Monitoring

By default, the resources are created with minimal level performance tiers to avoid unexpected Azure consumption. It is necessary to monitor KPIs to ensure the PaaS components are sufficiently scaled.

Related: set up automated health alerts so you're warned about stalls, and see Monitoring system performance for diagnosing slow imports and database growth.

Example of an under-scaled database:

In this case we would recommend upscaling the database to the next performance tier.

Another KPI to monitor: the app service plan CPU & memory percentage.

If you see CPU at 90-100% constantly, you may need to upscale the app-service.

Monitor System Messages in Application Insights

All system logging is also registered in the Application Insights instance created for web-tracking.

Example Log Analytics Queries

"Microsoft 365 importer" web-job messages (queries are multi-line):

traces
| where operation_Name == "Office365ActivityImporter"

See when "Microsoft 365 importer" has finished an import cycle (the wait message is logged between import loops):

traces
| where operation_Name == "Office365ActivityImporter" and message == "Waiting 10 mins..."

For call-logging specifically:

traces
| where operation_Name == "Office365ActivityImporter"
or operation_Name == "CallRecordWebhookController"

Important: a full import cycle should complete at least once every 24 hours. Any longer & the backlog will be growing quicker than the importer is importing. If this happens, you need to upscale the database & app-service-plan. To monitor this automatically, set up the alert described in Health Alerts.

"Application Insights importer" web-job messages:

traces
| where operation_Name == "AppInsightsImporter"

Graph API Webhook messages

traces
| where operation_Name == "CallRecordWebhookController"

Trace every call, failures and successes. A call's lifecycle spans two web-jobs: the webhook (CallRecordWebhookController) records each Graph notification and queues it, then the calls importer (Office365CallsImporter) reads it back off Service Bus and writes it to SQL. This returns every trace from both, newest first, with severityLevel so failures stand out (1 = Information, 2 = Warning, 3 = Error, 4 = Critical) and the Graph call ID pulled out wherever the message contains one:

traces
| where operation_Name in ("CallRecordWebhookController", "Office365CallsImporter")
| extend CallId = extract(@"ID '([^']+)'", 1, message)
| project timestamp, operation_Name, severityLevel, CallId, message
| order by timestamp desc

To see just the webhook side - every message it logs (test pings, valid/invalid change counts, each queued call and any errors), not only the successful ones - filter to the one operation:

traces
| where operation_Name == "CallRecordWebhookController"
| extend CallId = extract(@"ID '([^']+)'", 1, message)
| project timestamp, severityLevel, CallId, message
| order by timestamp desc

Just the failures across the whole calls pipeline (Error + Critical only):

traces
| where operation_Name in ("CallRecordWebhookController", "Office365CallsImporter")
| where severityLevel >= 3
| project timestamp, operation_Name, severityLevel, message
| order by timestamp desc

End-to-end: from webhook receipt to saved in the database. The webhook logs New call POSTed from Graph with ID '<id>' and the importer logs Added call ID '<id>' to database from ServiceBus for the same Graph call ID. Joining them shows which received calls actually made it into the database - a blank SavedAt means the call was received but never persisted (failed, abandoned or skipped):

let received =
    traces
    | where operation_Name == "CallRecordWebhookController"
    | where message startswith "New call POSTed from Graph with ID"
    | extend CallId = extract(@"ID '([^']+)'", 1, message)
    | project ReceivedAt = timestamp, CallId;
let saved =
    traces
    | where operation_Name == "Office365CallsImporter"
    | where message startswith "Added call ID"
    | extend CallId = extract(@"ID '([^']+)'", 1, message)
    | project SavedAt = timestamp, CallId;
received
| join kind=leftouter saved on CallId
| extend SecondsToProcess = datetime_diff('second', SavedAt, ReceivedAt)
| project ReceivedAt, CallId, SavedAt, SecondsToProcess
| order by ReceivedAt desc

Volume of calls recorded by the webhook over time:

traces
| where operation_Name == "CallRecordWebhookController"
| where message startswith "New call POSTed from Graph"
| summarize CallsRecorded = count() by bin(timestamp, 1h)
| render timechart

These queries use the classic Application Insights table names (traces, operation_Name, message, severityLevel). If you query a workspace-based resource from the Log Analytics workspace directly, use AppTraces, OperationName, Message and SeverityLevel instead.

Call records read from Service Bus

traces
| where message contains "ServiceBus"

General Exception Searching

If you want to see where an error may be logged but are unsure where, you can query for all exceptions being logged to get a start:

This gives you a good idea of what may be causing problems, although be careful; some exceptions are normal to see, depending on the circumstances.

For example, if audit data is loaded for external users then those users won’t be found when the web-job tries to load the user from your Azure AD and a "not found" exception will be logged. These errors need to be studied, but exception reporting can give a clue why a certain table isn’t being populated for example, if there’s a blocking issue.

Web-Job Log Files

If Application Insights does not show data for some reason, web-jobs also log files to the standard web-job logs on the app-service:

In Kudu, the app-service logs are available, including any output from either web-job.

Navigate to C:\home\data\jobs\continuous + name of web-job, to see the file-system log.

You can edit the file directly to see the contents or download to your local computer.

Recommended: Setup Health Alerts

Once everything is running, we highly recommend creating alerts for the system to monitor health & be able to respond to problems, should they happen.

See Health Alerts for the recommended set of Application Insights and Azure Monitor alert rules covering data-flow, infrastructure load and app-secret expiration.