Monitoring

By default, the resources are created with minimal level performance tiers to avoid unexpected Azure consumption. It is necessary to monitor KPIs to ensure the PaaS components are sufficiently scaled.

Related: set up automated health alerts so you're warned about stalls, and see Monitoring system performance for diagnosing slow imports and database growth.

Example of an under-scaled database:

In this case we would recommend upscaling the database to the next performance tier.

Another KPI to monitor: the app service plan CPU & memory percentage.

If you see CPU at 90-100% constantly, you may need to upscale the app-service.

Monitor System Messages in Application Insights

All system logging is also registered in the Application Insights instance created for web-tracking.

Example Log Analytics Queries

"Office 365 importer" web-job messages (queries are multi-line):

traces
| where operation_Name == "Office365ActivityImporter"

See when "Office 365 importer" has finished an import cycle (the wait message is logged between import loops):

traces
| where operation_Name == "Office365ActivityImporter" and message == "Waiting 10 mins..."

For call-logging specifically:

traces
| where operation_Name == "Office365ActivityImporter"
or operation_Name == "CallRecordWebhookController"

Important: a full import cycle should complete at least once every 24 hours. Any longer & the backlog will be growing quicker than the importer is importing. If this happens, you need to upscale the database & app-service-plan. To monitor this automatically, set up the alert described in Health Alerts.

"Application Insights importer" web-job messages:

traces
| where operation_Name == "AppInsightsImporter"

Graph API Webhook messages

traces
| where operation_Name == "CallRecordWebhookController"

Call records read from Service Bus

traces
| where message contains "ServiceBus"

General Exception Searching

If you want to see where an error may be logged but are unsure where, you can query for all exceptions being logged to get a start:

This gives you a good idea of what may be causing problems, although be careful; some exceptions are normal to see, depending on the circumstances.

For example, if audit data is loaded for external users then those users won’t be found when the web-job tries to load the user from your Azure AD and a "not found" exception will be logged. These errors need to be studied, but exception reporting can give a clue why a certain table isn’t being populated for example, if there’s a blocking issue.

Web-Job Log Files

If Application Insights does not show data for some reason, web-jobs also log files to the standard web-job logs on the app-service:

In Kudu, the app-service logs are available, including any output from either web-job.

Navigate to C:\home\data\jobs\continuous + name of web-job, to see the file-system log.

You can edit the file directly to see the contents or download to your local computer.

Recommended: Setup Health Alerts

Once everything is running, we highly recommend creating alerts for the system to monitor health & be able to respond to problems, should they happen.

See Health Alerts for the recommended set of Application Insights and Azure Monitor alert rules covering data-flow, infrastructure load and app-secret expiration.