This is a timer-trigger based Azure Function App written in Python to monitor an Azure-based application for any given event (in our case, we focused on exceptions). If events have occurred, it will send alerts to a given slack channel.
It was deliberately designed to be easily reusable and extendable by other teams. The code should be extremely easy to read and work with, even with little or no Python understanding.
The function is scheduled to run every 5 minutes (customisable) and performs the following tasks:
- Authenticates with an Azure Key Vault to retrieve relevant environment variables.
- Queries application insights to capture all log entries returned for a given query and timescale (both customisable).
- Filters unique operations (some log entries cover multiple operations, which can clutter up the returned logs.)
- Sends a second query to application insights to get the entire log history of a given operation.
- Builds a slack message containing a formatted table of unique event triggering operations in the given timeframe, with generated inline links to the relevant log histories.
- Sends a slack alert (via an environment variable-defined webhook url)
Example Slack alert
The reason for choosing Python in this specific instance was to address the "cold start" problem. It has the lowest execution time variability of all language options, and is second only to C# in median cold start duration.
This particular Azure function is essentially free in terms of both executions (8640 per month, comfortably within the free tier limit of 1 million) and resource consumption (again, easily within the 400k GB-s free tier range).
Alternatives to this approach generally use Azure Monitor Alerts which are more expensive ($1.50 per alert per month).
- Azure Functions Core Tools
- Python 3.7+
- Azure CLI
- An Azure account/subscription
- An Azure Key Vault
- An Application Insights instance you want to monitor
This function requires several environment variables (defined within the given keyvault)
api-key- An API key for your given app insights instance. You can obtain one of these via theAPI Accesssection in the left hand side navigation of your Application Insights instance.app-id- The 'Instrumentation Key' of the Application Insights instance. This can be found in the top part of the Overview section.slack-webhook-url- A slack webhook URL for you to send messages to. For this part you will likely need to contact myself (@Danny on Slack) or a Slack administrator to get a custom slack 'app' set up. This is much more trivial than it sounds, a few clicks at most.tenant-id- Standard for the entire organisation.resource-group-name- The resource group name that the Application Insights instance is stored within.app-insights-resource-name- The name of the Application Insights instance.subscription-id- The subscription id that the Application Insights instance is stored within.
- Clone the repository
git clone https://github.com/hmcts/et-slack-alerts.git
- Open the folder and install dependencies
cd [wherever you cloned it]
<optionally install a virtual environment using e.g. venv>
pip install -r requirements.txt
- Follow the instructions here to get it running locally and published to a given resource group. If you need any help, feel free to reach out.
- You will also need to ensure that the Function App has access to the Key Vault.
- Assign a managed identity to your Function App.
- Navigate to
Key Vault->Access Policies->Add Access Policy. SelectGet. - For
Select principal, choose your Function App's identity.
- Investigate whether it's worth adding a slight delay on log checking to compensate for Azure's logging latency.
- Use the same link generation approach for Azure Monitor Transaction Logs.
Feel free to send a PR with any possible improvements.