title | description | ms.topic | ms.date | ms.reviewer |
---|---|---|---|---|
Syslog collection with Container Insights |
This article describes how to collect Syslog from AKS nodes using Container insights. |
conceptual |
2/28/2024 |
damendo |
Container Insights offers the ability to collect Syslog events from Linux nodes in your Azure Kubernetes Service (AKS) clusters. This includes the ability to collect logs from control plane components like kubelet. Customers can also use Syslog for monitoring security and health events, typically by ingesting syslog into a SIEM system like Microsoft Sentinel.
- You need to have managed identity authentication enabled on your cluster. To enable, see migrate your AKS cluster to managed identity authentication. Note: Enabling Managed Identity will create a new Data Collection Rule (DCR) named
MSCI-<WorkspaceRegion>-<ClusterName>
- Port 28330 should be available on the host node.
- Minimum versions of Azure components
- Azure CLI: Minimum version required for Azure CLI is 2.45.0 (link to release notes). See How to update the Azure CLI for upgrade instructions.
- Azure CLI AKS-Preview Extension: Minimum version required for AKS-Preview Azure CLI extension is 0.5.125 (link to release notes). See How to update extensions for upgrade guidance.
- Linux image version: Minimum version for AKS node linux image is 2022.11.01. See Upgrade Azure Kubernetes Service (AKS) node images for upgrade help.
Navigate to your cluster. Open the Insights tab for your cluster. Open the Monitor Settings panel. Click on Edit collection settings, then check the box for Enable Syslog collection
:::image type="content" source="media/container-insights-syslog/syslog-enable.gif" lightbox="media/container-insights-syslog/syslog-enable.gif" alt-text="Screen recording of syslog being enabled from the Azure portal through the Monitor Settings panel in Container Insights." border="true":::
Use the following command in Azure CLI to enable syslog collection when you create a new AKS cluster.
az aks create -g syslog-rg -n new-cluster --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring --enable-syslog --generate-ssh-key
Use the following command in Azure CLI to enable syslog collection on an existing AKS cluster.
az aks enable-addons -a monitoring --enable-msi-auth-for-monitoring --enable-syslog -g syslog-rg -n existing-cluster
You can also use ARM templates for enabling syslog collection
-
Download the template in the GitHub content file and save it as existingClusterOnboarding.json.
-
Download the parameter file in the GitHub content file and save it as existingClusterParam.json.
-
Edit the values in the parameter file:
aksResourceId
: Use the values on the AKS Overview page for the AKS cluster.aksResourceLocation
: Use the values on the AKS Overview page for the AKS cluster.workspaceResourceId
: Use the resource ID of your Log Analytics workspace.resourceTagValues
: Match the existing tag values specified for the existing Container insights extension data collection rule (DCR) of the cluster and the name of the DCR. The name will be MSCI-<clusterName>-<clusterRegion> and this resource created in an AKS clusters resource group. If this is the first time onboarding, you can set the arbitrary tag values.enableSyslog
: Set to truesyslogLevels
: Array of syslog levels to collect. Default collects all levels.syslogFacilities
: Array of syslog facilities to collect. Default collects all facilities
Note
Syslog level and facilities customization is currently only available via ARM templates.
Deploy the template with the parameter file by using any valid method for deploying Resource Manager templates. For examples of different methods, see Deploy the sample templates.
New-AzResourceGroupDeployment -Name OnboardCluster -ResourceGroupName <ResourceGroupName> -TemplateFile .\existingClusterOnboarding.json -TemplateParameterFile .\existingClusterParam.json
The configuration change can take a few minutes to complete. When it's finished, a message similar to the following example includes this result:
provisioningState : Succeeded
az login
az account set --subscription "Subscription Name"
az deployment group create --resource-group <ResourceGroupName> --template-file ./existingClusterOnboarding.json --parameters @./existingClusterParam.json
The configuration change can take a few minutes to complete. When it's finished, a message similar to the following example includes this result:
provisioningState : Succeeded
To get a quick snapshot of your syslog data, customers can use our built-in Syslog workbook. There are two ways to access the built-in workbook.
Option 1 - The Reports tab in Container Insights. Navigate to your cluster. Open the Insights tab for your cluster. Open the Reports tab and look for the Syslog workbook.
:::image type="content" source="media/container-insights-syslog/syslog-workbook-cluster.gif" lightbox="media/container-insights-syslog/syslog-workbook-cluster.gif" alt-text="Video of Syslog workbook being accessed from Container Insights Reports tab." border="true":::
Option 2 - The Workbooks tab in AKS Navigate to your cluster. Open the Workbooks tab for your cluster and look for the Syslog workbook.
:::image type="content" source="media/container-insights-syslog/syslog-workbook-container-insights-reports-tab.gif" lightbox="media/container-insights-syslog/syslog-workbook-container-insights-reports-tab.gif" alt-text="Video of Syslog workbook being accessed from cluster workbooks tab." border="true":::
Customers can use our Syslog dashboard for Grafana to get an overview of their Syslog data. Customers who create a new Azure-managed Grafana instance will have this dashboard available by default. Customers with existing instances or those running their own instance can import the Syslog dashboard from the Grafana marketplace.
Note
You will need to have the Monitoring Reader role on the Subscription containing the Azure Managed Grafana instance to access syslog from Container Insights.
:::image type="content" source="media/container-insights-syslog/grafana-screenshot.png" lightbox="media/container-insights-syslog/grafana-screenshot.png" alt-text="Screenshot of Syslog Grafana dashboard." border="false":::
Syslog data is stored in the Syslog table in your Log Analytics workspace. You can create your own log queries in Log Analytics to analyze this data or use any of the prebuilt queries.
:::image type="content" source="media/container-insights-syslog/azmon-3.png" lightbox="media/container-insights-syslog/azmon-3.png" alt-text="Screenshot of Syslog query loaded in the query editor in the Azure Monitor Portal UI." border="false":::
You can open Log Analytics from the Logs menu in the Monitor menu to access Syslog data for all clusters or from the AKs cluster's menu to access Syslog data for only that cluster.
:::image type="content" source="media/container-insights-syslog/aks-4.png" lightbox="media/container-insights-syslog/aks-4.png" alt-text="Screenshot of Query editor with Syslog query." border="false":::
The following table provides different examples of log queries that retrieve Syslog records.
Query | Description |
---|---|
Syslog |
All Syslogs |
`Syslog | where SeverityLevel == "error"` |
`Syslog | summarize AggregatedValue = count() by Computer` |
`Syslog | summarize AggregatedValue = count() by Facility` |
`Syslog | where ProcessName == "kubelet"` |
`Syslog | where ProcessName == "kubelet" and SeverityLevel == "error"` |
To modify the configuration for your Syslog collection, you modify the data collection rule (DCR) that was created when you enabled it.
Select Data Collection Rules from the Monitor menu in the Azure portal.
:::image type="content" source="media/container-insights-syslog/dcr-1.png" lightbox="media/container-insights-syslog/dcr-1.png" alt-text="Screenshot of Data Collection Rules tab in the Azure Monitor portal UI." border="false":::
Select your DCR and then View data sources. Select the Linux Syslog data source to view the Syslog collection details.
Note
A DCR is created automatically when you enable syslog. The DCR follows the naming convention MSCI-<WorkspaceRegion>-<ClusterName>
.
:::image type="content" source="media/container-insights-syslog/dcr-3.png" lightbox="media/container-insights-syslog/dcr-3.png" alt-text="Screenshot of Data Sources tab for Syslog data collection rule." border="false":::
Select the minimum log level for each facility that you want to collect.
:::image type="content" source="media/container-insights-syslog/dcr-4.png" lightbox="media/container-insights-syslog/dcr-4.png" alt-text="Screenshot of Configuration panel for Syslog data collection rule." border="false":::
Once setup customers can start sending Syslog data to the tools of their choice
Read more
Share your feedback for this feature here: https://forms.office.com/r/BBvCjjDLTS