Skip to content

Latest commit

 

History

History
496 lines (338 loc) · 26.8 KB

connect-cef-syslog-ama.md

File metadata and controls

496 lines (338 loc) · 26.8 KB
Raw
title description author ms.author ms.topic ms.custom ms.date
Ingest Syslog and CEF messages to Microsoft Sentinel with the Azure Monitor Agent
Ingest and filter Syslog messages, including those in Common Event Format (CEF), from Linux machines and from network and security devices and appliances to your Microsoft Sentinel workspace, using data connectors based on the Azure Monitor Agent (AMA).
yelevin
yelevin
how-to
linux-related-content
04/22/2024

Ingest Syslog and CEF messages to Microsoft Sentinel with the Azure Monitor Agent

This article describes how to use the Syslog via AMA and Common Event Format (CEF) via AMA connectors to quickly filter and ingest Syslog messages, including those in Common Event Format (CEF), from Linux machines and from network and security devices and appliances. To learn more about these data connectors, see Syslog via AMA and Common Event Format (CEF) via AMA connectors for Microsoft Sentinel.

Prerequisites

Before you begin, you must have the resources configured and the appropriate permissions described in this section.

Microsoft Sentinel prerequisites

Log forwarder prerequisites

If you're collecting messages from a log forwarder, the following additional prerequisites apply:

  • You must have a designated Linux VM (your Log forwarder) to collect logs.

  • If your log forwarder isn't an Azure virtual machine, it must have the Azure Arc Connected Machine agent installed on it.

  • The Linux log forwarder VM must have Python 2.7 or 3 installed. Use the python --version or python3 --version command to check. If using Python 3 make sure it's set as the default command on the machine, or run the scripts below with the 'python3' command instead of 'python'.

  • The log forwarder must have either the syslog-ng or rsyslog daemon enabled.

  • For space requirements for your log forwarder, refer to the Azure Monitor Agent Performance Benchmark. You can also review this blog post, which includes designs for scalable ingestion.

  • Your log sources (your security devices and appliances) must be configured to send their log messages to the log forwarder's Syslog daemon instead of to their local Syslog daemon.

Avoid data ingestion duplication

Using the same facility for both Syslog and CEF messages may result in data ingestion duplication between the CommonSecurityLog and Syslog tables.

To avoid this scenario, use one of these methods:

  • If the source device enables configuration of the target facility: On each source machine that sends logs to the log forwarder in CEF format, edit the Syslog configuration file to remove the facilities used to send CEF messages. This way, the facilities sent in CEF won't also be sent in Syslog. Make sure that each DCR you configure in the next steps uses the relevant facility for CEF or Syslog respectively.

    To see an example of how to arrange a DCR to ingest both Syslog and CEF messages from the same agent, go to Syslog and CEF streams in the same DCR later in this article.

  • If changing the facility for the source appliance isn't applicable: Use an ingest time transformation to filter out CEF messages from the Syslog stream to avoid duplication, as shown in the query example below. The data will be sent twice from the collector machine to the workspace.

    source |
where ProcessName !contains "CEF"

Configure machine security

Make sure to configure the machine's security according to your organization's security policy. For example, you can configure your network to align with your corporate network security policy and change the ports and protocols in the daemon to align with your requirements. To improve your machine security configuration, secure your VM in Azure, or review these best practices for network security.

If your devices are sending Syslog and CEF logs over TLS (because, for example, your log forwarder is in the cloud), you need to configure the Syslog daemon (rsyslog or syslog-ng) to communicate in TLS:

Set up the data connectors

Select the appropriate tab to see the instructions for syslog or CEF.

Syslog

Set up the Syslog via AMA connector

The setup process for the Syslog via AMA connector has two parts:

  1. Install the Azure Monitor Agent and create a Data Collection Rule (DCR).

  2. If you're collecting logs from other machines using a log forwarder, run the "installation" script on the log forwarder to configure the Syslog daemon to listen for messages from other machines, and to open the necessary local ports.

CEF

Set up the Common Event Format (CEF) via AMA connector

The setup process for the CEF via AMA connector has two parts:

  1. Install the Azure Monitor Agent and create a Data Collection Rule (DCR).

  2. Run the "installation" script on the log forwarder to configure the Syslog daemon to listen for messages from other machines, and to open the necessary local ports.

Install the AMA and create a Data Collection Rule (DCR)

You can perform this step in one of two ways:

  • Deploy and configure the Syslog via AMA or Common Event Format (CEF) via AMA data connector in the Microsoft Sentinel portal. With this setup, you can create, manage, and delete DCRs per workspace. The AMA will be installed automatically on the VMs you select in the connector configuration.
    —OR—
  • Send HTTP requests to the Logs Ingestion API. With this setup, you can create, manage, and delete DCRs. This option is more flexible than the portal. For example, with the API, you can filter by specific log levels, where with the UI, you can only select a minimum log level. The downside is that you have to manually install the Azure Monitor Agent on the log forwarder before creating a DCR.

Select the appropriate tab below to see the instructions for each way.

Microsoft Sentinel portal

Open the connector page and start the DCR wizard

  1. Open the Azure portal and navigate to the Microsoft Sentinel service.

  2. Select Data connectors from the navigation menu

  3. Type Syslog in the Search box. From the results, select the Syslog via AMA connector.

  4. Select Open connector page on the details pane.

  5. In the Configuration area, select +Create data collection rule.

    :::image type="content" source="media/connect-cef-ama/syslog-connector-page-create-dcr.png" alt-text="Screenshot showing the CEF via AMA connector page." lightbox="media/connect-cef-ama/cef-connector-page-create-dcr.png":::

  6. In the Basic tab:

    • Type a DCR name.
    • Select your subscription.
    • Select the resource group where you want to locate your DCR.

    :::image type="content" source="media/connect-cef-ama/dcr-basics-tab.png" alt-text="Screenshot showing the DCR details in the Basic tab." lightbox="media/connect-cef-ama/dcr-basics-tab.png":::

  7. Select Next: Resources >.

Define resources (VMs)

In the Resources tab, select the machines on which you want to install the AMA—in this case, your log forwarder machine. (If your log forwarder doesn't appear in the list, it might not have the Azure Connected Machine agent installed.)

  1. Use the available filters or search box to find your log forwarder VM. You can expand a subscription in the list to see its resource groups, and a resource group to see its VMs.

  2. Select the log forwarder VM that you want to install the AMA on. (The check box will appear next to the VM name when you hover over it.)

    :::image type="content" source="media/connect-cef-ama/dcr-select-resources.png" alt-text="Screenshot showing how to select resources when setting up the DCR." lightbox="media/connect-cef-ama/dcr-select-resources.png":::

  3. Review your changes and select Next: Collect >.

Select facilities and severities and create the DCR

Note

Using the same facility for both Syslog and CEF messages may result in data ingestion duplication. Learn how to avoid data ingestion duplication.

  1. In the Collect tab, select the minimum log level for each facility. When you select a log level, Microsoft Sentinel collects logs for the selected level and other levels with higher severity. For example, if you select LOG_ERR, Microsoft Sentinel collects logs for the LOG_ERR, LOG_CRIT, LOG_ALERT, and LOG_EMERG levels.

    :::image type="content" source="media/connect-cef-ama/dcr-log-levels.png" alt-text="Screenshot showing how to select log levels when setting up the DCR.":::

  2. Review your selections and select Next: Review + create.

  3. In the Review and create tab, select Create.

    :::image type="content" source="media/connect-cef-ama/dcr-review-create.png" alt-text="Screenshot showing how to review the configuration of the DCR and create it.":::

  • The connector will install the Azure Monitor Agent on the machines you selected when creating your DCR.

  • You will see notifications from the Azure portal when the DCR is created and the agent is installed.

  • Select Refresh on the connector page to see the DCR displayed in the list.

Microsoft Sentinel portal

Open the connector page and start the DCR wizard

  1. Open the Azure portal and navigate to the Microsoft Sentinel service.

  2. Select Data connectors from the navigation menu

  3. Type CEF in the Search box. From the results, select the Common Event Format (CEF) via AMA connector.

  4. Select Open connector page on the details pane.

  5. In the Configuration area, select +Create data collection rule.

    :::image type="content" source="media/connect-cef-ama/cef-connector-page-create-dcr.png" alt-text="Screenshot showing the CEF via AMA connector page." lightbox="media/connect-cef-ama/cef-connector-page-create-dcr.png":::

  6. In the Basic tab:

    • Type a DCR name.
    • Select your subscription.
    • Select the resource group where you want to locate your DCR.

    :::image type="content" source="media/connect-cef-ama/dcr-basics-tab.png" alt-text="Screenshot showing the DCR details in the Basic tab." lightbox="media/connect-cef-ama/dcr-basics-tab.png":::

  7. Select Next: Resources >.

Define resources (VMs)

In the Resources tab, select the machines on which you want to install the AMA—in this case, your log forwarder machine. (If your log forwarder doesn't appear in the list, it might not have the Azure Connected Machine agent installed.)

  1. Use the available filters or search box to find your log forwarder VM. You can expand a subscription in the list to see its resource groups, and a resource group to see its VMs.

  2. Select the log forwarder VM that you want to install the AMA on. (The check box will appear next to the VM name when you hover over it.)

    :::image type="content" source="media/connect-cef-ama/dcr-select-resources.png" alt-text="Screenshot showing how to select resources when setting up the DCR." lightbox="media/connect-cef-ama/dcr-select-resources.png":::

  3. Review your changes and select Next: Collect >.

Select facilities and severities and create the DCR

Note

Using the same facility for both Syslog and CEF messages may result in data ingestion duplication. Learn how to avoid data ingestion duplication.

  1. In the Collect tab, select the minimum log level for each facility. When you select a log level, Microsoft Sentinel collects logs for the selected level and other levels with higher severity. For example, if you select LOG_ERR, Microsoft Sentinel collects logs for the LOG_ERR, LOG_CRIT, LOG_ALERT, and LOG_EMERG levels.

    :::image type="content" source="media/connect-cef-ama/dcr-log-levels.png" alt-text="Screenshot showing how to select log levels when setting up the DCR.":::

  2. Review your selections and select Next: Review + create.

  3. In the Review and create tab, select Create.

    :::image type="content" source="media/connect-cef-ama/dcr-review-create.png" alt-text="Screenshot showing how to review the configuration of the DCR and create it.":::

  • The connector will install the Azure Monitor Agent on the machines you selected when creating your DCR.

  • You will see notifications from the Azure portal when the DCR is created and the agent is installed.

  • Select Refresh on the connector page to see the DCR displayed in the list.

Logs Ingestion API

Install the Azure Monitor Agent

Follow these instructions, from the Azure Monitor documentation, to install the Azure Monitor Agent on your log forwarder. Remember to use the instructions for Linux, not those for Windows.

You can create Data Collection Rules (DCRs) using the Azure Monitor Logs Ingestion API. Learn more about DCRs.

Create the Data Collection Rule

  1. Prepare a DCR file in JSON format. The contents of this file will be the request body in your API request.

    For an example, see Syslog/CEF DCR creation request body.

    • Verify that the streams field is set to Microsoft-Syslog for Syslog messages, or to Microsoft-CommonSecurityLog for CEF messages.
    • Add the filter and facility log levels in the facilityNames and logLevels parameters. See examples below.

  2. Create an API request in a REST API client of your choosing.

    1. For the request URL and header, copy the request URL and header below by selecting the copy icon in the upper right corner of the frame.

      PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2022-06-01
      • Substitute the appropriate values for the {subscriptionId} and {resourceGroupName} placeholders.
      • Enter a name of your choice for the DCR in place of the {dataCollectionRuleName} placeholder.

    2. For the request body, copy and paste the contents of the DCR JSON file that you created (in step 1 above) into the request body.

  3. Send the request.

    For an example of the response you should receive, see Syslog/CEF DCR creation response

Associate the DCR with the log forwarder

Now you need to create a DCR Association (DCRA) that ties the DCR to the VM resource that hosts your log forwarder.

  1. Create an API request in a REST API client of your choosing.

  2. For the request URL and header, copy the request URL and header below by selecting the copy icon in the upper right corner of the frame.

    PUT 
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{virtualMachineName}/providers/Microsoft.Insights/dataCollectionRuleAssociations/{dataCollectionRuleAssociationName}?api-version=2022-06-01
    • Substitute the appropriate values for the {subscriptionId}, {resourceGroupName}, and {virtualMachineName} placeholders.
    • Enter a name of your choice for the DCR in place of the {dataCollectionRuleAssociationName} placeholder.

  3. For the request body, copy the request body below by selecting the copy icon in the upper right corner of the frame.

    {
  "properties": {
    "dataCollectionRuleId": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}"
  }
}
    • Substitute the appropriate values for the {subscriptionId} and {resourceGroupName} placeholders.
    • Enter a name of your choice for the DCR in place of the {dataCollectionRuleName} placeholder.

  4. Send the request.

Examples of facilities and log levels sections

Review these examples of the facilities and log levels settings. The name field includes the filter name.

For CEF message ingestion, the value for "streams" should be "Microsoft-CommonSecurityLog" instead of "Microsoft-Syslog".

This example collects events from the cron, daemon, local0, local3 and uucp facilities, with the Warning, Error, Critical, Alert, and Emergency log levels:

    "dataSources": {
      "syslog": [
        {
        "name": "SyslogStream0",
        "streams": [
          "Microsoft-Syslog"
        ],
        "facilityNames": [ 
          "cron",
          "daemon",
          "local0",
          "local3", 
          "uucp"
        ],
        "logLevels": [ 
          "Warning", 
          "Error", 
          "Critical", 
          "Alert", 
          "Emergency"
        ]
      }
    ]
  }
Syslog and CEF streams in the same DCR

This example shows how you can collect Syslog and CEF messages in the same DCR.

See Avoid data ingestion duplication earlier in this article for more information about steps to take when ingesting Syslog and CEF messages using a single agent and DCR.

The DCR collects CEF event messages for:

  • The authpriv and mark facilities with the Info, Notice, Warning, Error, Critical, Alert, and Emergency log levels
  • The daemon facility with the Warning, Error, Critical, Alert, and Emergency log levels

It collects Syslog event messages for:

  • The kern, local0, local5, and news facilities with the Critical, Alert, and Emergency log levels
  • The mail and uucp facilities with the Emergency log level
    "dataSources": {
      "syslog": [
        {
          "name": "CEFStream1",
          "streams": [ 
            "Microsoft-CommonSecurityLog"
          ],
          "facilityNames": [ 
            "authpriv", 
            "mark"
          ],
          "logLevels": [
            "Info",
            "Notice", 
            "Warning", 
            "Error", 
            "Critical", 
            "Alert", 
            "Emergency"
          ]
        },
        {
          "name": "CEFStream2",
          "streams": [ 
            "Microsoft-CommonSecurityLog"
          ],
          "facilityNames": [ 
            "daemon"
          ],
          "logLevels": [ 
            "Warning", 
            "Error", 
            "Critical", 
            "Alert", 
            "Emergency"
          ]
        },
        {
          "name": "SyslogStream3",
          "streams": [ 
            "Microsoft-Syslog"
          ],
          "facilityNames": [ 
            "kern",
            "local0",
            "local5", 
            "news"
          ],
          "logLevels": [ 
            "Critical", 
            "Alert", 
            "Emergency"
          ]
        },
        {
          "name": "SyslogStream4",
          "streams": [ 
            "Microsoft-Syslog"
          ],
          "facilityNames": [ 
            "mail",
            "uucp"
          ],
          "logLevels": [ 
            "Emergency"
          ]
        }
      ]
    }

Run the "installation" script

The "installation" script doesn't actually install anything, but it configures the Syslog daemon on your log forwarder properly to collect the logs.

  1. From the connector page, copy the command line that appears under Run the following command to install and apply the CEF collector: by selecting the copy icon next to it.

    :::image type="content" source="media/connect-cef-ama/run-install-script.png" alt-text="Screenshot of command line on connector page.":::

    You can also copy it from here:

    sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py

  2. Log in to the log forwarder machine where you just installed the AMA.

  3. Paste the command you copied in the last step to launch the installation script.
    The script configures the rsyslog or syslog-ng daemon to use the required protocol and restarts the daemon. The script opens port 514 to listen to incoming messages in both UDP and TCP protocols. To change this setting, refer to the Syslog daemon configuration file according to the daemon type running on the machine:

    • Rsyslog: /etc/rsyslog.conf
    • Syslog-ng: /etc/syslog-ng/syslog-ng.conf

    [!NOTE] To avoid Full Disk scenarios where the agent can't function, we recommend that you set the syslog-ng or rsyslog configuration not to store unneeded logs. A Full Disk scenario disrupts the function of the installed AMA. Read more about RSyslog or Syslog-ng.

Test the connector

  1. To validate that the syslog daemon is running on the UDP port and that the AMA is listening, run this command:

    netstat -lnptv

    You should see the rsyslog or syslog-ng daemon listening on port 514.

  2. To capture messages sent from a logger or a connected device, run this command in the background:

    tcpdump -i any port 514 -A -vv &

  3. After you complete the validation, we recommend that you stop the tcpdump: Type fg and then select Ctrl+C.

  4. To send demo messages, do one of the following:

    • Use the netcat utility. In this example, the utility reads data posted through the echo command with the newline switch turned off. The utility then writes the data to UDP port 514 on the localhost with no timeout. To execute the netcat utility, you might need to install an additional package.

      echo -n "<164>CEF:0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time" | nc -u -w0 localhost 514

    • Use the logger. This example writes the message to the local 4 facility, at severity level Warning, to port 514, on the local host, in the CEF RFC format. The -t and --rfc3164 flags are used to comply with the expected RFC format.

      logger -p local4.warn -P 514 -n 127.0.0.1 --rfc3164 -t CEF "0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time"

  5. To verify that the connector is installed correctly, run the troubleshooting script with one of these commands:

    • For CEF logs, run:

       sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --cef

    • For Cisco Adaptive Security Appliance (ASA) logs, run:

      sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --asa

    • For Cisco Firepower Threat Defense (FTD) logs, run:

      sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --ftd

Related content