Alert Logic Event Collector for Microsoft Azure Event Hubs

Overview

This repository contains the Microsoft Azure web application Node.js source code and Azure Resource Manager (ARM) template required to set up a data collector in Azure to collect and forward events from Event Hubs to Alert Logic.

Installation

To perform the setup required to grant Alert Logic permission to access Events Hub for event collection, you must have:

• A Microsoft Azure account with administrative privileges
• An Alert Logic user account with administrative privileges

Security Permissions for the Collector application

A collector function application uses Managed Service Identity feature for assigning and managing permissions granted to the application. By default, the event hub collector template grants the following roles/permissions to the application service principal:

• Contributor role is granted to the entire resource group where a collector is deployed.
• In case of data collection from an existing event hub, a Reader role is granted to the target resource group where existing event hub is located.

Create an Alert Logic Access Key

From the Bash command line in Azure Cloud Shell run the following commands, where <username> is your Alert Logic user name and <password> is your Alert Logic password:

export AL_USERNAME='<username>'
auth=$(curl -SX POST -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;

For accounts with multi-factor authentication (MFA) enabled:

export AL_USERNAME='<username>'
auth=$(curl -SX POST -d '{"mfa_code": "<mfa_code_here>" }' -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;

An example of a successful response is:

{
  "access_key_id": "712c0b413eef41f6",
  "secret_key": "1234567890b3eea8880d292fb31aa96902242a076d3d0e320cc036eb51bf25ad"
}

Note: If the output is blank, verify the Alert Logic user account permissions. You must have administrator access. For more information about AIMS APIs, see Access and Identity Management Service.

Note the access_key_id and the secret_key values for use in the deployment steps below.

Note: An account can create only five access keys. If you receive a "limit exceeded" response, you must delete some access keys before you can create more. Use the following command to list access keys:

curl -s -X GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq

In the following curl command, replace ACCESS_KEY_ID_HERE with the access key you want to delete.

curl -X DELETE -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys/<ACCESS_KEY_ID_HERE>

Download and deploy the ARM template

You can use either the Microsoft Azure portal or a command line to deploy the template. To perform either procedure, you must first log into the Azure portal.

Note: The steps in this section require an active Azure subscription. To verify your Azure subscription, visit Azure subscriptions blade.

If your organization uses multiple Active Directory tenants, log into the correct tenant where a collector's Managed Service Identity needs to be registered.

The ARM template can be used to configure a new Azure Event Hub or have the Alert Logic collector reuse an existing one.

Two versions of the ARM template are offered for deploying the Event Hub collector: the standard ARM template and the Premium plan ARM template.

• Standard ARM template: The standard ARM template uses the basic 'Consumption' hosting plan which does not support integration with virtual networks and private endpoints. Using this template therefore requires that the Azure Function App storage account has public network access enabled for all networks. The storage account in this case is reachable from the public internet, but remains inaccessible without proper authentication and authorization.
• Premium plan ARM template: The Premium plan ARM template deploys the collector onto a Premium hosting plan which does support integration with virtual networks and private endpoints. Using this template therefore enables the Azure Function App storage account to restrict inbound connections to those originating from within a specific virtual network in your Azure subscription. The use of a Premium hosting plan will incur additional costs in your Azure account.

For more information about these plans, refer to the Azure Functions Premium plan overview.

Deploy with the custom ARM Template in an Azure Subscription

Click the button below to start deployment.

-or-

Use below premium function ARM template.

To deploy the ARM template:

1. Access the ARM template page in Azure, which deploys the standard 'Consumption' plan ARM template.
   -or-
   Access the Premium plan ARM template page in Azure, which will deploy an Azure Functions Premium plan, as described above.

2. To start a deployment, provide the following required template parameters, and then click the Purchase button:

   • Application Name - Type the name of the log source to appear in the Alert Logic console.
   • Alert Logic Access Key ID - Type the access_key_id you created above.
   • Alert Logic Secret Key - Type the secret_key you created above.
   • Alert Logic API endpoint - Leave the default value (api.global-services.global.alertlogic.com).
   • Alert Logic Data Residency - Leave the value as "default".

   Note: The following template parameters are optional and need to be entered only if reusing an existing Event Hub:

   • Event Hub Resource Group - Type the resource group for the existing Event Hub; leave empty if creating a new Event Hub.
   • Event Hub Connection String - Type the connection string for the existing Event Hub; leave empty if creating a new Event Hub.
   • Event Hub Namespace - Type the namespace for the existing Event Hub; leave empty if creating a new Event Hub.
   • Event Hub Name - Type the name of the existing Event Hub.
   • Event Hub Max Throughput Units - The maximum number of throughput units for the Event Hub (Optional).
   • Event Hub Auto Inflate Enabled - Whether auto-inflate is enabled for the Event Hub (Optional).
   • Event Hub Partition Count - The number of partitions for the Event Hub (Optional).
   • Event Hub Retention Days - The number of days to retain data in the Event Hub (Optional).

   Note: This value defaults to insight-operational-logs. This Event Hub is created automatically by Azure when a subscription Log Profile is integrated with Event Hub through the Azure Monitor service. Follow this guide to Stream the Azure Activity Log to Event Hubs.

   Note: The wrong values configured for eventhub Maximum throughput units, Auto-inflate Partitions and Data retention period causes data loss which can't be recovered Follow this guide to Scaling with Event Hubs and Features and terminology in Azure Event Hubs.

   • Event Hub Filter Json - Type the filter in JSON format. For more details click here.
   • Event Hub Filter Regex - Type the filter in REGEX format. For more details click here.
   • Event Hub Consumer Group - Type the name of the consumer group of the existing Event Hub.

   Note: This value defaults to $Default; you can reuse this consumer group if there are no other consumers of this Event Hub. If there are other consumers of the Event Hub, a separate consumer group should be created for the Alert Logic collector, and its name typed here.

   Event Hub Filter Json - Type the filter in JSON format. example: {"resultType":"Success"}

   • Event Hub Filter Regex - Type the filter in REGEX format. example: /*.Policy or "Policy"

   Note: For "Event Hub Filter Json" and "Event Hub Filter Regex", only messages which contain the specified property will be collected. If both the filter values are provided then logs will be collected based on both the values.

   • Enable Application Insights - Enable or Disable Application Insights (Optional) for monitoring invocation logs. Default value is No. Follow this guide to monitor azure functions click here.

3. If you are using the Premium plan ARM template page in Azure, provide the following additional required template parameters.

   • AppService Plan SKU Name - Select App service plan options for Elastic premium (EP1, EP2 or EP3) from the dropdown.
   • Vnet Name - Type the name of the virtual network for virtual network integration. The default value is [format('vnet-{0}', uniqueString(resourceGroup().id))].
   • Function Subnet Name -Type the name of the virtual network subnet to be associated with the Azure Function app. The default value is (al-function-subnet).
   • Private Endpoint Subnet Name - Type the name of the virtual network subnet used for allocating IP addresses for private endpoints. The default value is (al-privateendpoint-subnet).
   • Vnet Address Prefix - Type the IP address space used for the virtual network. The default value is (10.100.0.0/16).
   • Function Subnet Address Prefix - Type the IP address space used for the Azure Function integration subnet. The default value is (10.100.0.0/24).
   • Private Endpoint Subnet Address Prefix - Type the IP address space used for the private endpoints default value is (10.100.1.0/24).

4. Click Purchase.

Note: If you choose to create new event hub via the template then the following event hub scaling parameters are used:

• 20 maximum throughput units,
• auto-inflate is enabled,
• 32 partitions count,
• 7 day data retention period.

If you would like to use other parameters please change respective variable values in the the template or contact Alert Logic.

Event Hub Filtering

a) Event Hub Filter Regex - Type the filter in REGEX format. example: /*.Policy or "Policy"

Note: For "Event Hub Filter Json" and "Event Hub Filter Regex", only messages which contain the specified property will be collected. If both the filter values are provided then logs will be collected based on both the values.

b) Event Hub Filter Json - Type the filter in JSON format. example json log:

   log = [
      {
         "resultType1":"Success1",
         "type":"result",
         "user":"user1"
      },
      {
         "resultType2": {
            "status":"Success2",
            "type":"result",
            "user":"user2"
         }
      },
      {
         "resultType3": {
            "status": {
               "result":"Success3"
            },
            "type": {
               "value": "result"
            },
            "user": {
               "value": "user3"
            }
         }
      }
   ]
   

Description	Filter	Output
Root level filtering example	{ "resultType1":"Success1" }	[ { "resultType1":"Success1", "type":"result", "user":"user1" } ]
Child level filtering example	{ "resultType2": { "status":"Success2" } }	[ { "resultType2": { "status":"Success2", "type":"result", "user":"user2" } } ]
Deeper child level filtering example	{ "resultType3": { "status": { "result":"Success3 } } }	[ { "resultType3": { "status": { "result":"Success3" }, "type": { "value": "result" }, "user":{ "value": "user3" } } } ]
AND condition filtering	[ { "resultType1":"Success1" }, { "resultType2": { "status":"Success2" } } ]	[ { "resultType1":"Success1", "type":"result", "user":"user1" } ]
OR condition filtering	[ { "resultType1":"Success1" }, { "someOtherResultType": { "status":"Success" } } ]	[ { "resultType2": { "status":"Success2", "type":"result", "user":"user2" } } ]
OR condition for same object filtering	[ { "resultType3":{ "status": { "result":"Success3" } } },{ "resultType3": { "type":{ "value":"result" } } } ]	[ { "resultType3": { "status": { "result":"Success3" }, "type": { "value": "result" }, "user":{ "value": "user3" } } } ]

Note: Child level filtering can go deep with the proper sequence of the object. Note: Filtering is based on case sensitiveness

Deploy through the Azure CLI

If you want to deploy the template through the Azure command line interface (CLI), you can use either Azure Cloud Shell or a local installation of Azure CLI.

To deploy through the Azure CLI:

1. In the command line, type the following to create a new resource group:

   Note: The example below creates a new resource group in the "Central US" location.

   az group create --name <new-resource-group-name> --location "Central US"
   

2. In the Azure portal, access the Resource groups blade, and then select the resource group you created.

3. In the command line, type the following command to deploy a template, and enter the required parameters when prompted.

   az group deployment create \
       --resource-group <new-resource-group-name> \
       --template-uri "https://raw.githubusercontent.com/alertlogic/ehub-collector/v1/templates/ehub.json"
   

Verify the Installation

To verify successful installation of the template:

1. In the Azure portal, access Function Apps, and then choose the Alert Logic Event Hub collector function.
2. Click Functions > Master > Monitor and verify the recent log entry has the status of OK and contains no error messages. Example: Ehub source checkin OK.
3. In the Alert Logic console, navigate to Configuration > Deployments > All Deployments > Log Sources, and then filter the list by Push (Office 365, EventHub) collection method.
4. Verify a new Azure Event Hub log source with the name provided during az group deployment create above appears with the source status as OK.

Integrating With Azure Event Hubs

The following links contain instructions to help you integrate different Azure services with Event Hubs.

• Azure Active Directory Logs
• Azure Diagnostic Logs
• Azure Activity Logs
• Azure Security Center Events
• Azure SQL Audit Logs

Uninstallation

If event hub collector was deployed into a new resource group then clean up the following:

• remove the entire resource group.
• delete collector source entry via the Alert Logic UI.

If a collector was installed into an existing resource group which already contained some other resources then please check deployment history for the target resource group to find a list of resources that belong to collector. Right now collector deploys the following resources:

• Function App
• App Service plan
• Storage account
• Optional, if customer not using existing event hub for collection. Eventhub namespace and event hub itself.
• After azure resources are deleted please delete collector source entry from the Alert Logic UI.

Here is the link how to check resource group deployment history

How the collector works

The template creates an AlertLogicIngest-<region-name>-<unique-string> Event Hubs namespace where the "insights-operational-logs" event hub is created. The Alert Logic collector listens to the Azure event hub and forwards incoming events to the Alert Logic Ingestion service. If event collection fails, Alert Logic stores the data in the alertlogic-dl Azure Blob container located in the storage account you specified during template deployment.

Master function

The Master function is a timer trigger function responsible for:

• Registering the Azure web application in the Alert Logic backend
• Reporting health checks to the backend

Note: When you release a new version of the collector, remember to increment the version number in the npm package.json file.

Updater function

The Updater function is a timer triggered function that runs a deployment sync operation every 12 hours to keep the web application up to date.

EHubGeneral function

The EHubGeneral function listens to insights-operational-logs, which is created during collector setup. Collected JSON objects are wrapped into the protobuf structure and then forwarded to the Alert Logic Ingestion service. If processing fails the data is stored in alertlogic-dl container for further processing.

DLBlob function

If EHubGeneral cannot process incoming event hub records, unprocessed messages are saved as blobs to the alertlogic-dl container, so collection can be tried again later. The alertlogic-dl container is located in the collector web application storage account created during collector setup. The DLBlob function processes dead letter blobs very 15 minutes. The DLBlob function lists all blobs located in alertlogic-dl container and processes them according to the function to which the dead letter blob belongs. After a blob is processed, it is removed from the container.

Local development

To enable local development, perform the following procedure:

1. Clone the repo git clone git@github.com:alertlogic/ehub-collector.git.
2. cd ehub-collector
3. Run ./local_dev/setup.sh.
4. Edit ./local_dev/dev_config.js.
5. Run the Master function locally: npm run local-master.
6. Run the Updater function locally: npm run local-updater.
7. Run the EHubGeneral function locally: npm run local-ehub-general.
8. Run npm test to perform code analysis and unit tests.

Please use the following code style as much as possible.

Setting environment in dev_config.js

• process.env.APP_TENANT_ID - The GUID of the tenant (such as alazurealertlogic.onmicrosoft.com)
• process.evn.APP_RESOURCE_GROUP - The name of the resource group where you deployed your application.
• process.env.CUSTOMCONNSTR_APP_CLIENT_ID - The GUID of your application that created the subscription. Note You can obtain this value from Azure > AD > App registrations > Your app name
• process.env.CUSTOMCONNSTR_APP_CLIENT_SECRET - The secret key of your application from App Registrations.
• process.env.CUSTOMCONNSTR_APP_CI_ACCESS_KEY_ID - The access key returned from AIMS above.
• process.env.CUSTOMCONNSTR_APP_CI_SECRET_KEY- The secret key returned from AIMS above.

Useful Links

• Node.js static code analysis tool
• How to monitor Azure functions?
• Server application to Web API authentication.
• Azure Web apps API reference