AlertLogic Office 365 Log Collector
The instructions in this document are for an older setup of the Microsoft Office 365 log collector. If this is your first time setting up Microsoft Office 365 collectors, see Configure Microsoft Office 365 Log Collector.
This repository contains the Microsoft Azure web application Node.js source code and an Azure Resource Manager (ARM) template to set up a data collector in Azure, which collects and forwards Microsoft Office 365 log data to Alert Logic Log Management.
To perform the set up required to grant Alert Logic permission access to collect Office 365 logs, you must have access to the following:
- A Microsoft Office 365 subscription with administrative privileges
- A Microsoft Azure account with administrative privileges
- An Alert Logic user account with administrative privileges
In the Office 365 portal, you must register a new Office 365 web application to collect Office 365 logs.
To register an Office 365 web application to collect logs:
- Log into the Office 365 portal as an Active Directory tenant administrator.
- Navigate to
Admin Centers
and thenAzure AD
. - In the left navigation area, click
Azure Active Directory
, and then selectApp Registrations
. - Click
+ New application registration
and then provide the following information:Name
- Provide a name for the new application (For examplealo365collector
).- Select Single tenant option for supported account types.
- Leave the Redirect URI blank.
- Click Register
- From the
All applications
tab on theApp registration (Preview)
blade, selectAll apps
, and then click the application name you created. - Note the
Application ID
, for example,a261478c-84fb-42f9-84c2-de050a4babe3
- On the main panel under the new application, select
View API Permissions
, and then click+ Add
. - Locate the
Office 365 Management APIs
, and then clickSelect
. - In
Application permissions
, expand then selectActivityFeed.Read
,ActivityFeed.ReadDlp
andServiceHealth.Read
. - Click
Select
, and then clickDone
. - Click
Grant Permissions
, and then clickYes
. Note: Only the Active Directory tenant administrator can grant permissions to an Azure Active Directory application. - On the
Settings
panel for the application, selectCertificated & Secrets
. - Select the
+ New client secret
button. - Type a key
Description
, and then setDuration
toNever expires
. - Click
Add
. Note: Save the key value, which you need during ARM template deployment. - From the
Registered App
blade, click the link underManaged application in local directory
, and then clickProperties
. - Get the
Service Principal ID
associated with the application. (TheService Principal ID
is labeled asObject ID
on the properties page.) Caution: This ID is not the sameObject ID
found under theRegistered app
view or under theSettings
.
We recommend creating an access key via the UI by following These Instructions. However, if you prefer you can use the command line instructions below.
From the Bash command line in Azure Cloud Shell run the following commands, where <username>
is your Alert Logic user name and <password>
is your Alert Logic password:
export AL_USERNAME='<username>'
auth=$(curl -SX POST -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;
For accounts with multi-factor authentication (MFA) enabled:
export AL_USERNAME='<username>'
auth=$(curl -SX POST -d '{"mfa_code": "<mfa_code_here>" }' -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;
An example of a successful response is:
{
"access_key_id": "712c0b413eef41f6",
"secret_key": "1234567890b3eea8880d292fb31aa96902242a076d3d0e320cc036eb51bf25ad"
}
Note: If the output is blank, verify the Alert Logic user account permissions. You must have administrator access. For more information about AIMS APIs, see Access and Identity Management Service.
Note the access_key_id
and the secret_key
values for use in the deployment steps below.
Note: An account can create only five access keys. If you receive a "limit exceeded" response, you must delete some keys to create more. Use the following command to list access keys:
curl -s -X GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq
Then use the selected access_key_id in the following curl command to delete the key:
curl -X DELETE -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys/<ACCESS_KEY_ID_HERE>
You can use either the Microsoft Azure portal or a command line to deploy the template. To perform either procedure, you must log into the Azure portal.
Note: The steps in this section require an active Azure subscription. To verify your Azure subscription, visit Azure subscriptions blade.
Note: This template will deploy a storage account in the active Azure subscription, alongside the function app, in order to successfully collect Office 365 logs.
If your organization uses multiple Active Directory tenants, log into the same tenant used to Register a New Office 365 Web Application. To find your Office 365 tenant ID, see Find your Office 365 tenant ID.
Click the button below to start deployment.
- Provide the following required template parameters and click the
Purchase
button to start a deployment:Name
- Type the name of the log source to appear in the Alert Logic console.Resource Group
- We recommend that you create a new resource group for the collector.Alert Logic Access Key ID
- Theaccess_key_id
you created aboveAlert Logic Secret Key
- Thesecret_key
you created above.Alert Logic API endpoint
- Leave the default value (api.global-services.global.alertlogic.com).Alert Logic Data Residency
- Leave the value asdefault
.Office365 Content Streams
- The log types you want to collect. Valid values are: ["Audit.AzureActiveDirectory","Audit.Exchange","Audit.SharePoint","Audit.General"]Service Principal ID
- TheObject ID
of the application that created the subscription. Noe You can obtain this value from Azure -> AD -> App registrations -> Your app name -> Link under Managed application in local directory -> Properties -> Object ID.App Client ID
- The GUID of your application that created the subscription. Note You can obtain this value from Azure -> AD -> App registrations -> Your app nameApp Client Secret
- The secret key of your application from App Registrations
- Click
Purchase
.
If you want to deploy the template through the Azure command line (CLI), you can use either Azure Cloud Shell or a local installation of Azure CLI.
To deploy through the Azure CLI:
- In the command line, type the following to create a new resource group.
Note The example below creates a new resource group in the "Central US" location.)
az group create --name <new-resource-group-name> --location "Central US"
- In the Azure portal, access the
Resource groups
blade, and then select the resource group you created. - Select
Access Control (IAM)
, and addWebsite Contributor
role to the Active Directory application identity you created above. - In the command line, type the following command to deploy a template, and enter the required parameters when prompted.
az group deployment create \ --resource-group <new-resource-group-name> \ --template-uri "https://raw.githubusercontent.com/alertlogic/azure-collector/master/template.json"
To verify successful installation of the template:
- In the Azure portal, access
Function Apps
, and then choose the Alert Logic Office 365 collector function. - Click
Functions
->Master
->Monitor
and verify the recent log entry has the status of "OK" and contains no error messages. Example:O365 source checkin OK
. - In the Alert Logic console, navigate to
Configuration
->Deployments
->All Deployments
->Log Sources
, and then filter the list byPush (Office 365, CloudWatch)
collection method. - Verify a new Office 365 log source with the name provided during
az group deployment create
above appears with the source status asOK
.
Note: The following Azure functions use the Application/O365 tenant ID (APP_TENANT_ID
web application setting) as a PublisherIdentifier
during Office 365 management API requests. For more information about PublisherIdentifier
, see Requesting content blobs and throttling.
The Master
function is a timer trigger function responsible for:
- Registering the Azure web app in Alertlogic backend
- Reporting health-checks to the backed
- Performing log source configuration updates, which happen via Alertlogic UI
Note: When you release a new version of the collector, remember to increment the version number in
npm package.json file. To display the current version locally, issue npm run local-version
The Updater
function is a timer triggered function that runs a deployment sync operation every 12 hours to keep web application up to date.
The O365WebHook
function exposes an HTTP API endpoint https://<app-name>/o365/webhook
that is registered as an Office 365 webhook and processes Office 365 activity notifications
Notification Example
[
{
"contentType": "Audit.AzureActiveDirectory",
"contentId": "20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
"contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
"notificationStatus": "Succeeded",
"contentCreated": "2017-07-21T12:16:56.798Z",
"notificationSent": "2017-07-21T12:16:56.798Z",
"contentExpiration": "2017-07-28T12:16:08.709Z"
},
{
"contentType": "Audit.AzureActiveDirectory",
"contentId": "20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
"contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
"notificationStatus": "Succeeded",
"contentCreated": "2017-07-21T12:16:56.798Z",
"notificationSent": "2017-07-21T12:16:56.798Z",
"contentExpiration": "2017-07-28T12:16:25.590Z"
}
]
A notification contains a link to the actual data, which is retrieved by the O365WebHook
, wrapped into a protobuf structure, and then sent to Alert Logic Ingest service.
Note: Audit content may not be available for up to 24 hours. Please follow this link to find the time it takes for the different services in Office 365.
- Clone the repo
git clone git@github.com:alertlogic/azure-collector.git
. cd azure-collector
- Run
./local_dev/setup.sh
. - Edit
./local_dev/dev_config.js
. - Run the Master function locally:
npm run local-master
. - Run the Updater function locally:
npm run local-updater
. - Run the O365WebHook function locally:
npm run local-o365webhook
. - Run
npm test
to perform code analysis and unit tests.
Please use the following code style as much as possible.
process.env.APP_TENANT_ID
- The GUID of the tenant (such asalazurealertlogic.onmicrosoft.com
)process.evn.APP_RESOURCE_GROUP
- The name of the resource group where you deployed your application.process.env.CUSTOMCONNSTR_APP_CLIENT_ID
- The GUID of your application that created the subscription. Note You can obtain this value from Azure -> AD -> App registrations -> Your app nameprocess.env.CUSTOMCONNSTR_APP_CLIENT_SECRET
- The secret key of your application from App Registrations.process.env.CUSTOMCONNSTR_APP_CI_ACCESS_KEY_ID
- The access key returned from AIMs above.process.env.CUSTOMCONNSTR_APP_CI_SECRET_KEY
- The secret key returned from AIMs above.
- Sometimes deployments fail after siteSync action. We need better updater to handle that in order not to wait for 12 hours for the next update attempt.