diff --git a/content/en/integrations/guide/azure-troubleshooting.md b/content/en/integrations/guide/azure-troubleshooting.md index 34d2b9065d9..cf9541760e3 100644 --- a/content/en/integrations/guide/azure-troubleshooting.md +++ b/content/en/integrations/guide/azure-troubleshooting.md @@ -52,25 +52,15 @@ Turning on Diagnostics allows ARM deployed VMs to collect logging information wh If you have Azure resources with the same resource name as one of the default parameters, it can lead to naming conflicts. Azure does not allow resources to share resource names within an individual subscription. Datadog recommends renaming the default parameter with a unique name that does not already exist within your environment. -For example, use the -EventhubName flag to change the default name of the Eventhub resource, if you already possess an Eventhub named `datadog-eventhub`. - -{{< code-block lang="powershell" filename="Example" >}} - -./resource_deploy.ps1 -ApiKey -SubscriptionId -EventhubName - -{{< /code-block >}} - -**Note:** Navigate to the [Optional Parameters][4] section to find the list of configurable parameters. - -**Note:** If you are re-running the script due to this failure, it is also advised that you remove the entire resource group to create a fresh execution. +**Note:** If you are re-running the template due to this failure, it is also advised that you remove the entire resource group to create a fresh deployment. ### Unregistered resource provider -If your script execution is failing due to the error **The subscription is not registered to use namespace 'Microsoft.EventHub'**: +If your template deployment is failing due to the error **The subscription is not registered to use namespace 'Microsoft.EventHub'**: Azure has resource providers for each of its services, for example: `Microsoft.EventHub` for the Azure EventHub. If your Azure subscription is not registered to a required resource provider the script fails. You can fix this issue by registering with the resource provider. Run this command in CloudShell. -{{< code-block lang="powershell" filename="Example" >}} +{{< code-block lang="shell" filename="Example" >}} az provider register --namespace Microsoft.EventHub @@ -80,7 +70,7 @@ az provider register --namespace Microsoft.EventHub Did you install the script successfully, but you are still not seeing activity/platform logs within the Logs Explorer? -Ensure that you have not exceeded your [daily quota][5] for log retention. +Ensure that you have not exceeded your [daily quota][4] for log retention. **Note:** It is advised that you take at least five minutes after the execution of the script to start looking for logs in the Logs Explorer. @@ -91,5 +81,4 @@ Ensure that you have not exceeded your [daily quota][5] for log retention. [1]: https://portal.azure.com [2]: https://manage.windowsazure.com [3]: /help/ -[4]: /integrations/azure/?tab=azurecliv20#optional-parameters -[5]: /logs/indexes/#set-daily-quota +[4]: /logs/indexes/#set-daily-quota diff --git a/content/en/logs/guide/azure-logging-guide.md b/content/en/logs/guide/azure-logging-guide.md index df56b5d6ec3..e1cf0732f30 100644 --- a/content/en/logs/guide/azure-logging-guide.md +++ b/content/en/logs/guide/azure-logging-guide.md @@ -26,264 +26,34 @@ Follow these steps to send Azure logs to any Datadog site. **US3**: Organizations on the Datadog US3 site can simplify Azure log forwarding using the Azure Native integration. This method is recommended and is configured through the [Datadog resource in Azure][5], replacing the Azure Event Hub process. See the [Azure Native Logging Guide][4] for more details.
-Starting April 30, 2025, Azure no longer supports Node.js 18. To ensure compatibility, first update your forwarder code, then upgrade to the latest Azure LTS version of Node.js (20). -If you previously deployed with an ARM template, you can update using the template with the same parameters. +Starting April 30, 2025, Azure no longer supports Node.js 18. To ensure compatibility, update using the Azure Resource Manager (ARM) template with the same parameters.
## Setup {{< tabs >}} -{{% tab "Automated installation" %}} +{{% tab "Event Hub" %}} -To get started, click the button below and fill in the form on Azure Portal. The Azure resources required to get activity logs streaming into your Datadog account will be deployed for you. +To get started, click the button below and fill in the form on Azure Portal. The Azure resources required to get activity logs streaming into your Datadog account will be deployed for you. To forward Activity Logs, set the **Send Activity Logs** option to true. -[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FDataDog%2Fdatadog-serverless-functions%2Fmaster%2Fazure%2Fdeploy-to-azure%2Fparent_template.json) - -Alternatively, Datadog provides automated scripts you can use for sending Azure activity logs and Azure platform logs (including resource logs). - -### Azure activity logs - -Follow these steps to run the script that creates and configures the Azure resources required to stream activity logs into your Datadog account. These resources include activity log diagnostic settings, Azure Functions, Event Hub namespaces, and Event Hubs. - -1. In the Azure portal, navigate to your **Cloud Shell**. - {{< img src="integrations/azure/azure_cloud_shell.png" alt="azure cloud shell" popup="true" style="width:100%">}} -2. Run the command below to download the automation script into your Cloud Shell environment. You can also [view the contents of the script][100]. - -{{< code-block lang="powershell" filename="Activity Logs Step 1" >}} -(New-Object System.Net.WebClient).DownloadFile("https://raw.githubusercontent.com/DataDog/datadog-serverless-functions/master/azure/eventhub_log_forwarder/activity_logs_deploy.ps1", "activity_logs_deploy.ps1") -{{< /code-block >}} - -3. Invoke the script by running the command below, while replacing **``**, with your [Datadog API token][101], and **``**, with your Azure Subscription ID. Add [Optional Parameters](#optional-parameters) to configure your deployment. - -{{< code-block lang="powershell" filename="Activity Logs Step 2" >}} -./activity_logs_deploy.ps1 -ApiKey -SubscriptionId -{{< /code-block >}} +[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FDataDog%2Fdatadog-serverless-functions%2Frefs%2Fheads%2Fmaster%2Fazure%2Feventhub_log_forwarder%2Fparent_template.json) ### Azure platform logs -To send Azure platform logs (including resource logs), you can deploy an Event Hub and log forwarder function pair. -After deploying, create diagnostic settings for each of the log sources to stream logs to Datadog. - -**Note**: Resources can only stream to Event Hubs in the same Azure region. - -1. In the Azure portal, navigate to your **Cloud Shell**. - -2. Run the PowerShell command below to download the automation script into your Cloud Shell environment. You can also [view the contents of the script][102]. - -{{< code-block lang="powershell" filename="Platform Logs Step 1" >}} -(New-Object System.Net.WebClient).DownloadFile("https://raw.githubusercontent.com/DataDog/datadog-serverless-functions/master/azure/eventhub_log_forwarder/resource_deploy.ps1", "resource_deploy.ps1") -{{< /code-block >}} - -3. Invoke the script by running the PowerShell command below, replacing **``**, with your [Datadog API token][101], and **``**, with your Azure Subscription ID. You can also add other optional parameters to configure your deployment. See [Optional Parameters](#optional-parameters). - -{{< code-block lang="powershell" filename="Platform Logs Step 2" >}} -./resource_deploy.ps1 -ApiKey -SubscriptionId -{{< /code-block >}} - -4. Create diagnostic settings for all Azure resources sending logs to Datadog. Configure these diagnostic settings to stream to the Event Hub you just created. - -All of the Azure resources deployed for the Platform Logs pipeline contain its ResourceGroup-Location appended to its default name. For example, `datadog-eventhub-westus`. However, you can alter this convention by overriding the parameter. - -**Note**: Resources can only stream to Event Hubs in the same Azure region, so you need to replicate step 2 for each region you want to stream resource logs from. - -### Set up both activity and resource logs - -To stream both activity logs and resource logs, run the first script including the optional parameter `-ResourceGroupLocation `. Activity logs are a subscription-level source, so you can create your pipeline for them in any region. Once this is deployed, send resource logs through the same Event Hub by adding diagnostic settings on your resources in `westus`. - -**Note**: This integration does not collect events. - -### Optional parameters - -**Note**: Ensure that your custom resource names are unique when you customize the following parameters. Validate that the resource name does not already exist within your list of other Azure resources. - -| -Flag `` | Description | -|---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| -DatadogSite `` | Customize your Datadog instance by adding this flag with another Datadog site as a parameter. Your Datadog site is: {{< region-param key="dd_site" code="true" >}}. | -| -Environment `` | Manage storage in Azure independent clouds by adding this flag as a parameter. Additional options are `AzureChinaCloud`, `AzureGermanCloud`, and `AzureUSGovernment`. | -| -ResourceGroupLocation `` | You can choose the region in which your Azure resource group and resources are getting deployed by adding this flag with an updated Azure region. | -| -ResourceGroupName `` | Customize the name of your Azure resource group by adding this flag with an updated parameter. | -| -EventhubNamespace `` | Customize your Azure Event Hub namespace by adding this flag with an updated parameter. By default, `datadog-ns-` is generated. | -| -EventhubName `` | Customize the name of your Azure Event Hub by adding this flag with an updated parameter. | -| -FunctionAppName `` | Customize the name of your Azure function app by adding this flag with an updated parameter. By default, `datadog-functionapp-` is generated. | -| -FunctionName `` | Customize the name of your Azure Function by adding this flag with an updated parameter. | -| -DiagnosticSettingName `` | Customize the name of your Azure diagnostic setting by adding this flag with an updated parameter. **(Only relevant for sending activity logs)** | - -Installation errors? See [Automated log collection][103] for common error cases. - - -[100]: https://github.com/DataDog/datadog-serverless-functions/blob/master/azure/eventhub_log_forwarder/activity_logs_deploy.ps1 -[101]: https://app.datadoghq.com/organization-settings/api-keys -[102]: https://github.com/DataDog/datadog-serverless-functions/blob/master/azure/eventhub_log_forwarder/resource_deploy.ps1 -[103]: /integrations/guide/azure-troubleshooting/#automated-log-collection - -{{% /tab %}} - -{{% tab "Manual installation" %}} - -This section describes the manual setup process to forward your Azure logs to Datadog: - -1. Create an [Azure Event Hub](#create-an-azure-event-hub). -2. Set up the [Datadog-Azure function with an Event hub trigger](#create-the-datadog-azure-function) to forward logs to Datadog. -3. Create [diagnostic settings](#create-diagnostic-settings) to forward your Azure [Activity logs](#activity-logs), [resource logs](#resource-logs), or both to your Event Hub. - -The instructions below walk through a basic, initial setup using the Azure Portal. All of these steps can be performed with the CLI, PowerShell, or resource templates by referring to the Azure documentation. +After the template deployment finishes, set up diagnostic settings for each log source to send Azure platform logs (including resource logs) to the Event Hub created during deployment. **Note**: Resources can only stream to Event Hubs in the same Azure region. -#### Create an Azure Event Hub +If you run into any problems during deployment, see [Automated log collection][100] for common error cases. -##### Create an Event Hubs namespace -If you already have an Event Hubs namespace configured with an Event Hub connection string, skip to [Add an Event Hub to your Event Hubs namespace](#add-an-event-hub-to-your-event-hubs-namespace). - -1. In the Azure portal, navigate to the [Event Hubs][208] overview and click **Create**. -2. Fill in the **Project Details** and **Instance Details** sections as desired. - **Note**: If you plan to collect [Azure resource logs][209], the Event Hub must be in the same **Location** as the resource you want to collect logs from. For activity logs or other account-wide log sources, you can choose any region. -3. Click **Review + create** to validate the resource. If validation is successful, click **Create**. - -See the [Azure Event Hubs Quickstart][201] for additional information. - -##### Add an Event Hub to your Event Hubs namespace - -1. In the Azure portal, navigate to your new or existing Event Hubs namespace. -2. Click **+ Event Hub**. -3. Configure the **Basics** and **Capture** tabs as desired. -4. Click **Review + create** to validate the resource. If validation is successful, click **Create**. - -##### Configure shared access - -1. In the detail page of your Event Hub, click **Shared access policies** under the **Settings** tab to the left. -2. Click **+ Add**. -3. Provide a policy name and select **Listen**. -4. Copy the **Connection string-primary key** value and keep it somewhere safe. This is needed to allow the Datadog-Azure function to communicate with the Event Hub. - -{{< img src="integrations/azure/eventhub_connection_string.png" alt="The connection string primary-key value of an event hub's shared access policy" popup="true" style="width:100%">}} - -#### Create the Datadog-Azure function - -##### Create a function app - -If you already have a function app configured with an Event Hub connection string, skip to [Add a new function to your Function App using the Event Hub trigger template](#add-a-new-function-to-your-function-app-using-the-event-hub-trigger-template). - -1. In the Azure portal, navigate to the [Function App overview][211] and click **Create**. -2. In the **Instance Details** section, configure the following settings: - a. Select the **Code** radio button - b. For **Runtime stack**, select `Node.js` - c. For **Version**, select `20 LTS`. -3. Configure other settings as desired. -4. Click **Review + create** to validate the resource. If validation is successful, click **Create**. - -See [Azure Event Hubs trigger for Azure Functions][202] for more information. - -##### Configure your function app with the Event Hub connection string - -1. In the detail page of your function app, click **Environment variables** under the **Settings** tab to the left. -2. In the **App settings** tab, provide a name for the connection string. -3. Paste the value obtained earlier from the [Configure shared access section](#configure-shared-access). -4. Click **Apply**. - -**Note**: If you don't want to paste your Datadog API key value directly into the function's code, create an additional environment variable for the Datadog API key value. - -##### Add a new function to your Function App using the Event Hub trigger template - -1. Select your new or existing function app from the [Function App overview][211]. -2. Under the **Functions** tab, click **Create**. -3. For the **Development environment** field, select **Develop in portal**. -3. Under **Select a template**, choose [Azure Event Hub trigger][202]. -4. Under **Event Hub connection**, select your namespace and Event Hub. -5. Click **Create**. - -See [Getting started with Azure functions][215] for more information. - -##### Point your Event Hub trigger to Datadog - -1. On the detail page of your Event Hub trigger function, click **Code + Test** under the **Developer** side menu. -2. Add the [Datadog-Azure Function code][204] to the function's `index.js` file. -3. Add your Datadog API key through a `DD_API_KEY` environment variable, or copy it into the function code by replacing `` on line 21. -4. If you're not using the Datadog US1 site, set your [Datadog site][207] with a `DD_SITE` environment variable under the configuration tab of your function app, or copy the site parameter into the function code on line 22. -5. **Save** the function. -6. Click **Integration** under the **Developer** side menu. -7. Click **Azure Event Hubs** under **Trigger and inputs**. -8. Confirm the following settings are in place: - a. **Event hub connection** is set to the name of your connection string environment variable. - b. **Event parameter name** is set to `eventHubMessages`. - c. **Event hub name** is set to the name of your Event Hub. - d. **Event hub cardinality** is set to `Many`. - e. **Event hub data type** is left empty. -9. To validate your setup, click **Code + Test** under the **Developer** side menu. -10. Click **Test/Run** and enter a test message in valid JSON format. -11. Find your test message in the [Datadog Log Explorer][206]. - -#### Create diagnostic settings - -##### Activity logs - -1. In the Azure portal, navigate to the [Activity log][212]. -2. Click **Export Activity Logs**. -3. Click **+ Add diagnostic setting**. -4. Under **Categories**, select the categories of logs you want to send to Datadog. -5. Under **Destination details**, select **Stream to an event hub**. -6. Set the **Event hub namespace** and **Event hub name** with the names of the Event Hub namespace and Event Hub name, respectively, that were used to create your Event Hub trigger. -7. For **Event hub policy name**, you can select `RootManageSharedAccessKey` if desired. **Optionally**, create your own shared access policy at the Event Hub **namespace** level: - a. In the Event Hub **namespace**, click **Shared access policies** under the **Settings** tab to the left. - b. Click **+ Add**. - c. Provide a policy name and select **Send** or **Manage**. - d. Click **Save**. - e. Return to the diagnostic setting page and select your shared access policy for the **Event hub policy name** field. You may need to refresh the page. - **Note**: See [Authorizing access to Event Hubs resources using Shared Access Signatures][214] for more information. -8. Verify your setup is correct by checking the [Datadog Log Explorer][206] for your activity logs. - -See [Diagnostic settings in Azure monitor][213] for more information. - -##### Resource logs - -Configure your Azure resources to forward their logs to the Event Hub with a [diagnostic setting][203]. - -1. In the Azure portal, navigate to the resource that you want to forward logs to Datadog. -2. In the **Monitoring** section of the resource blade, click **Diagnostic settings**. -3. Click **Add diagnostic setting**. -4. Provide a name and select the sources of the data you want to forward.. -5. Under **Destination details**, select **Stream to an event hub**. -6. Set the **Event hub namespace** and **Event hub name** with the names of the Event Hub namespace and Event Hub name, respectively, that were used to create your Event Hub trigger. -7. For **Event hub policy name**, you can select `RootManageSharedAccessKey` if desired. **Optionally**, create your own shared access policy at the Event Hub **namespace** level: - a. In the Event Hub **namespace**, click **Shared access policies** under the **Settings** tab to the left. - b. Click **+ Add**. - c. Provide a policy name and select **Send** or **Manage**. - d. Click **Save**. - e. Return to the diagnostic setting page and select your shared access policy for the **Event hub policy name** field. You may need to refresh the page. - **Note**: See [Authorizing access to Event Hubs resources using Shared Access Signatures][214] for more information. -8. Click **Save**. -9. Verify your setup is correct by checking the [Datadog Log Explorer][206] for logs from this resource. - -See [Diagnostic settings in Azure monitor][213] for more information. - -[201]: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create -[202]: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger -[203]: https://docs.microsoft.com/en-us/azure/azure-monitor/platform/diagnostic-settings -[204]: https://github.com/DataDog/datadog-serverless-functions/blob/master/azure/activity_logs_monitoring/index.js -[205]: https://app.datadoghq.com/organization-settings/api-keys -[206]: https://app.datadoghq.com/logs -[207]: https://docs.datadoghq.com/getting_started/site/ -[208]: https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.EventHub%2Fnamespaces -[209]: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/tutorial-resource-logs -[210]: https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-azure-function -[211]: https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Web%2Fsites/kind/functionapp -[212]: https://portal.azure.com/#view/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/~/activityLog -[213]: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings?WT.mc_id=Portal-Microsoft_Azure_Monitoring -[214]: https://learn.microsoft.com/en-us/azure/event-hubs/authorize-access-shared-access-signature -[215]: https://learn.microsoft.com/en-us/azure/azure-functions/functions-get-started +[100]: /integrations/guide/azure-troubleshooting/#automated-log-collection {{% /tab %}} {{% tab "Blob Storage" %}} -{{% site-region region="us5,gov,ap1" %}} -
- Log forwarding from Azure Blob Storage is not supported for your selected Datadog site ({{< region-param key="dd_site_name" >}}). -
-{{% /site-region %}} - Datadog recommends using the Event Hub setup for Azure log collection. However, you can also follow the steps in this section to forward all of your Azure App Services logs from Azure Blob Storage: 1. If you haven't already set up [Azure Blob Storage][301], use one of the following methods to get started: @@ -358,13 +128,10 @@ PCI DSS compliance for APM and Log Management is only available for Datadog orga To set up PCI-compliant Log Management, you must meet the requirements outlined in [PCI DSS Compliance][6]. Send your logs to the dedicated PCI compliant endpoint: -- `agent-http-intake-pci.logs.datadoghq.com:443` for Agent traffic -- `http-intake-pci.logs.datadoghq.com:443` for non-Agent traffic -``` -const DD_SITE = process.env.DD_SITE || 'datadoghq.com'; -const DD_HTTP_URL = process.env.DD_URL || 'http-intake-pci.logs.' + DD_SITE; -``` +Under **Settings > Environment variables**, click **Add** to set the following environment variable: +- Name: `DD_URL` +- Value: `http-intake-pci.logs.datadoghq.com` ## Log Archiving