Merge pull request #6289 from jsubirat/forward_azure_logs_improvements

Update forwarding logs from Azure documentation

src/content/docs/logs/forward-logs/azure-log-forwarding.mdx

@@ -6,23 +6,66 @@ tags:
   - Enable log monitoring in New Relic
   - Azure
   - Cloud logs
+  - EventHub
+  - Event Hub
+  - Blob Storage
 metaDescription: 'Install and configure New Relic logging for Microsoft Azure Resources Manager (ARM), so you can use enhanced log management capabilities.'
 redirects:
   - /docs/logs/enable-log-management-new-relic/enable-log-monitoring-new-relic/azure-log-forwarding
 ---
 
-If your log data is already being monitored by [Microsoft Azure (ARM)](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/overview), you can use our template to forward and enrich your log data in New Relic.
+If your logs are already being collected in Azure, you can use our [Microsoft Azure Resource Manager (ARM)](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/overview) templates to forward and enrich them in New Relic.
 
-Forwarding your Azure EventHub logs to New Relic will give you enhanced log management capabilities to collect, process, explore, query, and alert on your log data.
+Forwarding your Azure logs to New Relic will give you enhanced log management capabilities to collect, process, explore, query, and alert on your log data.
 
-## Azure Resources Manager logs [#requirements]
+We currently offer two ARM templates to achieve this: the EventHub-based (recommended) and the Blob Storage-based templates.
 
-The Microsoft Azure Resources Manager (ARM) template provided by New Relic One helps you set up Azure to:
+## Send logs from an Azure Event Hub (recommended) [#azure-eventhub]
 
-* [Forward logs from EventHub](#application-logs) to New Relic One.
-* [Forward activity logs](#activity-logs) to New Relic One through EventHub.
+The [New Relic Event Hub ARM template](https://github.com/newrelic/newrelic-azure-functions/blob/master/armTemplates/azuredeploy-eventhubforwarder.json) allows you to attach a consumer to an existing or new Event Hub to forward the incoming stream of logs to New Relic. By using this setup, you can configure multiple Azure resources to send their logs to an Event Hub and have these logs automatically forwarded to New Relic. The template also allows you to easily configure your **subscription** Activity Logs to be sent to New Relic.
 
-The setup process is almost the same for both use cases. As part of the setup process, you can select which [Azure activity logs](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log) you want to forward to New Relic, including:
+To send the logs from your Event Hub:
+1. [Deploy](#eventhub-arm-setup) the New Relic Event Hub ARM template.
+2. Optional: [configure your **subscription** Activity Logs to be sent to New Relic](#subscription-activity-logs).
+3. Optional: [configure a given Azure resource to send its individual Activity Logs](#resource-activity-logs).
+4. [Explore your log data](#find-data).
+
+### Deploy the New Relic Event Hub ARM template [#eventhub-arm-setup]
+
+Follow these steps:
+
+1. Make sure you have a [New Relic license key](/docs/apis/intro-apis/new-relic-api-keys/#ingest-license-key).
+2. Log in to **[one.newrelic.com > Logs](https://one.newrelic.com/launcher/logger.log-launcher)** and click **Add more data sources** on the top right of the page.
+3. Under **Log ingestion**, click the `Microsoft Azure Event Hub` tile:
+
+![Microsoft Azure Event Hub data sources button](./images/azure-eventhub-tile.png)
+
+4. Select the account you want to send the logs, and click **Continue**.
+5. Click **Generate API Key** and copy the generated API key.
+6. Click **Deploy to Azure** and a new tab will be open with the ARM template loaded in Azure.
+7. Select the **Resource Group** where you want to create the necessary resources, and a **Region**. Despite not being mandatory, we recommend installing the template in a new resource group, to avoid deleting any of the components it creates accidentally.
+8. In the **New Relic License Key** field, paste the previously copied API key.
+9. Ensure the [New Relic One endpoint](/docs/logs/log-api/introduction-log-api/#endpoint) is set to the one corresponding to your account.
+10. Optional: Set to `true` the [Azure subscription activity logs](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log) you want to forward. See [the subscription information](#subscription-activity-logs) in this document for more details.
+11. Click **Review + create**, review the data you've inserted, and click **Create**.
+
+Note that the template is idempotent. You can start forwarding logs from Event Hub and then rerun the same template to configure [Azure Subscription Activity Logs](#subscription-activity-logs) forwarding by completing step 10.
+
+### Optional: send Azure Activity Logs from your subscription [#subscription-activity-logs]
+
+[Azure Activity Logs](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log) provide:
+
+- More visibility of your Azure resources
+- Activity of the Azure resources
+- Information about performed actions
+- Events and their timestamps
+- The user who performed an action, if applicable
+
+These are all subscription-level events. If you wish to forward Activity Logs from a specific resource instead, please refer to the [Resource Activity Logs information](#resource-activity-logs) in this document.
+
+For more information about the shape of the Activity Logs, see the [Microsoft Azure Activity Log event schema](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log-schema).
+
+The [New Relic Event Hub ARM template deployment](#eventhub-arm-setup) optionally allows you to select which [Azure Activity Logs](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log) you want to forward to New Relic, including:
 
 * Administrative Azure Activity Logs
 * Alert Azure Activity Logs
@@ -33,20 +76,75 @@ The setup process is almost the same for both use cases. As part of the setup pr
 * Security Azure Activity Logs
 * Service Health Azure Activity Logs
 
-## Use the Azure Resource Manager (ARM) template [#using-the-template]
+### Optional: configure an Azure resource to send its Activity Logs [#resource-activity-logs]
+
+By default, this template only configures the function and resources needed to forward logs from an Event Hub to New Relic One. We can also configure the subscription Activity Logs to be forwarded, but there isn't a default log forwarding from your Azure resources. If you want to forward logs from any resource that produces them, you need to configure it by creating a diagnostic setting for the given resource.
+
+For example, if you have a function running on Azure and you want to forward the logs to New Relic One, you'll need to configure a diagnostic setting to forward the logs to Event Hub. For more information, see the [Microsoft documentation to create diagnostic settings for sending platform logs and metrics to different destinations](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings?tabs=CMD).
+
+In the following example, we will demonstrate how to forward the Activity Logs from a Kubernetes Service resource running on Azure.
+
+1. [Deploy the New Relic Event Hub ARM template](#eventhub-arm-setup).
+2. Navigate to your Kubernetes service:
+
+![Kubernetes Service Button](./images/azure-eventhub-k8s-service.png)
+
+3. In the left-hand menu, select **Monitoring` > `Diagnostic Settings**:
+
+![Diagnostic Settings menu](./images/azure-eventhub-diagnosticsettings.png)
+
+4. Click **Add diagnostic setting**:
+
+![Add Diagnostic Setting button](./images/azure-eventhub-add-diagnostic-setting.png)
 
-The template is idempotent. You can start forwarding logs from EventHub and then rerun the same template to configure Azure activity logs forwarding by completing step 10.
+5. Give your new setting a meaningful name:
+
+![Diagnostic setting name](./images/azure-eventhub-diagnosticsetting-name.png)
+
+6. Select the Kubernetes (control pane) logs you want to collect:
+
+![Kubernetes logs](./images/azure-eventhub-diagnostic-logs.png)
+
+7. On the **Destination details**, select **Stream to an event hub**, and configure the `Event hub namespace`, `Event hub name`, and `Event hub policy name`. If you opted to create a new Event Hub and a namespace during the [ARM template deployment](#eventhub-arm-setup), select the following automatically created settings (the namespace name will have a different suffix):
+
+![Destination details settings](./images/azure-eventhub-destination-details-settings.png)
+
+8. Click **Save** to start forwarding your Kubernetes logs to New Relic:
+
+![Diagnostic setting save button](./images/azure-eventhub-save-button.png)
+
+## Send logs from Azure Blob storage [#azure-blob-storage]
+
+[Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) allows you to store massive amounts of unstructured data, including log files. Using the [New Relic Blob Storage ARM template](https://github.com/newrelic/newrelic-azure-functions/blob/master/armTemplates/azuredeploy-blobforwarder.json), you will be able to create a function that forwards the contents of a [container placed in a Storage Account](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#blob-storage-resources).
+
+<Callout variant="important">
+New Relic Blob Storage ARM template deploys a function that forwards all the blob files within the specified container **at their current state**. If any of these blobs is modified thereafter, the **complete** contents of the file will be resent.
+
+This solution aims to forward static blob files to New Relic, and it does not support file tailing. If you need to forward a stream of logs, we recommend to send your application logs to an `Event Hub` and use the [Event Hub based template](#azure-eventhub) instead.
+</Callout>
+
+To send the blobs from a container in your Storage Account, follow these steps:
+
+1. [Deploy](#blobstorage-arm-setup) the New Relic Blob Storage ARM template.
+2. [Explore your log data](#find-data).
+
+### Deploy the New Relic Blob Storage ARM template [#blobstorage-arm-setup]
+
+Follow these steps:
 
 1. Make sure you have a [New Relic license key](/docs/apis/intro-apis/new-relic-api-keys/#ingest-license-key).
 2. Log in to [New Relic Logs](https://one.newrelic.com/launcher/logger.log-launcher) and click **Add more data sources** on the top right of the page.
-3. Under **Log ingestion**, click the ![Microsoft Azure data sources button](./images/azure-tile.png) tile.
+3. Under **Log ingestion**, click the `Microsoft Azure Blob Storage` tile.
+
+![Microsoft Azure Blob Storage data sources button](./images/azure-blobstorage-tile.png)
+
 4. Select the account you want to send the logs, and click **Continue**.
 5. Click **Generate API Key** and copy the generated **API Key**.
 6. Click **Deploy to Azure** and a new tab will be open with the ARM template loaded in Azure.
-7. Select the **Resource Group** where you want to create the necessary resources, and a **Region**.
+7. Select the **Resource Group** where you want to create the necessary resources, and a **Region**. Despite not being mandatory, we recommend installing the template in a new resource group, to avoid deleting any of the components it creates accidentally.
 8. In the **New Relic License Key** field, paste the previously copied **API Key**.
-9. Ensure the New Relic One endpoint set is the one corresponding to your account.
-10. Optional: Set to `true` the [Azure activity logs](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log) you want to forward.
+9. Introduce the names for the storage account and the container that you wish to forward.
+10. Ensure the [New Relic One endpoint](/docs/logs/log-api/introduction-log-api/#endpoint) is set to the one corresponding to your account.
 11. Click **Review + create**, review the data you've inserted, and click **Create**.
 
 ## View log data [#find-data]
@@ -68,24 +166,6 @@ SELECT * FROM Log where plugin.type='azure'
 
 If no data appears after you enable our log management capabilities, follow our [standard log troubleshooting procedures](/docs/logs/log-management/troubleshooting/no-log-data-appears-ui/).
 
-## Send logs from Azure resources [#application-logs]
-
-By default, this template only configures the needed function and resources to forward logs to New Relic One. We can also configure the activity logs to be forwarded, but there isn't a default log forwarding from your Azure resources. If you want to forward logs from any resource that produces them, you need to configure it by creating a diagnostic setting for the given resource.
-
-For example, if you have a function running on Azure and you want to forward the logs to New Relic One, you'll need to configure a diagnostic setting to forward the logs to EventHub. For more information, see the [Microsoft documentation to create diagnostic settings for sendig platform logs and metrics to different destinations](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings?tabs=CMD).
-
-## Azure activity logs [#activity-logs]
-
-Activating the Azure activity logs forwarding is optional. It provides:
-
-- More visibility of your Azure resources
-- Activity of the Azure resources
-- Information about performed actions
-- Events and their timestamps
-- The user who performed an action, if applicable
-
-These logs give your organization more control over the resources. However, be aware of wrong or unintentional changes on your resources and even unexpected actions. For more information about this kind of event, see the [Microsoft Azure Activity Log event schema](https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log-schema).
-
 ## What's next? [#what-next]
 
 Explore logging data across your platform with the [New Relic One UI](/docs/logs/log-management/ui-data/use-logs-ui/).

src/nav/logs.yml

@@ -33,7 +33,7 @@ pages:
         path: /docs/logs/forward-logs/kubernetes-plugin-log-forwarding
       - title: Logstash plugin
         path: /docs/logs/forward-logs/logstash-plugin-log-forwarding
-      - title: Microsoft Azure logs ARM template
+      - title: Microsoft Azure logs ARM templates
         path: /docs/logs/forward-logs/azure-log-forwarding
       - title: Vector plugin
         path: /docs/logs/forward-logs/vector-output-sink-log-forwarding