From aad6c94a615375ebe58051c5a0f324663f2da998 Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 16 Jun 2025 13:30:11 -0400 Subject: [PATCH 1/9] update msentinel shortcode --- .../microsoft_sentinel.md | 65 ++++++++++++------- 1 file changed, 42 insertions(+), 23 deletions(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index e3ca659ac49..e4ddc0468b5 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -1,38 +1,57 @@ #### Prerequisites -To set up the Microsoft Sentinel destination, you need the following information: +To set up the Microsoft Sentinel destination, you need to create a Workspace in Azure if you haven't already. In that workspace: +1. [Add Microsoft Sentinel][10163] to that workspace. +1. [Create a Data Collection Endpoint (DCE)][10164]. +1. [Create a Logs Analytics Workspace][10165] in your workspace if you haven't already. +1. In the Logs Analytics Workspace, navigate to **Settings** > **Tables**. + 1. Click **+ Create**. + 1. Define a custom table (for example, `Custom-MyLogs_CL`). **Note**: The table name must start with `Custom-`. `CL` is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination. + 1. Select **New Custom Log (DCR-based)**. + 1. Click **Create a new data collection rule** and select the DCE you create earlier. + 1. Click **Next**. + 1. Upload a sample JSON Log. For this example, the following JSON is used for the **Schema and Transformation**, where `TimeGenerated` is required: + ```json + {"TimeGenerated":"2024-07-22T11:47:51Z","event": {}} + ``` + 1. Click **Create**. +1. In Azure, navigate to **Microsoft Entra ID**. + 1. Click **Add** > **App Registration**. + 1. Click **Create**. + 1. On the overview page, click **Client credentials: Add a certificate or secret**. + 1. Click **New client secret**. + 1. Enter a name for the secret and click **Add**. + 1. Take note of the **Tenant ID**, **Client ID**, and **Client Secret**. You need this information when you [set up the Observability Pipelines Microsoft Sentinel destination](#set-up-the- + destination-in-observability-pipelines). +1. In Azure Portal's [Data Collection Rules][10166] page, search for and select the DCR you created earlier. + 1. Click **Access Control (IAM)** in the left nav. + 1. Click **Add** and select **Add role assignment**. + 1. Add the **Monitoring Metrics Publisher** role. + 1. On the Members page, select **User, group, or service principal**. + 1. Click **Select Members** and search for the application you created in the app registration step. + 1. Click **Review + Assign**. + +The table below summarizes the Azure and Microsoft Sentinel information you need when you [set up the Observability Pipelines Microsoft Sentinel destination](#set-up-the-destination-in-observability-pipelines): | Name | Description | |------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Application (client) ID | The Azure Active Directory (AD) application's client ID. See [Register an application in Microsoft Entra ID][10161] for information on creating a new application.
**Example**: `550e8400-e29b-41d4-a716-446655440000` | -| Directory (tenant) ID | The Azure AD tenant ID. See [Register an application in Microsoft Entra ID][10161] for information on creating a new application.
**Example**: `72f988bf-86f1-41af-91ab-2d7cd011db47` | -| Table (Stream) Name | The name of the stream which matches the table chosen when configuring the Data Collection Rule (DCR).
**Example**: `Custom-MyLogs_CL` | +| Application (client) ID | The Azure Active Directory (AD) application's client ID. See [Register an application in Microsoft Entra ID][10161] for more information.
**Example**: `550e8400-e29b-41d4-a716-446655440000` | +| Directory (tenant) ID | The Azure AD tenant ID. See [Register an application in Microsoft Entra ID][10161] for more information.
**Example**: `72f988bf-86f1-41af-91ab-2d7cd011db47` | +| Table (Stream) Name | The name of the stream which matches the table chosen when configuring the Data Collection Rule (DCR). **Note**: The table name must start with `Custom-`. `CL` is automatically appended to the end of the table name.
**Example**: `Custom-MyLogs_CL` | | Data Collection Rule (DCR) immutable ID | This is the immutable ID of the DCR where logging routes are defined. It is the **Immutable ID** shown on the DCR Overview page.
**Note**: Ensure the Monitoring Metrics Publisher role is assigned in the DCR IAM settings.
**Example**: `dcr-000a00a000a00000a000000aa000a0aa`
See [Data collection rules (DCRs) in Azure Monitor][10162] to learn more about creating or viewing DCRs. | -Do the following to get that information: - -1. Create or identify a Data Collection Rule (DCR). - 1. In the Azure Portal, navigate to **Azure Monitor** → **Data Collection Rules**. - 1. Create a DCR or use an existing one for custom logs. See [Data collection rules (DCRs) in Azure Monitor][10162] to learn more about creating or viewing DCRs. - 1. Take note of the DCR Immutable ID and, if you are using private links, the DCR's Data Collection Endpoint (DCE). You need this information when you set up the Microsoft Sentinel destination. - 1. Define a custom table (for example, `Custom-MyLogs_CL`) in the DCR, which is where Observability Pipelines sends logs to. -1. Get the ingestion URL. - 1. In the DCR, locate the **Logs Ingestion API endpoint**. The endpoint has the format: `https://.ingest.monitor.azure.com/dataCollectionRules//streams/?api-version=2023-01-01`, where the `` typically matches your custom table (for example, `Custom-MyLogs_CL`). - 1. The ingestion URL is needed when you set up you Microsoft Sentinel destination's environment variable. -1. To authenticate the Observability Pipelines Worker with Microsoft Sentinel: - 1. In the Azure Portal, navigate to **Azure AD** > **App Registrations** and register an Azure Active Directory (AD) application. See [Register an application in Microsoft Entra ID][10161] for information on creating a new application. - 1. Generate a **Client Secret**. - 1. Assign it the **Monitoring Metrics Publisher** role on the Log Analytics workspace - 1. Take note of the **Tenant ID**, **Client ID**, and **Client Secret**. You need this information when you set up the Microsoft Sentinel destination. - -#### Set up destination in Observability Pipelines +#### Set up the destination in Observability Pipelines To set up the Microsoft Sentinel destination in Observability Pipelines: 1. Enter the client ID for your application, such as `550e8400-e29b-41d4-a716-446655440000`. 1. Enter the directory ID for your tenant, such as `72f988bf-86f1-41af-91ab-2d7cd011db47`. This is the Azure AD tenant ID. -1. Enter the name of the table, such as `Custom-MyLogs`, to which you are sending logs. +1. Enter the name of the table to which you are sending logs. An example table name: `Custom-MyLogs_CL`. 1. Enter the Data Collection Rule (DCR) immutable ID, such as `dcr-000a00a000a00000a000000aa000a0aa`. [10161]: https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app?tabs=certificate%2Cexpose-a-web-api -[10162]: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview \ No newline at end of file +[10162]: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview +[10163]: https://portal.azure.com/#browse/microsoft.securityinsightsarg%2Fsentinel +[10164]: https://portal.azure.com/#view/HubsExtension/BrowseResource.ReactView/resourceType/microsoft.insights%2Fdatacollectionendpoints +[10165]: https://portal.azure.com/#create/Microsoft.LogAnalyticsOMS +[10166]: https://portal.azure.com/#view/HubsExtension/BrowseResource.ReactView/resourceType/microsoft.insights%2Fdatacollectionrules \ No newline at end of file From f8ee1ed90018257880841934d5b7cc6b90e948e1 Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 16 Jun 2025 13:33:11 -0400 Subject: [PATCH 2/9] test for fix --- .../destination_settings/microsoft_sentinel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index e4ddc0468b5..3369ddcf147 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -11,7 +11,7 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in 1. Click **Create a new data collection rule** and select the DCE you create earlier. 1. Click **Next**. 1. Upload a sample JSON Log. For this example, the following JSON is used for the **Schema and Transformation**, where `TimeGenerated` is required: - ```json + ``` {"TimeGenerated":"2024-07-22T11:47:51Z","event": {}} ``` 1. Click **Create**. From cdaa9ab2b88a5e167b5cd2b2b645dfdf2c402f0f Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 16 Jun 2025 13:35:07 -0400 Subject: [PATCH 3/9] another test --- .../destination_settings/microsoft_sentinel.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index 3369ddcf147..215adaefcf3 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -11,8 +11,11 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in 1. Click **Create a new data collection rule** and select the DCE you create earlier. 1. Click **Next**. 1. Upload a sample JSON Log. For this example, the following JSON is used for the **Schema and Transformation**, where `TimeGenerated` is required: - ``` - {"TimeGenerated":"2024-07-22T11:47:51Z","event": {}} + ```json + { + "TimeGenerated": "2024-07-22T11:47:51Z", + "event": {} + } ``` 1. Click **Create**. 1. In Azure, navigate to **Microsoft Entra ID**. From 129290db6487e99e3fce2190784187653d703e49 Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 16 Jun 2025 13:45:56 -0400 Subject: [PATCH 4/9] revert fix that didn't work --- .../destination_settings/microsoft_sentinel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index 215adaefcf3..eaf13a8509e 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -11,7 +11,7 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in 1. Click **Create a new data collection rule** and select the DCE you create earlier. 1. Click **Next**. 1. Upload a sample JSON Log. For this example, the following JSON is used for the **Schema and Transformation**, where `TimeGenerated` is required: - ```json + ``` { "TimeGenerated": "2024-07-22T11:47:51Z", "event": {} From 81104643c4ad6f28a4e1f5a61f5ceb9ea0babbc5 Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 16 Jun 2025 16:16:26 -0400 Subject: [PATCH 5/9] small edit --- .../destination_settings/microsoft_sentinel.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index eaf13a8509e..b5d3a7daff9 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -1,9 +1,9 @@ #### Prerequisites To set up the Microsoft Sentinel destination, you need to create a Workspace in Azure if you haven't already. In that workspace: -1. [Add Microsoft Sentinel][10163] to that workspace. +1. [Add Microsoft Sentinel][10163] to the workspace. 1. [Create a Data Collection Endpoint (DCE)][10164]. -1. [Create a Logs Analytics Workspace][10165] in your workspace if you haven't already. +1. [Create a Logs Analytics Workspace][10165] in the workspace if you haven't already. 1. In the Logs Analytics Workspace, navigate to **Settings** > **Tables**. 1. Click **+ Create**. 1. Define a custom table (for example, `Custom-MyLogs_CL`). **Note**: The table name must start with `Custom-`. `CL` is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination. From 7b25c16b66c6b361c62f6b8289e160e707d3eb5b Mon Sep 17 00:00:00 2001 From: May Lee Date: Mon, 30 Jun 2025 14:47:29 -0400 Subject: [PATCH 6/9] apply suggestions --- .../destination_env_vars/microsoft_sentinel.md | 2 +- .../destination_settings/microsoft_sentinel.md | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/microsoft_sentinel.md index 09d31de51ff..d03bc6e582c 100644 --- a/layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/configure_existing_pipelines/destination_env_vars/microsoft_sentinel.md @@ -1,5 +1,5 @@ - Data collection endpoint (DCE) - - The DCE endpoint URL is shown as the **Logs Ingestion Endpoint** or **Data Collection Endpoint** on the DCR Overview page. An example URL: `https://.ingest.monitor.azure.com/dataCollectionRules//streams/?api-version=2023-01-01`. + - The DCE endpoint URL is shown as the **Logs Ingestion Endpoint** or **Data Collection Endpoint** on the DCR Overview page. An example URL: `https://.ingest.monitor.azure.com`. - Stored as the environment variable `DD_OP_DESTINATION_MICROSOFT_SENTINEL_DCE_URI` - Client secret - This is the Azure AD application's client secret, such as `550e8400-e29b-41d4-a716-446655440000`. diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index b5d3a7daff9..4b3dabba9eb 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -6,7 +6,8 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in 1. [Create a Logs Analytics Workspace][10165] in the workspace if you haven't already. 1. In the Logs Analytics Workspace, navigate to **Settings** > **Tables**. 1. Click **+ Create**. - 1. Define a custom table (for example, `Custom-MyLogs_CL`). **Note**: The table name must start with `Custom-`. `CL` is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination. + 1. Define a custom table (for example, `Custom-MyLogs_CL`). + - **Notes**:
- The table name must start with `Custom-`. `CL` is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination.
- You can also use an Azure Table instead of a custom table. 1. Select **New Custom Log (DCR-based)**. 1. Click **Create a new data collection rule** and select the DCE you create earlier. 1. Click **Next**. @@ -23,16 +24,15 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in 1. Click **Create**. 1. On the overview page, click **Client credentials: Add a certificate or secret**. 1. Click **New client secret**. - 1. Enter a name for the secret and click **Add**. - 1. Take note of the **Tenant ID**, **Client ID**, and **Client Secret**. You need this information when you [set up the Observability Pipelines Microsoft Sentinel destination](#set-up-the- - destination-in-observability-pipelines). + 1. Enter a name for the secret and click **Add**. **Note**: Make sure to take note of the client secret, which gets obfuscated after 10 minutes. + 1. Also take note of the **Tenant ID** and **Client ID**. You need this information, along with the client secret, when you [set up the Observability Pipelines Microsoft Sentinel destination](#set-up-the-destination-in-observability-pipelines). 1. In Azure Portal's [Data Collection Rules][10166] page, search for and select the DCR you created earlier. 1. Click **Access Control (IAM)** in the left nav. 1. Click **Add** and select **Add role assignment**. 1. Add the **Monitoring Metrics Publisher** role. 1. On the Members page, select **User, group, or service principal**. 1. Click **Select Members** and search for the application you created in the app registration step. - 1. Click **Review + Assign**. + 1. Click **Review + Assign**. **Note**: It can take up to 10 minutes for the IAM change to take effect. The table below summarizes the Azure and Microsoft Sentinel information you need when you [set up the Observability Pipelines Microsoft Sentinel destination](#set-up-the-destination-in-observability-pipelines): From 8ea7498bd0654bf2c42594373a4ab0fd957166bd Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 3 Jul 2025 11:17:22 -0400 Subject: [PATCH 7/9] Update layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md --- .../destination_settings/microsoft_sentinel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index 4b3dabba9eb..85b4e9a8116 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -7,7 +7,7 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in 1. In the Logs Analytics Workspace, navigate to **Settings** > **Tables**. 1. Click **+ Create**. 1. Define a custom table (for example, `Custom-MyLogs_CL`). - - **Notes**:
- The table name must start with `Custom-`. `CL` is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination.
- You can also use an Azure Table instead of a custom table. + - **Notes**:
- For custom tables, the table name must start with `Custom-`. `CL` is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination.
- You can also use an Azure Table instead of a custom table. 1. Select **New Custom Log (DCR-based)**. 1. Click **Create a new data collection rule** and select the DCE you create earlier. 1. Click **Next**. From 15c2c05c46cbb27e37752b1cc630f9950e3ee0b6 Mon Sep 17 00:00:00 2001 From: May Lee Date: Wed, 9 Jul 2025 13:50:34 -0400 Subject: [PATCH 8/9] Apply suggestions from code review Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com> --- .../destination_settings/microsoft_sentinel.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index 85b4e9a8116..2f32ab95e16 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -3,13 +3,13 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in Azure if you haven't already. In that workspace: 1. [Add Microsoft Sentinel][10163] to the workspace. 1. [Create a Data Collection Endpoint (DCE)][10164]. -1. [Create a Logs Analytics Workspace][10165] in the workspace if you haven't already. -1. In the Logs Analytics Workspace, navigate to **Settings** > **Tables**. +1. [Create a Log Analytics Workspace][10165] in the workspace if you haven't already. +1. In the Log Analytics Workspace, navigate to **Settings** > **Tables**. 1. Click **+ Create**. 1. Define a custom table (for example, `Custom-MyLogs_CL`). - **Notes**:
- For custom tables, the table name must start with `Custom-`. `CL` is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination.
- You can also use an Azure Table instead of a custom table. 1. Select **New Custom Log (DCR-based)**. - 1. Click **Create a new data collection rule** and select the DCE you create earlier. + 1. Click **Create a new data collection rule** and select the DCE you created earlier. 1. Click **Next**. 1. Upload a sample JSON Log. For this example, the following JSON is used for the **Schema and Transformation**, where `TimeGenerated` is required: ``` From f753ba82d0895dd8995cb3be80b34945158b3616 Mon Sep 17 00:00:00 2001 From: May Lee Date: Wed, 9 Jul 2025 13:50:44 -0400 Subject: [PATCH 9/9] Update layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com> --- .../destination_settings/microsoft_sentinel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md index 2f32ab95e16..69afa53e03c 100644 --- a/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md +++ b/layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md @@ -12,7 +12,7 @@ To set up the Microsoft Sentinel destination, you need to create a Workspace in 1. Click **Create a new data collection rule** and select the DCE you created earlier. 1. Click **Next**. 1. Upload a sample JSON Log. For this example, the following JSON is used for the **Schema and Transformation**, where `TimeGenerated` is required: - ``` + ```json { "TimeGenerated": "2024-07-22T11:47:51Z", "event": {}