New-AzDataCollectionRule. does not create DCR endoints (logs& Metrics)

[ChristopheLux]

Description

Hello
Contrary to ARM deployment the DCR logs and metrics endpoints are not generated when created a new DRC with PowerShell

Issue script & Debug output

New-AzDataCollectionRule -ResourceGroupName 'RGxxxxx' -Name 'DCR-ReproTest' -JsonFilePath '/home/azadm/New_DCR_AZPolicyComplianceDetails.json'
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand begin processing with ParameterSet 'GetSingleContext'.
DEBUG: 11:45:31 AM - [ConfigManager] Got [False] from [DisplayBreakingChangeWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:45:31 AM - GetAzureRMContextCommand end processing.
DEBUG: [CmdletBeginProcessing]: Starting command
DEBUG: CmdletBeginProcessing: 
DEBUG: CmdletProcessRecordStart: 
DEBUG: CmdletGetPipeline: 
DEBUG: CmdletBeforeAPICall: 
DEBUG: URLCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: RequestCreated: /subscriptions/yyyyyy/resourceGroups/RGxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01
DEBUG: HeaderParametersAdded: 
DEBUG: BodyContentSet: 
DEBUG: 11:45:31 AM - [ConfigManager] Got nothing from [DisableInstanceDiscovery], Module = [], Cmdlet = []. Returning default value [False].
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/yyyy/resourceGroups/RGxxxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest?api-version=2022-06-01

Headers:
x-ms-unique-id                : 2
x-ms-client-request-id        :yyyyyyyy
CommandName                   : New-AzDataCollectionRule
FullCommandName               : New-AzDataCollectionRule_CreateViaJsonFilePath
ParameterSetName              : __AllParameterSets
User-Agent                    : AzurePowershell/v12.1.0,PSVersion/v7.4.3,Az.DataCollectionRule/5.2.1

Body:
{
  "location": "westeurope",
  "properties": {
    "streamDeclarations": {
      "Custom-Historical_AzPolicyComplianceDetails_CL": {
        "columns": [
          {
            "name": "policyAssignmentId",
            "type": "string"
          },
          {
            "name": "policyDefinitionId",
            "type": "string"
          },
          {
            "name": "policyDefinitionReferenceId",
            "type": "string"
          },
          {
            "name": "policyDefinitionGroupNames",
            "type": "string"
          },
          {
            "name": "policyDefinitionAction",
            "type": "string"
          },
          {
            "name": "numberOfNonCompliantResources",
            "type": "int"
          },
          {
            "name": "numberOfCompliantResources",
            "type": "int"
          },
          {
            "name": "details",
            "type": "dynamic"
          }
        ]
      }
    },
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/xxxxxx/resourceGroups/rgxxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
          "workspaceId": "xxxx",
          "name": "myworkspace"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": [
          "Custom-Historical_AzPolicyComplianceDetails_CL"
        ],
        "destinations": [
          "myworkspace"
        ],
        "transformKql": "source\n| extend TimeGenerated = now()\n",
        "outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
      }
    ]
  }
}


DEBUG: BeforeCall: 
DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
Vary                          : Accept-Encoding
x-ms-ratelimit-remaining-subscription-resource-requests: 149
Request-Context               : appId=cid-v1:x
x-ms-correlation-request-id   : x
x-ms-client-request-id        : x
x-ms-routing-request-id       : WESTEUROPE:xx
x-ms-request-id               : xxxx
api-supported-versions        : 2019-11-01-preview, 2021-04-01, 2021-09-01-preview, 2022-06-01, 2023-03-11, 2024-03-11
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
X-Cache                       : CONFIG_NOCACHE
X-MSEdge-Ref                  : Ref A: xxxx Ref B: xxxx Ref C: 2024-08-01T11:45:31Z
Date                          : Thu, 01 Aug 2024 11:45:33 GMT

Body:
{
  "properties": {
    "immutableId": "dcr-ewfwff3",
    "streamDeclarations": {
      "Custom-Historical_AzPolicyComplianceDetails_CL": {
        "columns": [
          {
            "name": "policyAssignmentId",
            "type": "string"
          },
          {
            "name": "policyDefinitionId",
            "type": "string"
          },
          {
            "name": "policyDefinitionReferenceId",
            "type": "string"
          },
          {
            "name": "policyDefinitionGroupNames",
            "type": "string"
          },
          {
            "name": "policyDefinitionAction",
            "type": "string"
          },
          {
            "name": "numberOfNonCompliantResources",
            "type": "int"
          },
          {
            "name": "numberOfCompliantResources",
            "type": "int"
          },
          {
            "name": "details",
            "type": "dynamic"
          }
        ]
      }
    },
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/9yyyyy/resourceGroups/rg-int-dgs-lab-its-itinfra-1/providers/microsoft.operationalinsights/workspaces/policyworkspace",
          "workspaceId": "zzzzzz",
          "name": "myworkspace"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": [
          "Custom-Historical_AzPolicyComplianceDetails_CL"
        ],
        "destinations": [
          "myworkspace"
        ],
        "transformKql": "source\n| extend TimeGenerated = now()\n",
        "outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
      }
    ],
    "provisioningState": "Succeeded"
  },
  "location": "westeurope",
  "id": "/subscriptions/yyyyy/resourceGroups/RGxxxxx/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest",
  "name": "DCR-ReproTest",
  "type": "Microsoft.Insights/dataCollectionRules",
  "etag": "\"b6009126-0000-0d00-0000-66ab755d0000\"",
  "systemData": {
    "createdBy": "xxxxx",
    "createdByType": "User",
    "createdAt": "2024-08-01T11:45:31.8835297Z",
    "lastModifiedBy": "xxxxx",
    "lastModifiedByType": "User",
    "lastModifiedAt": "2024-08-01T11:45:31.8835297Z"
  }
}


DEBUG: ResponseCreated: 
DEBUG: BeforeResponseDispatch: 
DEBUG: Finally: 
DEBUG: CmdletAfterAPICall: 
DEBUG: [CmdletProcessRecordAsyncEnd]: Finish HTTP process
DEBUG: CmdletProcessRecordAsyncEnd: 
DEBUG: CmdletProcessRecordEnd: 
DEBUG: 11:45:33 AM - [ConfigManager] Got [True] from [DisplaySecretsWarning], Module = [], Cmdlet = [].

DataCollectionEndpointId                  : 
DataFlow                                  : {{
                                              "streams": [ "Custom-Historical_AzPolicyComplianceDetails_CL" ],
                                              "destinations": [ "myworkspace" ],
                                              "transformKql": "source\n| extend TimeGenerated = now()\n",
                                              "outputStream": "Custom-Historical_AzPolicyComplianceDetails_CL"
                                            }}
DataSourceDataImportEventHubConsumerGroup : 
DataSourceDataImportEventHubName          : 
DataSourceDataImportEventHubStream        : 
DataSourceExtension                       : 
DataSourceIisLog                          : 
DataSourceLogFile                         : 
DataSourcePerformanceCounter              : 
DataSourcePlatformTelemetry               : 
DataSourcePrometheusForwarder             : 
DataSourceSyslog                          : 
DataSourceWindowsEventLog                 : 
DataSourceWindowsFirewallLog              : 
Description                               : 
DestinationAzureMonitorMetricName         : 
DestinationEventHub                       : 
DestinationEventHubsDirect                : 
DestinationLogAnalytic                    : {{
                                              "workspaceResourceId": "/subscriptions/xxxxxxx/resourceGroups/rgxxxxxx/providers/microsoft.operationalinsights/workspaces/policyworkspace",
                                              "workspaceId": "4yyyyy",
                                              "name": "myworkspace"
                                            }}
DestinationMonitoringAccount              : 
DestinationStorageAccount                 : 
DestinationStorageBlobsDirect             : 
DestinationStorageTablesDirect            : 
Etag                                      : "b6009126-0000-0d00-0000-66ab755d0000"
Id                                        : /subscriptions/9XXXXX/resourceGroups/RGXXXXXX1/providers/Microsoft.Insights/dataCollectionRules/DCR-ReproTest
IdentityPrincipalId                       : 
IdentityTenantId                          : 
IdentityType                              : 
IdentityUserAssignedIdentity              : {
                                            }
ImmutableId                               : dcr-2e40a7469fXXXXXX
Kind                                      : 
Location                                  : westeurope
MetadataProvisionedBy                     : 
MetadataProvisionedByResourceId           : 
Name                                      : DCR-ReproTest
ProvisioningState                         : Succeeded
ResourceGroupName                         : RGXXXX
StreamDeclaration                         : {
                                              "Custom-Historical_AzPolicyComplianceDetails_CL": {
                                                "columns": [
                                                  {
                                                    "name": "policyAssignmentId",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionId",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionReferenceId",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionGroupNames",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "policyDefinitionAction",
                                                    "type": "string"
                                                  },
                                                  {
                                                    "name": "numberOfNonCompliantResources",
                                                    "type": "int"
                                                  },
                                                  {
                                                    "name": "numberOfCompliantResources",
                                                    "type": "int"
                                                  },
                                                  {
                                                    "name": "details",
                                                    "type": "dynamic"
                                                  }
                                                ]
                                              }
                                            }
SystemDataCreatedAt                       : 8/1/2024 11:45:31 AM
SystemDataCreatedBy                       : xxxx
SystemDataCreatedByType                   : User
SystemDataLastModifiedAt                  : 8/1/2024 11:45:31 AM
SystemDataLastModifiedBy                  : xxx
SystemDataLastModifiedByType              : User
Tag                                       : {
                                            }
Type                                      : Microsoft.Insights/dataCollectionRules

DEBUG: AzureQoSEvent:  Module: Az.Monitor:5.2.1; CommandName: New-AzDataCollectionRule; PSVersion: 7.4.3; IsSuccess: True; Duration: 00:00:02.5406602; SanitizeDuration: 00:00:00.0186410

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.3
PSEdition                      Core
GitCommitId                    7.4.3
OS                             CBL-Mariner/Linux
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     3.0.2                 Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     8.0.0                 Az.Compute                          {Add-AzImageDataDisk, Add-AzVhd, Add-AzVMAdditionalUnattendContent, Add-AzVMDataDisk…}
Script     7.8.0                 Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Add-AzApplicationGatewayBackendAddressPool, Add-AzApplicationGatewayBackendHttpSetting, Add-AzApplicationGatewayBackendSetting…}
Script     7.1.0                 Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script     7.0.0                 Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, Close-AzStorageFileHandle…}
Script     1.1.3                 Az.Tools.Predictor                  {Disable-AzPredictor, Enable-AzPredictor, Open-AzPredictorSurvey, Send-AzPredictorRating}
Script     0.0.0.10              AzureAD.Standard.Preview            {Add-AzureADApplicationOwner, Add-AzureADDeviceRegisteredOwner, Add-AzureADDeviceRegisteredUser, Add-AzureADDirectoryRoleMember…}
Script     0.9.3                 AzurePSDrive

Error output

No response

[JustinGrote]

Ran into this as well today, I'm pretty sure because it is still using the 2022-06-01 as seen in the trace, and needs to be bumped to the 2023-03-11 api version
https://learn.microsoft.com/en-us/rest/api/monitor/data-collection-rules/create?view=rest-monitor-2023-03-11&tabs=HTTP

[ChristopheLux]

Is there any way we can force the API version except going again to Invoke-RestMethod...

[JustinGrote]

@ChristopheLux I made a custom one that uses the newer API version and I still didn't see the endpoint getting populated, so I'm not sure what's going on, I was going to open a ticket and report back.

EDIT: https://gist.github.com/JustinGrote/22c4963f7eb5af08399c26cbf60bc3ae

[JustinGrote]

OK, I think I figured it out.

As of the API spec, there is an ingestion endpoint example where you have to specify the kind as "Direct", note there is a typo, there's an extraneous space in this.

https://learn.microsoft.com/en-us/rest/api/monitor/data-collection-rules/create?view=rest-monitor-2023-03-11&tabs=HTTP#create-or-update-data-collection-rule-with-embedded-ingestion-endpoints

Even though the specs for the Kind parameter say only Windows and Linux are supported values.

I updated my script to specify the kind as Direct, and now I got ingestionEndpoints populated.

    "endpoints": {
      "logsIngestion": "https://xxxx-westus3.logs.z1.ingest.monitor.azure.com",
      "metricsIngestion": "https://xxx-westus3.metrics.z1.ingest.monitor.azure.com"
    },

[JustinGrote]

With some more experimentation in regards to Kind, by supplying invalid data to the API, I get back an error that says these are the actual valid values:
Direct,Linux,Windows,WorkspaceTransforms,AgentDirectToStore,AgentSettings,PlatformTelemetry

These appear to be undocumented with a quick google search other than Linux and Windows, these do come back via the 2022 API,

and a test of the 2022 API with Direct does populate the endpoints it seems (there's a significant delay, it's not immediate, some sort of provisioning delay) EDIT: Later testing shows this is not the case

and the endpoints can ONLY be seen with the 2023 API.

[ChristopheLux]

Very nice job...I wasn't able to work on this today

[ChristopheLux]

@JustinGrote
in the documentation for the PowerShell there is https://learn.microsoft.com/en-us/powershell/module/az.monitor/new-azdatacollectionrule?view=azps-12.1.0 the -Kind.
Stupid me

[JustinGrote]

@JustinGrote in the documentation for the PowerShell there is https://learn.microsoft.com/en-us/powershell/module/az.monitor/new-azdatacollectionrule?view=azps-12.1.0 the -Kind. Stupid me

yeah but in my initial testing it doesn't seem to populate endpoints unless the API version is 2023 for the PUT, I'm testing that now.

[JustinGrote]

OK based on this testing with my custom cmdlet:

New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2023Direct' -ApiVersion '2023-03-11'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022Direct' -ApiVersion '2022-06-01'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022DirectReplace' -ApiVersion '2022-06-01'
New-JAzDataCollectionRule @testDcrParams -Name 'TestRule2022DirectReplace' -ApiVersion '2023-03-11' #Overwrites previous

#Additional custom attempt of 2023 API but with Kind not specified at all.

Findings

• You must use 2023 API, no 2022 attempts caused the endpoints to populate
• You can use PUT or PATCH to update a 2022 API to 2023 and as long as the Kind is Direct, it will get the endpoint property
• You must query using 2023 to see the endpoints property, doesn't exist in 2022
• Leaving the kind property unspecified does NOT populate the endpoints even with 2023, it looks like you have to use Kind: Direct

Pretty annoying the DCR docs don't mention that Kind: Direct is required

So currently getting DCR endpoints populated is not possible until the API rev gets bumped on this command, you have to use my custom workaround script. I'll updated it and relink
#25727 (comment)

[JustinGrote]

@isra-fel the DataCollectionRule.Autorest needs a bump to 2023-03-11 to resolve this issue.

[isra-fel]

Great findings 👍
Will plan and prioritize this