[AzureMonitorExporter] Add _OTELRESOURCE_ metrics

[rajkumar-rangaraj]

This pull request updates the exporter to convert all resource attributes to metric telemetry with a name of _OTELRESOURCE_, which now includes all resource attributes as custom dimensions. The _OTELRESOURCE_ metric is included in every transmission to the Application Insights ingestion service, enabling the storage of resource objects in a new AppResources LA table. Below is a sample telemetry payload to demonstrate these changes.

Please note that this is a work in progress and the AppResources table may not be available at this point. As a result, _OTELRESOURCE_ will be available in metrics blade..

{
   "name":"Metric",
   "time":"2023-05-05T20:05:02.6710627Z",
   "iKey":"00000000-0000-0000-0000-000000000000",
   "tags":{
      "ai.cloud.role":"[my-namespace]/my-service",
      "ai.cloud.roleInstance":"my-instance",
      "ai.internal.sdkVersion":"dotnet7.0.5:otel1.4.0:ext1.0.0-alpha.20230505.1"
   },
   "data":{
      "baseType":"MetricData",
      "baseData":{
         "metrics":[
            {
               "name":"_OTELRESOURCE_",
               "value":0
            }
         ],
         "properties":{
            "service.name":"my-service",
            "service.namespace":"my-namespace",
            "service.instance.id":"my-instance",
            "foo":"bar"
         },
         "ver":2
      }
   }
}

The AppResources table can be used with a join operation with other tables in Application Insights. For example, the AppResources table can be joined with the AppRequests table to gain insights into the application's resource usage.

[azure-sdk]

API change check

API changes are not detected in this pull request.

[vishweshbankwar]

LGTM - tests need improvement, that can be done as a separate PR.

[treyjones-ms]

@rajkumar-rangaraj - On first glance, it looks like this one covered everything my PR did (which I'm glad this capability is being addressed, regardless if it's my change or someone else's, so much appreciated) to cover logs, traces, and metrics. Assuming the serviceVersion attribute would get picked up as well? My PR had added a configuration option at @cijothomas request to control whether the custom attributes were picked up or not, but this one does not. However, yours does have a check for maximum length of the attribute key and value, which I didn't consider (or realize was an option). I did notice that there wasn't a CHANGELOG.md update for this PR, so that should probably be addressed.

Did you confirm that those attributes show up in the blades in the Azure portal when looking at the traces in Search Transactions?

[rajkumar-rangaraj]

Thank you for your feedback. Yes, the serviceVersion attribute should also be picked up with this change. Regarding the configuration option for picking up custom attributes, it is a different feature and we need to have a specification from the ingestion to address it. As all other language exporters need to follow the same rules, we can't introduce this feature. Already 2 issues reported for this feature and we are keeping track of it.

Thank you for bringing up the missing CHANGELOG.md update. Created a new PR (#36136) to address this issue.

Did you confirm that those attributes show up in the blades in the Azure portal when looking at the traces in Search Transactions?

These attributes will not be available for lookup in Search Transactions. They will only be available in the metrics blade as custom dimensions. Please note that AppResource is a work in progress and not available now.

[nicolaslattuada]

Hi @rajkumar-rangaraj
Thanks for adding this change. Is it correct to assume that this change will cover the use case, where a user wants to filter metrics recorded for a given deployment.environment? if configured like this:

var resourceBuilder =
    ResourceBuilder
        .CreateDefault()
        .AddService(serviceName: serviceName, serviceVersion: serviceVersion)
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment"] = "production",
        });

If yes, is it already available in 1.0.0-beta.11? and please can you point me to some documentation if any.
Thanks

[rajkumar-rangaraj]

@nicolaslattuada this PR does not cover that scenario. In the current implementation, resources attributes are not added to metric dimensions. We are actively tracking this issue. As a workaround, please use .NET Meter APIs to add custom dimensions.

[nicolaslattuada]

@nicolaslattuada this PR does not cover that scenario. In the current implementation, resources attributes are not added to metric dimensions. We are actively tracking this issue. As a workaround, please use .NET Meter APIs to add custom dimensions.

Thanks, I would rather wait for it to be implemented in the exporter. I am not sure to understand what is the current status though. Is it going to be implemented in #36027?