Azure Monitor Exporter: Telemetry is not exported to Azure Monitor

[akakarikos]

Component(s)

exporter/azuremonitor

What happened?

Description

After upgrading to otel-contrib image version 0.115.1 no telemetry is exported to Azure Monitor (Application Insights).

Steps to Reproduce

Upgrade otel-contrib image version to 0.115.1 and add the exporters/azuremonitor to the logs and traces pipelines.

Expected Result

All telemetry (logs,traces) should be exported to Azure Monitor (Application Insights).

Actual Result

No telemetry is exported/shown in Azure Monitor.

Collector version

0.115.0

Environment information

Environment

OTel Version: 0.115.0.
OTel Contrib Image version: 0.115.1

OpenTelemetry Collector configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  config.yaml: |-
    receivers:
      zipkin:
        endpoint: localhost:9411
      otlp:
        protocols:
          grpc:
            endpoint: localhost:4317
          http:
            endpoint: localhost:4318
            
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 30

      batch:
        send_batch_size: 50
    
    exporters:
      debug:
        verbosity: detailed
    
      azuremonitor:
        instrumentation_key: ${env:OTEL_INSTRUMENTATION_KEY}
        spaneventsenabled: true

      azuredataexplorer:
        cluster_uri: ${env:OTEL_ADX_CLUSTER_URI}
        application_id: ${env:OTEL_APPLICATION_ID}
        application_key: ${env:OTEL_APPLICATION_KEY}
        tenant_id: ${env:OTEL_TENANT_ID}
        db_name: ${env:OTEL_DB_NAME}
        metrics_table_name: ${env:OTEL_METRICS_TABLE_NAME}
        logs_table_name: ${env:OTEL_LOGS_TABLE_NAME}
        traces_table_name: ${env:OTEL_TRACES_TABLE_NAME}
        ingestion_type : 'managed'
    
    service:
      pipelines:
        traces:
          receivers: [zipkin,otlp]
          processors: [memory_limiter,batch]
          exporters: [azuredataexplorer,azuremonitor,debug]
        logs:
          receivers: [otlp]
          processors: [memory_limiter,batch]
          exporters: [azuredataexplorer,azuremonitor,debug]

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/azuremonitor: @pcwiese

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[hgaol]

I can't reproduce and can send data to app insight in the latest main branch. Is there any log?

[akakarikos]

@hgaol I don't get any errors in the logs. When I'm switching to the 0.99.0 version it works fine. Did you try using an OTel Collector configuration similar to mine? Did you use the Instrumentation Key to connecto to the Azure App Insights? Thanks.

[hgaol]

@akakarikos could you try connection string? For instrumentation key, if you didn't provide endpoint, it'll use a default one, which may not be the correct one. While connection string has endpoint info and azuremonitorexporter will parse and use the endpoint in it. BTW, you can enable debug log to see if any abnormal logs.

service:
  pipelines:
    metrics/1:
      receivers: [otlp]
      exporters: [azuremonitor/3]
  telemetry:
    logs:
      level: debug

When I use similar config as yours, I found below error.

2024-12-07T23:26:30.588+0800	debug	azuremonitorexporter/factory.go:128	Response: 404	{"kind": "exporter", "data_type": "metrics", "name": "azuremonitor/3"}
2024-12-07T23:26:30.588+0800	debug	azuremonitorexporter/factory.go:128	Telemetry transmitted in 854.166065ms	{"kind": "exporter", "data_type": "metrics", "name": "azuremonitor/3"}

[akakarikos]

@hgaol you're right, the connection string worked fine. Also, thanks for the tip with the debug logs.

It's a bit weird though because when downgrading to version 0.99.0 it works fine with the instrumentation key. Maybe something has changed in the newer versions and works only with the connection string? If so, I think it should be documented.

[hgaol]

Hi @akakarikos , I found it's truly a bug. I've created a PR to fix it and have validated locally. After PR merged, you can try again with InstrumentationKey.

[akakarikos]

That makes sense. Thanks for the investigation.