Too much traces in AppInsights

[WaterKnight1998]

I have the following function code:

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

logging.getLogger().handlers = []

configure_azure_monitor(connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
tracer = trace.get_tracer(__name__)

app = fastapi.FastAPI()

@app.get("/x")
async def x():
   with tracer.start_as_current_span("aaaaa"):
      logging.info("a",extras={"b":1})

def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
    return func.AsgiMiddleware(app).handle(req, context)

With this code I removed the duplicated logs and I can see custom dimensions. However I am seeing 3 taces each 5s creating too much logs:

Request URL: 'https://westeurope-5.in.applicationinsights.azure.com//v2.1/track' Request method: 'POST' Request header
Response status: 200 Response headers: 'Transfer-Encoding': 'chunked' 'Content-Type': 'application/json; charset=utf-8
Transmission succeeded: Item received: 3. Items accepted: 3

I tried the following env vars:

OTEL_PYTHON_EXCLUDED_URLS=.*applicationinsights\.azure\.com/.*
OTEL_PYTHON_REQUESTS_EXCLUDED_URLS=.*applicationinsights\.azure\.com/.*
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=django,asgi,wsgi,fastapi,flask,requests,urllib,urllib3
OTEL_TRACES_EXPORTER=none

However, I am still seeing this. I have seen that your sdk calls this endpoint. https://github.com/Azure/azure-sdk-for-python/blob/11437bbf7be952401581641f556ac8c897667e59/sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/_generated/operations/_azure_monitor_client_operations.py#L39

But i don't know which library you use for http requests, probably this one: https://github.com/Azure/azure-sdk-for-python/blob/11437bbf7be952401581641f556ac8c897667e59/sdk/core/azure-core/azure/core/utils/_pipeline_transport_rest_shared.py#L27.

Any idea on how to disable this logs being sent to AI?

[WaterKnight1998]

I have seen that those logs come from:

Request URL: 'https://westeurope-5.in.applicationinsights.azure.com//v2.1/track'
Request method: 'POST'
Request headers:
'Content-Type': 'application/json'
'Content-Length': '12891'
'Accept': 'application/json'
'User-Agent': 'azsdk-python-azuremonitorclient/unknown Python/3.9.17 (Linux-5.15.116.1-1.cm2-x86_64-with-glibc2.31)'
A body is sent with the request

sdkVersion
py3.9.17:otel1.20.0:ext1.0.0b17

Same as the explicit log that I do.

[WaterKnight1998]

I have tried raising logs to:

logging.basicConfig(level=logging.WARNING)

But I still see them

[WaterKnight1998]

Fixed with following code:

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace
import logging

logging.getLogger().handlers = []
configure_azure_monitor(connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
logging.getLogger().setLevel(logging.WARNING)

tracer = trace.get_tracer(__name__)

app = fastapi.FastAPI()

@app.get("/x")
async def x():
   with tracer.start_as_current_span("aaaaa"):
      logging.warning("a",extras={"b":1})

def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
    return func.AsgiMiddleware(app).handle(req, context)

[macieyng]

Do this and it should suppress those verbose logs.

configure_azure_monitor(connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
logging.getLogger("azure.core.pipeline.policies.http_logging_policy").setLevel(logging.WARNING)
logging.getLogger("azure.monitor.opentelemetry.exporter.export").setLevel(logging.WARNING)

[WaterKnight1998]

This helped, thank you very much @macieyng . The only thing that I don't understand is why disabling the automatic instrumentations and excluding urls didn't help

[macieyng]

It didn't work because Azure SDK is instrumenting itself while suppressing any other instrumentation. Read this comment: Azure/azure-sdk-for-python#31292 (comment)

[lzchen]

@WaterKnight1998

These additional logs are appearing in the "traces" table in your Application Insights resource correct? We have recently implemented a feature to define the specific logger that you want to collect traces telemetry. This should allow you to define the logger to your application logger instead of the default which is the root logger. This is why you are seeing logs collected from the SDK itself. This feature should be available soon in the next release.

[jeremydvoss]

Migrating to Azure sdk: Azure/azure-sdk-for-python#32503