-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
What's the problem with duplicated logs? #316
Comments
Hi @macieyng it's unimportant. The original way we planned our diagnostic logging system with an "enable" method that would fork all of the logs under additional loggers, such as the exporter logger, to a file for debugging purposes. I found that when extending this to the exporter from the distro, there were certain logs that could be duplicated. This was a note to see if there was way to further this approach without duplicating any logs. However, I have a PR out that changes how diagnostic logging is done making this irrelevant. So, in short, this was a comment about a potential issue with a proposed future feature that is no longer relevant. Apologies if the comment was overly broad and vague. |
Reopening to ask: Unrelated, but could you explain more what you mean by "configuring logging level for Host.Function.Console in host.json" Some customers notice duplicate logs when they have both Function's host's .NET logging enabled in addition to their own Python logging. I usually solve this a different way am curious how you did. |
@jeremydvoss I have duplicated logs too but with different custom dimensions in my azure function. Is like they are not being merged. this is my code; from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace
configure_azure_monitor(connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
logger = logging.getLogger(__name__)
tracer = trace.get_tracer(__name__)
app = fastapi.FastAPI()
@app.get("/x")
async def x():
logger.info("a",extras={"b":1})
def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
return func.AsgiMiddleware(app).handle(req, context) |
Fixed by doing: from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace
logging.getLogger().handlers = []
configure_azure_monitor(connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
tracer = trace.get_tracer(__name__)
app = fastapi.FastAPI()
@app.get("/x")
async def x():
with tracer.start_as_current_span("aaaaa"):
logging.info("a",extras={"b":1})
def main(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
return func.AsgiMiddleware(app).handle(req, context) Azure Function Python Worker by default has a handler that causes duplicates |
However, now I am seeing too much traces. 3 traces each 5s. They looks like this:
I tried disabling this URLS from tracing: ENV VARS
|
I thought that this is a well known issue and currently there's no way around it. I'm not that familiar with how those runners, hosts, python workers work, but we're using Azure Functions v1 for Python. We have those Nevertheless, duplicates are probably caused by the fact that our app have a handler sending out logs to AI and also to azure functions host console, which then sends it to AI too. As of now, we just changed Most of our {
"version": "2.0",
...
"logging": {
"fileLoggingMode": "never",
"logLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Azure.Messaging.ServiceBus": "Warning",
"Azure.Core": "Warning",
"Microsoft.AspNetCore": "Warning",
"Host": "Information",
"Host.Controllers.Host": "Warning",
"Host.Function.Console": "Warning"
},
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request,Trace,Exception"
}
}
},
...
} I want to add that, I would not want to remove those default things logged by host, because they contain some useful stuff we didn't quite figured out yet like invocationId, operation_name, etc and logs like when function was triggered, what was the trigger, the final result (success/failure), etc.
that's because Watch this excellent talk https://youtu.be/5f4o3nyAz5w?si=2EJ2t1027wv37-mt - so much distilled information about logging. When it comes to those two OTEL_PYTHON_EXCLUDED_URLS=.*applicationinsights\.azure\.com/.*
OTEL_PYTHON_REQUESTS_EXCLUDED_URLS=.*applicationinsights\.azure\.com/.* These were excluded by default in OpenCensus to avoid self reporting. Not sure why and how they were lost during transition to OTEL. As per why {
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"default": {
"class": "opentelemetry.sdk._logs.LoggingHandler",
"logger_provider": get_logger_provider(),
},
},
"loggers": {
"": {
"handlers": ["default"],
},
},
} If we follow this logic then it would mean that, if we create new loggers via {
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"default": {
"class": "opentelemetry.sdk._logs.LoggingHandler",
"logger_provider": get_logger_provider(),
},
"other": {
"class": "logging.StreamHandler"
},
},
"loggers": {
"": {
"handlers": ["default"],
},
"__app__": {
"handler": ["other"],
},
},
} If you could look at As to @app.get("/x")
async def x():
with tracer.start_as_current_span("aaaaa"):
logging.info("a",extras={"b":1}) vs @app.get("/x")
async def x():
logger.info("a",extras={"b":1}) Try the second one with span context manager and it should yield similar results in customDimensions, like so: @app.get("/x")
async def x():
with tracer.start_as_current_span("aaaaa"):
logger.info("a",extras={"b":1}) As a sidenote, I think that we used to have a problem that all logs sent through StreamHandler->Host->AI used to have Error level displayed simply because StreamHandler logs to stderr and everything was interpreted as ERROR. We changed that because our DevOps team was complaining about that we push everything to stderr. We changed that and correct levels started to show up in AI. If anything I wrote contradicts any other part I wrote, it's me trying to figure out those things over and over again. Sorry if it adds communication noise. |
@jeremydvoss I set those 2 vars but I continue to see the logs. I even disable all automatic exporters: Line 108 in cdf59db
Do you know how I can get the python function not to register that in the log? |
@WaterKnight1998 I think that you should create a new issue for that. It has nothing to do with the original question. |
@macieyng @jeremydvoss I created a new issue #317 |
Our monitoring story with Azure functions has not been fleshed out that clearly. There hasn't been a clear guidance or separation on which component is responsible for collecting what (distro or azure functions). We will be working closely with the functions team to sort out these bugs and provide more clarity.
We could possibly change the exporter to send logs on the debug level. Our recommendation to users is to use namespaced loggers so users will not get sdk/exporter logs. |
Closed in favor of #317 which was then migrated to Azure/azure-sdk-for-python#32503 |
I saw this comment: https://github.com/microsoft/ApplicationInsights-Python/blob/main/azure-monitor-opentelemetry/azure/monitor/opentelemetry/_constants.py#L86 and wanted to ask if someone can elaborate?
We used to have duplicated logs with custom otel azure functions extension and configuring logging level for
Host.Function.Console
inhost.json
kind of solved it for us. Other way would be to removeStreamHandler
from logging configuration. I don't know if that's related, that's why I'd want learn more about this duplicated logs issue.The text was updated successfully, but these errors were encountered: