Getting failed to send traces to Datadog Agent when DD_TRACE_ENABLED=false #4179
Comments
|
Hi @fredrikaverpil, I'm trying to reproduce this on my end but I'm not seeing the first error log |
|
Ok, I'm pretty sure I found I have a custom logger, which is based on loguru. When setting up this logger, I also intercept all standard library logging: import logging
from loguru import logger
class LoguruInterceptHandler(logging.Handler):
"""Intercept standard logging messages toward Loguru sinks.
Notes:
This will e.g. make it possible to see the SQL debug logs from
ORMs such as Peewee.
"""
def emit(self, record):
# Get corresponding Loguru level if it exists
try:
level = loguru_logger.level(record.levelname).name
except ValueError:
level = record.levelno
# Find caller from where originated the logged message
frame, depth = logging.currentframe(), 2
while frame.f_code.co_filename == logging.__file__:
frame = frame.f_back
depth += 1
loguru_logger.opt(depth=depth, exception=record.exc_info).bind(
metadata={"intercepted_from_std_logging": True}
).log(
level,
str(record.getMessage()),
)
def setup_interceptor():
# intercept everything at the root logger
logging.root.handlers = [LoguruInterceptHandler()]
logging.root.setLevel("INFO")
# remove every other logger's handlers
# and propagate to root logger
for name in logging.root.manager.loggerDict.keys():
logging.getLogger(name).handlers = []
logging.getLogger(name).propagate = True
setup_interceptor()This code executes late in my service. Meaning, it executes when most libraries etc have created their loggers. When I leave this out, the traceback no longer appear in the terminal when e.g. running pytest in my project. But I believe the ddtrace error log actually is just silenced and is executing anyway. |
|
Could it be that the ddtrace library silently logs this message (here), and my interceptor simply found it and also picked up the raised exception traceback? If I add a |
|
If the above assumption is correct, I don't think ddtrace should even attempt to send traces when |
|
Hi @fredrikaverpil, just so I can clarify what might be the problem could you add a few debug statements next to the
|
|
@Yun-Kim Sure! I added some prints: def flush_queue(self, raise_exc=False):
print("In flush_queue")
# type: (bool) -> None
try:
...
else:
print("Debug logging for Datadog", flush=True)
print(f"{n_traces=}", flush=True)
print(f"{encoded=}", flush=True)
print(f"{os.getenv('DD_TRACE_ENABLED')=}", flush=True)
log.error("failed to send traces to Datadog Agent at %s", self._agent_endpoint, exc_info=True)The results: n_traces=1encoded=b'\x91\x91\x89\xa8trace_id\xcfM\x08\xf2\xdcVq\xbeG\xa7span_id\xcf6\x1d\x8aA\xc9\x9c\x99\xce\xa7service\xb8conversations-migrations\xa8resource\xadinitContainer\xa4name\xd9 conversations.run_migrations.run\xa5start\xcf\x17\x14\n\xa3\xdc\x7f\x1a`\xa8duration\xce\x1ae\xc4H\xa4meta\x82\xaaruntime-id\xd9 e1b588fb6af148f9a628ebe93351bc40\xa8_dd.p.dm\xa2-0\xa7metrics\x85\xad_dd.agent_psr\xcb?\xf0\x00\x00\x00\x00\x00\x00\xaasystem.pid\xcd\xbbe\xad_dd.top_level\x01\xb5_sampling_priority_v1\x01\xad_dd.tracer_kr\xcb?\xf0\x00\x00\x00\x00\x00\x00'os.getenv('DD_TRACE_ENABLED')='false'I'm not using else:
from ddtrace.tracer import enabled
print(f"{enabled=}", flush=True)
log.error("failed to send traces to Datadog Agent at %s", self._agent_endpoint, exc_info=True)At first, I noticed that I did not have my db container running, and so I received a very large value full of tracebacks in I ran pytest to trigger all of this. The full output is below: DetailsIn case you're wondering what def flush_queue(self, raise_exc=False):
print("In flush_queue")
# type: (bool) -> None |
@fredrikaverpil That import does not work. The You'll want to use something like this for debugging: from ddtrace import tracer
print(f"{tracer.enabled=}", flush=True) |
|
@fredrikaverpil Just to add on, how are you running ddtrace and setting the environment variable |
We wrap our microservices with ddtrace-run in FROM python:3.10-slim-bullseye
...
ENTRYPOINT ["ddtrace-run", "python"]
CMD ["app.py"]But when I run pytest I don't run ddtrace at all, and this is where I am having these issues. Somehow tracing is enabled when running pytest. I'm running this command in my terminal with activated venv: pytest --count=100 -s -k <test_name>I have the following pytest packages installed: and I've got the following in my [tool.pytest.ini_options]
testpaths = "tests"
env_files = ["env/local_test.env"]
addopts = "--cov --cov-report=html --log-level=INFO -rxXs"In the TEST=1
DD_TRACE_ENABLED=false
LOG_LEVEL="INFO"
PYTHONDEVMODE="1" |
Right! I got this: |
|
Hi Fred, taking over for Yun here. Could you also try just printing out the environment variable value for
It seems that for some reason the tracer is still running with your test code despite you not running with ddtrace-run (unless hidden somewhere?) in the tests. My original thought was that you were enabling the tracer somewhere in your code by running |
I actually already did that in #4179 (comment) Yeah, this is a mighty weird one. I'll dig into this tomorrow (late night here now) to see if I can narrow this down further. To me it seems something is enabling the tracer even if the environment variable is present and set to "false". |
|
I think I just found what is causing all this. I have this in one place in my code that gets executed when running tests in pytest: @tracer.wrap(service="xxx", resource="yyy")
def run():
...As soon as I comment this line out, I can no longer see the error log and traceback. I've reproduced this now like 20 times, so I am 100% sure about this. The @tracer.wrap(service="xxx", resource="yyy")
def run():
logger.info(f"{os.getenv('DD_TRACE_ENABLED')=}")
logger.info(f"{tracer.enabled=}")What's weird here is that even if I remove the But I can 100% verify here that when removing |
|
That is odd as The only other thing you should be able to do to modify I could only recreate this behavior when calling |
|
Could there be a regression issue? See this issue: #3671 |
|
@fredrikaverpil If this is a regression issue of the pytest plugin restarting a tracer, could you try disabling the plugin as specified on #3674? Specifically in your pytest config file, add: |
|
@Yun-Kim doing that made no difference. I still see the traceback and So I just removed any ddtrace imports everywhere from my codebase. But I still have ddtrace installed in my venv. When I check Do you have some sort of hook that plugs into pytest and enables the tracer? |
Just to confirm, this did not disable the plugin on your end? We do have hooks in our pytest plugin which does start a global tracer, but as long as you set your environment variable |
Correct. This did not disable the tracer. I have this in my [tool.pytest.ini_options]
testpaths = "tests"
env_files = ["env/local_test.env"]
addopts = "--cov --cov-report=html -rxXs"...and so I added what you suggested: [tool.pytest.ini_options]
testpaths = "tests"
env_files = ["env/local_test.env"]
addopts = "--cov --cov-report=html -rxXs -p no:ddtrace"But the error log and the traceback still appears in the terminal, if I have a All in all, to me it seems there has to be something off with the pytest plugin, as I've removed all ddtrace imports and all ddtrace from my codebase, and ddtrace still shows up in When I loop through mods = []
for mod in sys.modules:
if "ddtrace" in mod:
mods.append(sys.modules[mod])
|
|
From the
I'm now also running with these [tool.pytest.ini_options]
testpaths = "tests"
env_files = ["env/local_test.env"]
addopts = "-p no:ddtrace -p no:ddtrace.pytest_bdd" |
|
@Yun-Kim so it seems by adding both options Additional info, not sure if it is helpful: I've put some logging in the If I then use the pytest opts And finally, if I keep the pytest opts above enabled, and import ddtrace in my code and also add a |
|
@fredrikaverpil fantastic! Happy to hear you were able to resolve the issue 🥳 |
|
@Yun-Kim yes, finally 😅 However, don't you think there's a problem here with ddtrace silently logging a traceback (the raised exception's I mean, it seems to me it should've been enough to set |
|
You're right, it is problematic that our pytest plugins don't check the value of |
|
Hi @fredrikaverpil, unfortunately we still weren't able to reproduce your issue on our end. The proposed fix #4204 doesn't directly address the underlying issue so we can't merge it until we figure out why we're seeing the pytest plugin disregards the value of |
|
@Yun-Kim hi, sorry for the late reply. I'll drop by in Slack and maybe we can schedule a session or something. |
|
I'll first post a reproducible example below 👇 |
|
@Yun-Kim so I think we've had some issues where we have been using dotenv file solutions to set However, the problem remains for local development, as the pytest options It would be great to have a way to tell pytest not to enable tracing (preferably as a commandline option), without requiring the environment variable |
|
You may wonder why my test is creating a custom span. Well, the reason is that this is what the source code being tested is doing. For e.g. kubernetes cronjobs, we don't have incoming traces but we would still like to have errors reported into APM. We also do this for unexpected crashes via |
|
I also am observing this behavior, although I'm not sure if it goes away when setting the flag or not -- but what's odd is that we use pytest-socket and the test does not fail even though all socket calls should be blocked and cause failures. |
|
I'm observing this as well. pytest always hangs if ddtrace is installed, it always installs the plugin, none of the suggestions above disable it. As suggested above, the plugin should at the very least respect the DD_TRACE_ENABLED=false env variable. |
|
In fact, I strongly believe that the default behavior should be for it to be off unless it's explicitly enabled. |
|
If it helps at all, ddtrace is version 1.6.3 one of two things happens here or, hangs and ctrl-c |
|
fwiw, nosetests also hangs after running trying to send it's profile, even though the env variables say it should not, so maybe not just a pytest thing. Maybe don't automatically turn on your tooling by default. |
|
Manually doing what the Github Action was supposed to do (fixed in #7835): This issue has been automatically closed after six months of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error. |
|
I also am observing this behavior - in my case the tooling isn't respecting our dotenv under local development. Pretty disappointing it hasn't been addressed after @fredrikaverpil provided a minimal repro. Please re-open. |
|
This isn't exactly a CI Visibility product issue, so I have to relinquish this back to our core folks. @mabdinur , I'm not sure if you want to reopen this or track it elsewhere. |
Which version of dd-trace-py are you using?
1.4.4
Which version of pip are you using?
22.2.2
Which version of the libraries are you using?
N/A
How can we reproduce your problem?
N/A
What is the result that you get?
Even if I have
DD_TRACE_ENABLED=falseI get this a lot of times, when running code locally:I am in my code calling e.g.
span.set_tagsandspan.set_exc_info(...)but I wouldn't expect traces to be sent off to Datadog because of this.Do you have any idea what could be triggering a trace to be sent, even if
DD_TRACE_ENABLED=false?What is the result that you expected?
I would not expect an attempt to send traces to Datadog when
DD_TRACE_ENABLED=false.More info:
The text was updated successfully, but these errors were encountered: