Enable debug logs for debugger scenarios

[jpbempel]

Motivation

need to have debug logs to understand what is going on at tracer level, specially for flaky tests

Changes

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

utils/_context/_scenarios/debugger.py                                   @DataDog/system-tests-core

[cbeauchesne]

In theory, debug log must not be activated by default for QA reason (being as close as possible from our customer setups).

If you plan to run this in a dedicated CI env, or locally, you can achieve the same, but cleaner with the dedicated option to activate debug logs : --force-dd-trace-debug. To use it, you can add this in the configure method (line 39) :

    if config.option.force_dd_trace_debug:
        self.weblog_container.environment["JAVA_OPTS"] = " -Ddatadog.slf4j.simpleLogger.log.com.datadog.debugger=debug "

Then, just add --force-dd-trace-debug to the run.sh command. Let me know if this work for you ?

[jpbempel]

there is more and more flakiness in the tests, and not having the debug logs for tracer is problematic to catch and troubleshoot what exactly what's going on. this is only happening in CI and not reproducible locally.
Apart from performance I don't see how debug logs will bring QA difference of what we are trying to achieve with system-tests.

[cbeauchesne]

Apart from performance

Yes, it's performance : it usually trigger many bugs that we are not interested.

But I understand your issue. It'll be unreasonnably complex to offer a per-scenario option in the official system-tests workflow, so I presume it's the best course of action.

Just keep in mind that once you've solved the flakyness you'll need to remove it? And may I ask you top add a small comment why we have temporary added this ?

[jpbempel]

it usually trigger many bugs that we are not interested.

that's sound weird to me.

Keep it mind that all the runtimes are not equivalent on this...