[python] improved local development experience with dd-trace-py

[sabrenner]

Motivation

Faster developer experience using local dd-trace-py for PARAMETRIC and INTEGRATION_FRAMEWORKS scenarios.

The current options outlined in the binaries documentation mostly point to pre-building wheels or pointing to a pip installable that would be installed on ever run, which is potentially slow, and I couldn't seem to make work with either utils/scripts/watch.sh ../dd-trace-py-ing my local dd-trace-py clone.

This PR looks to utilize PYTHONPATH SSI-like injection to set the ddtrace dependency from the mounted dd-trace-py, which has been used in a previous legacy test harness successfully.

Changes

Modifies the parametric and integration_frameworks scenarios to have their client servers use a PYTHONPATH pointed to a volume-mounted reference dd-trace-py.

The changes to make this possible are:

1. Update the PARAMETRIC and INTEGRATION_FRAMEWORKS scenarios to read from /binaries/python-load-from-local if it exists, and use that to mount a /volumes/dd-trace-py and set a PYTHONPATH entry for ddtrace to use that mounted dd-trace-py (this involved an underlying change to the DockerFixturesScenario base class
2. Special-case a step in install_ddtrace.sh to still install ddtrace for additional prod dependencies (like envier, etc.). It seems that even if they are installed locally they are not properly mounted - probably because they are installed in a virtual environment for those working on dd-trace-py. Maybe there's a way to leverage that? Either way, installing ddtrace from PyPi doesn't take too long.
3. The parametric client also needs to specifically take in container args, which it wasn't doing. Including when printing the ddtrace version, so it can reference the correct installation of ddtrace.
4. A script is added to build the necessary native extensions. Otherwise, client containers die instantly with

Traceback (most recent call last):
  File "/usr/local/bin/ddtrace-run", line 5, in <module>
    from ddtrace.commands.ddtrace_run import main
  File "/volumes/dd-trace-py/ddtrace/__init__.py", line 10, in <module>
    from ._logger import configure_ddtrace_logger
  File "/volumes/dd-trace-py/ddtrace/_logger.py", line 6, in <module>
    from ddtrace.internal.telemetry import get_config
  File "/volumes/dd-trace-py/ddtrace/internal/telemetry/__init__.py", line 11, in <module>
    from ddtrace.internal.settings._agent import config as agent_config
  File "/volumes/dd-trace-py/ddtrace/internal/settings/_agent.py", line 9, in <module>
    from ddtrace.internal.settings._core import DDConfig
  File "/volumes/dd-trace-py/ddtrace/internal/settings/_core.py", line 9, in <module>
    from ddtrace.internal.native import get_configuration_from_disk
  File "/volumes/dd-trace-py/ddtrace/internal/native/__init__.py", line 6, in <module>
    from ._native import AgentError  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'ddtrace.internal.native._native'

Testing

This is the method I used locally to verify these changes, for both scenarios. For both scenarios, add a python-load-from-local file under binaries/ whose contents are the relative path to your dd-trace-py

1. PARAMETRIC
   a. Run the utils/scripts/build_ddtrace_py_native.sh file if necessary (if ddtrace/internal/_encoding.cpython-311-aarch64-linux-gnu.so already exists, this step is not needed)
   b. In your local dd-trace-py, add an early return for the set_metric method (we're looking to break a test)
   c. Run this specific test ./run.sh PARAMETRIC -L python -vv tests/parametric/test_tracer.py::Test_Tracer::test_tracer_span_top_level_attributes, which tries to set a metric on the span.
   d. Confirm this test fails with the missing metric. Undoing the early return and re-running the test should restore it.
2. INTEGRATION_FRAMEWORKS
   a. Run the utils/scripts/build_ddtrace_py_native.sh file if necessary (if ddtrace/internal/_encoding.cpython-311-aarch64-linux-gnu.so already exists, this step is not needed)
   b. n your local dd-trace-py, add an early return for the _llmobs_set_tags_from_embedding method (we're looking to break a test)
   c. Run this specific test ./run.sh INTEGRATION_FRAMEWORKS -L python --weblog google_genai-py@latest -vv tests/integration_frameworks/llm/google_genai/test_google_genai_llmobs.py::TestGoogleGenAiEmbedContent::test_embed_content, which tries to set a metric on the span.
   d. Confirm this test fails with the missing metadata. Undoing the early return and re-running the test should restore it.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

docs/execute/binaries.md                                                @DataDog/system-tests-core
utils/_context/_scenarios/_docker_fixtures.py                           @DataDog/system-tests-core
utils/_context/_scenarios/integration_frameworks.py                     @DataDog/system-tests-core
utils/_context/_scenarios/parametric.py                                 @DataDog/system-tests-core
utils/build/docker/python/install_ddtrace.sh                            @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

[ncybul]

Tested this out locally and everything works as expected. Will leave the approval for other reviewers with more context on the system test changes!

[cbeauchesne]

Just added a blocking RC, to be able to discuss about the build script.

All the rest if good to me 😉

[sabrenner]

@cbeauchesne just updated the binaries.md and removed the script 😄