ci: install upstream ddtrace wheel during PR-triggered unit tests

[purple4reina]

What does this PR do?

Plumbs UPSTREAM_PIPELINE_ID (set by dd-trace-py's serverless lambda tests job when it triggers this repo's pipeline) into the unit-test, lint, and publish-pypi jobs so they exercise the dd-trace-py PR's wheel instead of the released ddtrace.

• New scripts/_spec_ddtrace_dep.sh — shared helper that rewrites pyproject.toml's ddtrace dep based on DD_TRACE_COMMIT > DD_TRACE_COMMIT_BRANCH > DD_TRACE_WHEEL > UPSTREAM_PIPELINE_ID (in that precedence). For the S3 path it tries the smaller ddtrace_serverless wheel first and falls back to ddtrace.
• New scripts/setup_python_env.sh — replaces the inline .python-before-script anchor. Backs up pyproject.toml, sources the helper, rewrites the dep, then pip install .[dev] resolves the whole graph in one pass.
• Refactored scripts/build_layers.sh — drops 60+ lines of inline duplicated logic and sources the same helper.
• ci/input_files/build.yaml.tpl — removes .python-before-script anchor; lint python, unit-test, publish-pypi-package now call setup_python_env.sh.

Motivation

dd-trace-py's serverless lambda tests job triggers this repo's CI on every dd-trace-py PR, passing UPSTREAM_PIPELINE_ID. build_layers.sh correctly used that ID to pull the PR's wheel for the layer build, but the unit-test job ran pip install .[dev] against the released ddtrace — so dd-trace-py PRs that broke lambda-python imports went undetected.

That's how dd-trace-py v4.8.0 shipped and required emergency hotfix #811 (fix: pin ddtrace to <4.8.0 to avoid import errors).

Testing Guidelines

• This PR's own pipeline runs without UPSTREAM_PIPELINE_ID → spec_ddtrace_dep is a no-op → released ddtrace is used → tests pass.
• To validate the linkage end-to-end, manually trigger a pipeline on this branch via GitLab's "Run Pipeline" UI with UPSTREAM_PIPELINE_ID set to a recent dd-trace-py main pipeline ID; the unit-test job should fetch the upstream wheel from S3 and exercise it.

Additional Notes

• Single-pass dependency resolution: rewriting pyproject.toml before pip install .[dev] (rather than pip install --force-reinstall-ing the wheel afterward) lets pip surface conflicts as install errors instead of test-time surprises if a dd-trace-py PR adds new deps or tightens version pins.
• Also fixes a latent bug: the original build_layers.sh used ddtrace_serverless as the dep name on the git/file paths, but the docker pip install runs without DD_SERVERLESS_BUILD=1 and produces a ddtrace-named dist that wouldn't have matched. The new helper uses ddtrace consistently for the git paths; the ddtrace_serverless name is only used when actually pulling a serverless-built wheel from S3.
• DD_TRACE_COMMIT / DD_TRACE_COMMIT_BRANCH build ddtrace from source and require cargo + cmake + a C/C++ toolchain — intended for local repro / git-bisect, not CI.

Types of Changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)

[purple4reina]

Confirmed the connection is working between this and dd-trace-py. PR used the latest from main in this pipeline https://gitlab.ddbuild.io/DataDog/datadog-lambda-python/-/jobs/1650520618

[purple4reina]

Tests are failing because we haven't yet merged #812