Use uv to freeze transitive dependency versions before parallel builds#22998
Use uv to freeze transitive dependency versions before parallel builds#22998
Conversation
Add a pre-resolve job to the resolve-build-deps workflow that uses uv pip compile --exclude-newer to generate a constraints file pinning all transitive dependencies at the commit timestamp. This eliminates race conditions where a new package version published on PyPI during the build window causes inconsistent lockfiles across platforms.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 349d82dc53
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| uv pip compile ${{ env.DIRECT_DEPENDENCY_FILE }} \ | ||
| --extra-index-url https://agent-int-packages.datadoghq.com/external \ | ||
| --exclude-newer "${{ steps.timestamp.outputs.cutoff }}" \ | ||
| --python-version ${{ env.PYTHON_VERSION }} \ | ||
| --output-file constraints.txt |
There was a problem hiding this comment.
I checked uv pip compile --help: --python-platform resolves for a specific target, while --universal is the mode that generates one file compatible with all OSes and architectures. This job runs once on Ubuntu and passes neither flag, so constraints.txt is a Linux resolution. That leaves non-Linux targets unfrozen: for example, a Linux uv pip compile of pywin32==311; sys_platform == 'win32' produces no pinned output at all, so Windows/macOS-only dependency trees can still pick newer transitive releases during build / build-macos and drift or fail.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The pre-resolve step intentionally resolves on a single platform. Its purpose is to pin pure-Python transitive dependencies (like attrs, typing-extensions, etc.) that are platform-independent and are the
ones susceptible to the race condition this change addresses. Platform-conditional direct dependencies (like pywin32) are already pinned in agent_requirements.in, and their platform-specific transitive deps
are either pure-Python (resolved on Linux too since they have no platform markers) or extremely rare.
We considered two alternatives for full platform coverage:
uv pip compile --universal: resolves for all platforms in a single run, but can introduce resolution conflicts when packages have incompatible version requirements across platforms.- Per-platform pre-resolve jobs: running 5 parallel pre-resolve jobs on matching runner OS types (ubuntu, ubuntu-arm, windows, macos-x86, macos-arm), each producing platform-specific constraints. This would
give exact coverage but adds significant complexity — 5 extra jobs, 5 separate artifacts, wiring each build to download the correct one, and using expensive macOS/Windows runners just for auv pip compilethat
takes seconds.
Both were discarded as the added complexity outweighs the negligible practical risk. We can revisit if a concrete case surfaces.
| if [[ -n "${HEAD_COMMIT_TIMESTAMP}" ]]; then | ||
| echo "cutoff=${HEAD_COMMIT_TIMESTAMP}" >> "${GITHUB_OUTPUT}" | ||
| else | ||
| echo "cutoff=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}" |
There was a problem hiding this comment.
This workflow is triggered by pull_request and workflow_dispatch as well as push, but github.event.head_commit.timestamp is only populated for push payloads. In the other two modes this branch falls back to date -u, so rerunning the exact same SHA later will compile against a newer cutoff and can pull different transitive versions. That breaks the reproducibility this new pre-resolve step is trying to add for PR and manual runs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This was already addressed — the timestamp step now uses git log -1 --format=%cI which derives the cutoff from the checked-out commit, not from the event payload. This works for all event types (push,
pull_request, workflow_dispatch) and is reproducible across re-runs of the same SHA.
1 participant
What does this PR do?
Adds a
pre-resolvejob to theresolve-build-depsworkflow that usesuv pip compile --exclude-newerto generate a pip constraints file before the parallel platform builds start. This pins all transitive dependency versions at the commit timestamp, ensuring all 5 build targets resolve the same versions regardless of what gets published on PyPI during the build window.Changes:
resolve-build-deps.yaml: Newpre-resolvejob usingastral-sh/setup-uvanduv pip compile --exclude-newer. Bothbuildandbuild-macosjobs now depend on it and download the constraints artifact.build.py: New--constraintsargument in bothbuild_image()andbuild_macos(). For Docker builds, the constraints file is copied into the mount directory and passed viaPIP_CONSTRAINTenv var. For macOS builds,PIP_CONSTRAINTis set directly in the environment.build_wheels.py: No changes — it already readsPIP_CONSTRAINTfrom the environment (line 413–414).uvis only used on the host runner in the pre-resolve step to produce a plain text constraints file. The actual wheel builds inside containers continue to usepipexactly as they do now.Motivation
On 2026-03-19,
attrs 26.1.0was published on PyPI at 14:22:23 UTC — right in the middle of a dependency resolution build window. Some platforms resolvedattrs==25.4.0(before the release) and others resolvedattrs==26.1.0(after), producing inconsistent lockfiles across platforms in PR #22987. The PR had to be closed and the workflow re-run (PR #22990) once the package was fully propagated.This is an inherent race condition: each build target calls
pip wheel -r requirements.inindependently, and there is no mechanism to ensure all targets resolve against the same snapshot of PyPI. This PR eliminates that race by pre-resolving all versions once with--exclude-newerand constraining all builds to the result.See AI-6709 for the full investigation and design.
See the example run in this PR for the constraints contents.
Review checklist (to be filled by reviewers)
qa/skip-qalabel if the PR doesn't need to be tested during QA.backport/<branch-name>label to the PR and it will automatically open a backport PR once this one is merged