Enforce binary-only constraints during early requirements cache attempts#38688
Enforce binary-only constraints during early requirements cache attempts#38688shunping wants to merge 4 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request improves the reliability of the Apache Beam Stager by implementing a fallback mechanism for manylinux platform tags. By attempting downloads with progressively more compatible platform tags when the preferred tag fails, the system avoids unnecessary errors during the staging of dependencies that may not have published modern wheels. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a fallback mechanism for downloading Python dependencies using pip, iterating through a list of older manylinux platform tags if the preferred modern tag fails. The review feedback highlights that the hardcoded _MANYLINUX_PLATFORMS list only supports _x86_64 architectures, which bypasses the fallback mechanism on ARM64 (aarch64) environments. To resolve this, it is recommended to define base tags and dynamically append the architecture suffix, while filtering out unsupported combinations like manylinux2010 on ARM64.
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
cc: @claudevdm |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
|
cc'ed @derrickaw |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #38688 +/- ##
=========================================
Coverage 58.38% 58.39%
Complexity 14637 14637
=========================================
Files 2725 2725
Lines 271438 271498 +60
Branches 11884 11884
=========================================
+ Hits 158490 158539 +49
- Misses 106801 106812 +11
Partials 6147 6147
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Previously, pip download was executed with preferred platform configurations (e.g., manylinux_2_28_x86_64). However, if no matching binary wheel (.whl) was found, pip would by default fall back to downloading the source distribution (sdist) directly.
We modified the fallback logic here by injecting
--only-binary=:all:to enforce downloading only binary wheels in the early attempts. For example, for numpy 2.2.6, we first try downloading themanylinux_2_28_x86_64wheel. When it fails due to no such wheel, we then try downloading themanylinux2014_x86_64wheel. If we could not find any wheels for the platforms on the list, we will fallback to source distribution in the end.This fixed timeout errors in PreCommit Python 3.10 (for
MLTest.test_ml_preprocessing_yamlin https://github.com/apache/beam/actions/runs/26420471407/job/77774009271), and PostCommit Python 3.10 (forVertexAIImageEmbeddingsTest.test_image_embedding_pipeline_from_pathin https://github.com/apache/beam/actions/runs/26424456835/job/77785233507), because rather than staging and building numpy from source, we only need to download the correct version of wheels, which shortens the expansion service startup time during pytest.Also related to GoogleCloudPlatform/DataflowTemplates#3533