Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt "cross-builds" of sdists for foreign platforms. #2075

Merged
merged 5 commits into from
Mar 2, 2023

Conversation

jsirois
Copy link
Member

@jsirois jsirois commented Mar 1, 2023

The "cross-building" is in scare-quotes because this is not actually
cross-building, it is just attempting a build of the sdist, and, if
successful, seeing if the resulting wheel matches the foreign platform.
This enables sdist-only projects to be used in foreign platform Pex
operations when it turns out the sdist is multi-platform.

Closes #2073

The "cross-building" is in scare-quotes because this is not actually
cross-building, it is just attempting a build of the sdist, and, if
successful, seeing if the resulting wheel matches the foreign platform.
This enables sdist-only projects to be used in foregin platform Pex
operations when it turns out the sdist is multi-platform.

Closes pex-tool#2073
@jsirois jsirois requested review from kaos and benjyw March 1, 2023 05:10
)
),
reason="Test requires a manylinux2014_x86_64 compatible interpreter.",
)
@applicable_pip_versions
def test_download_platform_issues_1355(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test changed substantially since ansicolors is now an example of a distribution which is "cross-buildable" for foreign platforms, but it did so to reflect the actual original issue: #1355

That issue is now verified directly and exactly.

@jsirois
Copy link
Member Author

jsirois commented Mar 1, 2023

This is a bit of a graduation for Pex. As of this change, which many prior changes over the last 2-3 years have laid the groundwork for, I see no more buggy-by-design behaviors left over from the Twitter years that can be fixed without breaking people. Time permitting, Pex 3.0.0 is ready to be prepared for (writing a real doc site) and cut.

@jsirois
Copy link
Member Author

jsirois commented Mar 1, 2023

I'm on Linux.

Old:

$ python -mpex.cli lock create --style universal --platform macosx-10.13-x86_64-cp-36-m psutil==5.7.0 -o psutil.lock.json
Failed to resolve compatible artifacts from lock for 1 target:
1. cp36-cp36m-macosx_10_13_x86_64:
    Failed to resolve all requirements for abbreviated platform cp36-cp36m-macosx_10_13_x86_64:

Configured with:
    build: False
    use_wheel: True

Dependency on psutil not satisfied, 1 incompatible candidate found:
1.) psutil 5.7 (via: psutil==5.7.0) does not have any compatible artifacts:
    https://files.pythonhosted.org/packages/0b/6b/f613593812c5f379c6d609bf5eca36a409812f508e13c704acd25712a73e/psutil-5.7.0-cp27-none-win32.whl
    https://files.pythonhosted.org/packages/4f/3c/205850b172a14a8b9fdc9b1e84a2c055d6b9aea226431da7685bea644f04/psutil-5.7.0-cp36-cp36m-win_amd64.whl
    https://files.pythonhosted.org/packages/54/25/7825fefd62635f7ca556c8e0d44369ce4674aa2ca0eca50b8ae4ff49954b/psutil-5.7.0-cp37-cp37m-win32.whl
    https://files.pythonhosted.org/packages/63/d5/f34a9433a0299d944605fb5a970306a89e076f5412164179dc59ebf70fa9/psutil-5.7.0-cp38-cp38-win32.whl
    https://files.pythonhosted.org/packages/65/c2/0aeb9f0cc7e4be2807aa052b3fd017e59439ed6d830b461f8ecb35b2f367/psutil-5.7.0-cp35-cp35m-win_amd64.whl
    https://files.pythonhosted.org/packages/74/e6/4a0ef10b1a4ca43954cd8fd9eac02cc8606f9d2a5a66859a283f5f95452b/psutil-5.7.0-cp35-cp35m-win32.whl
    https://files.pythonhosted.org/packages/79/b1/377fa0f28630d855cb6b5bfb2ee4c1bf0df3bc2603c691ceefce59a95181/psutil-5.7.0-cp27-none-win_amd64.whl
    https://files.pythonhosted.org/packages/86/f7/385040b90dd190edc28908c4a26af99b00ae37564ee5f5c4526dc1d80c27/psutil-5.7.0-cp37-cp37m-win_amd64.whl
    https://files.pythonhosted.org/packages/86/fe/9f1d1f8c1c8138d42fc0e7c06ca5004e01f38e86e61342374d8e0fa919e4/psutil-5.7.0-cp38-cp38-win_amd64.whl
    https://files.pythonhosted.org/packages/c4/b8/3512f0e93e0db23a71d82485ba256071ebef99b227351f0f5540f744af41/psutil-5.7.0.tar.gz
    https://files.pythonhosted.org/packages/c9/37/b94930ae428b2d67d505aecc5ba84c53a0b75479a8a87cd35cc9a2c6eb7e/psutil-5.7.0-cp36-cp36m-win32.whl

New:

$ python -mpex.cli lock create --style universal --platform macosx-10.13-x86_64-cp-36-m psutil==5.7.0 -o psutil.lock.json
No pre-built wheel was available for psutil 5.7.0.
Successfully built the wheel psutil-5.7.0-cp310-cp310-linux_x86_64.whl from the sdist psutil-5.7.0.tar.gz but it is not compatible with the requested foreign target abbreviated platform cp36-cp36m-macosx_10_13_x86_64.
You'll need to build a wheel from psutil-5.7.0.tar.gz on the foreign target platform and make it available to Pex via a `--find-links` repo or a custom `--index`.

Copy link
Collaborator

@benjyw benjyw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! A couple of minor nits that are not worth fixing and triggering another CI run, but maybe worth tacking on to a future unrelated change.

tests/integration/test_issue_2073.py Outdated Show resolved Hide resolved
pex/pip/foreign_platform.py Show resolved Hide resolved
@@ -119,6 +120,13 @@ def pex_repository(py27, py310, foreign_platform, manylinux):
)


@pytest.mark.skipif(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm, it looks like these 2 Python 3.5 failures are not bitrot. Main is definitely green for them. I'll circle back in the am and figure these two out before proceeding.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, yeah - this caught a bug. When I eliminated iter_platform_args(), that killed sending Pip the --platform ... --implementation ... --python-version ... --abi ... quad. The foreign_platform patches cover --platform ... --implementation ... --abi ..., aka: wheel tag matching, but they do not cover --python-version ..., aka Requires-Python matching. Pip requires you send the quad as a whole; so the --python-version ... / Requires-Python matching must be patched too. That patching is already implemented in the locker patches; so I'll need to share that code here for foreign platform resolves.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've reverted the test skips and updated the PR to include patching Requires-Python for foreign platform resolves. The PR update is large, but its mainly shuffling. The code to patch Requires-Python already existed to support universal lock resolves and so 95% of the delta here is extracting that bit of patching and re-arranging for re-use in the foreign platform patches.

@jsirois jsirois merged commit 0dbac1e into pex-tool:main Mar 2, 2023
@jsirois jsirois deleted the issues/2073 branch March 2, 2023 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support attempts to build sdists for foreign platforms.
3 participants