Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Export a matrix of resolve+python versions #15817

Open
cognifloyd opened this issue Jun 14, 2022 · 4 comments · May be fixed by #20516
Open

Export a matrix of resolve+python versions #15817

cognifloyd opened this issue Jun 14, 2022 · 4 comments · May be fixed by #20516
Labels
backend: Python Python backend-related issues enhancement

Comments

@cognifloyd
Copy link
Member

cognifloyd commented Jun 14, 2022 •
edited
Loading

Is your feature request related to a problem? Please describe.
I would like to export venvs for multiple python versions in each of my resolves.

I keep an one instance of every major python version (2.7 and 3.4+) available on my dev box. When resolving the interpreter, pants/pex defaults to the lowest available that meets the given constraints. That's fine, until I want to work with a particular python version.

Describe the solution you'd like
Enhance python venv exports so that they multiple python versions can be exported for each resolve.

The default behavior is fine: one venv for each resolve using the lowest python version in the interpreter constraints.
In the python backend, add two options to the export subsystem that allows a user to:

  • limit the exported venvs to a selected (set of) resolve(s), and
    • filter targets to only include the indicated resolve(s)
  • define which python interpreter to use.
    • If only one resolve is given: Error if that interpreter does not match the resolve's interpreter constraints.
    • If multiple resolves are given (like the default which does all possible resolves): Warn and ignore any resolves that don't match the given interpreter

Describe alternatives you've considered
It could create a venv for each python version that is (a) available locally, and (b) matches the interpreter constraints.
That seems a bit too shotgun. I'm happy to guide pants as to which venvs I want created.

Additional context

./pants export ::

Here is the pants.toml in the project I'm using to play with this: https://github.com/st2sandbox/st2/blob/pants/pants.toml#L112-L138
It's got multiple resolves, one of which has a different set of intepreter constraints (pants-plugins follows pants' constraints, not my project's constraints).

I don't need tool resolves to have this behavior (at this point).

So, I want to get this matrix of virtualenvs (only some can be exported today, indicated by + in the today column):

resolve python today
python-default 3.6 +
python-default 3.8
pants-plugins 3.7 +
pants-plugins 3.8
pants-plugins 3.9
pylint_plugins 3.6 +
pylint_plugins 3.8

This is where the matrix is generated today:

pants/src/python/pants/backend/python/goals/export.py

Lines 212 to 218 in a53a94a

venvs = await MultiGet(
Get(
ExportResult,
_ExportVenvRequest(resolve if python_setup.enable_resolves else None, tuple(tgts)),
)
for resolve, tgts in resolve_to_root_targets.items()
)

I'm not sure how to add extra cli args for the export goal or how to determine the python version to use here.
@cognifloyd
Copy link
Member Author

As far as selecting the resolves goes, that is probably related to these:

And about selecting the python version, these are probably related:

@cognifloyd
Copy link
Member Author

I just found (err - found again, because I forgot about it) the add/remove option syntax: https://www.pantsbuild.org/docs/options#addremove-semantics

So, with my venvs already exported using python 3.6, I tried running this:

./pants export :: --python-interpreter-constraints='-["CPython==3.6.*"]'

But, that deleted the python3.6 venvs instead of just adding the 3.8 venvs. I want to have both.

@Eric-Arellano
Copy link
Contributor

I think we in general could benefit from export having more options for what to generate. It is currently "everything", including tool PEXes. Compared to generate-lockfiles --resolve=black, for example, only generating the lock for Black.

@thejcannon thejcannon added the backend: Python Python backend-related issues label Aug 19, 2022
@cognifloyd
Copy link
Member Author

These improve the UX for selecting which resolve to export (landing in 2.15.x):

These facilitate allowing multiple export invocations, and creating multiple venv exports per resolve (for multiple interpreters) (landing in 2.16.x):

Related issue:

Now, we need some more usability features to allow selecting the python version or range of versions to export venvs for.

@cognifloyd cognifloyd linked a pull request Feb 9, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backend: Python Python backend-related issues enhancement
Projects
2023 Priorities
Status: No status
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Export venvs for a matrix of multiple python interpreters pantsbuild/pants
3 participants
@cognifloyd @thejcannon @Eric-Arellano