Interest in supporting PEP 665?

[brettcannon]

What's the problem this feature will solve?

Standardizing on a file format that fixes some shortcomings of requirements files.

Describe the solution you'd like

Two possibilities.

One, pip freeze emits PEP 665 files.

Two, -r (or some new option) can read in and install from a PEP 665 file.

Alternative Solutions

Status quo.

Additional context

https://discuss.python.org/t/pep-665-take-2-a-file-format-to-list-python-dependencies-for-reproducibility-of-an-application/11736

I'm basically looking for tool support to quote in the PEP at this point to help drive its support/acceptance.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

I feel pip freeze is a wrong approach, similar to the request for pip freeze --generate-hashes (which we get quasi-often). PEP 665 requires artifact hashes by default, but pip freeze operates on an environment, not artifacts, so those hashes are not available. A better approach would be a new command pip resolve (or say pip install --dry-run but IIRC pip resolve is the more popuilar proposal?) that emits this file from a set of requirements (command line arguments and/or -r).

+1 to making -r accept the PEP 665 format. Also -c. This would also be a good time to add something like --remove-unused to install so it's possible to "sync" the lock file to an environment, instead of simply applying the lock file on top of existing packages (which is the semantic of pip install -r). Alternatively we can also have pip sync just for this.

[pradyunsg]

To answer the interest question: Yes, to the extent that multiple maintainers are co-authors. ;)

[pradyunsg]

For the UI design discussion:

As noted, pip freeze doesn't have the information necessary to generate the lockfile. +1 to having this in pip resolve -- it likely makes sense as the default output format as well.

-1 to allowing this in -r. I'd much rather we create a clearly separate option to consume the lockfile. pip install --lock-file whatever.pylock.toml is clearer, will avoid having conditionals and -l is available. There's no reason to overload an existing option, to have multiple behaviours from my PoV.

I also think it would make sense for pip sync-style behaviour to be the mode of operation for consuming from a lockfile -- I can't think of any good reason to "add" to an environment using a lockfile... so I'd prefer to defer adding that functionality until we have a user express interest in doing things like that (plus, we'll know more about the exact usecase, which is also good).

[uranusjr]

I'd prefer to defer adding that functionality until we have a user express interest in doing things like that (plus, we'll know more about the exact usecase, which is also good).

Arguably we already do? #5823 #9118 #9166 #9799

[pradyunsg]

Hmm... I think there's a misunderstanding. Allow me to try again. What I'm saying we should do is:

• Leave pip install -r [path] as is. It modifies the current environment to ensure that these requirements are available and satisfied; installing and upgrading as necessary.
• Add pip install --lock-file [path]. It'll modify the current environment to match the environment described in the lockfile; installing, upgrading and uninstalling as necessary. This is the pip sync-style behaviour that the issues you've linked to are asking for.

I don't think we should have a mode where you can disable the uninstall semantics when using the lockfile; not unless someone explicitly asks for it. In other words, I think we shouldn't add is "install things from this lockfile to the environment, in addition to whatever is already installed" (think of howpip install -r requirements.txt works right now).

[sbidoul]

+1 for supporting this in pip.

What follows is not much more than instinctive reactions at this stage, as I have not digested the draft PEP yet.

For installation, a -l option to pip install looks good to me. I don't think it should uninstall anything though, as that would be counter-intuitive for an install command.

I'm not sure pip should venture in the environment management space just yet. Although if we want to, pip sync resonates well with me.

Regarding generation of the lock file I always thought that both the pip freeze and pip-compile approaches are valuable. Freeze does require recording information about the original artifact (its hash at least) but that sounds doable although it probably requires a new PEP.

Also regarding the generation of the lock file, I have a question: are there thoughts or examples of how the UI/UX would look like for generating a multi-platform lock file (with pip or in general) ?

[pfmoore]

My instinct is that "installing a lock file" has to mean "making the environment look like what the file specifies". Otherwise there are too many questions around what happens in the case of conflicts, etc. But I agree that it may be confusing if pip install starts doing uninstalls as well.

I think if we're supporting lockfiles, we need a new subcommand (+1 for pip sync) and we need to accept that we're getting into "environment management". There are too many options to pip install that would need thinking through and could cause issues if we have a lockfile, so I think starting with a clean slate would be better.

One possible issue though - in general, I'd expect a lockfile would definitely not specify that pip was to be installed (nothing depends on pip, after all). But we'd have to have pip in the environment to do the install, so "install exactly what is in the lockfile" won't be 100% accurate anyway, we'll need to make an exception for pip. Not a big deal, but let's be careful to be explicit here...

[uranusjr]

Hmm... I think there's a misunderstanding.

Yeah, I think I read your response wrong.

The one thing I don't like about having pip install -l do the sync semantic is it makes pip install -l <file> and pip install -r <file> do different things, which can be confusing for newcomers. It also brings weird combinations like pip install -l <file> [more packages] which obviously does not make sense and we need to manually reject. It definitely makes more sense to me to have a new command doing the new semantic. pip install -l should either match -r, or not exist at all.

[pradyunsg]

Yea, a pip sync commands works too for me -- as long as we're not mimicking the -r behaviours, I'm happy. :)

[pradyunsg]

Also regarding the generation of the lock file, I have a question: are there thoughts or examples of how the UI/UX would look like for generating a multi-platform lock file (with pip or in general) ?

Right now, the workflow that I've got bouncing around in my head is:

$ pip resolve -r requirements.txt mousebender packaging -o something.pylock.toml
$ pip sync something.pylock.toml

We'd effectively be providing the workflow that pip-tools provides; albeit with a standard-backed format (instead of requirements.txt) as the intermediary.

[sbidoul]

@pradyunsg yep, but my keyword was "multi-platform" :) I understand the lock file would support multiple platforms ? Or is it up to the user to create one lockfile for each platform ? Or does pip resolve try different platforms ?

[pradyunsg]

Well...

1. We can generate a platform-specific lockfile -- using the metadata table in the format (https://www.python.org/dev/peps/pep-0665/#metadata) to restrict where the packages can be installed.
2. We could also explore generating multi-platform lockfiles, although that might not be feasible in a reasonable time frame with the way our codebase works currently. In that case, we'd just not set the metadata keys that restrict compatible platforms.

Realistically though, we'd likely only implement 1 -- the platform-specific lock file -- at least in the first implementation. We can explore either "expanding" what we output to be platform-agnostic in a follow up, and/or defer to other tooling in the ecosystem to have them generate platform agnostic files.

[sbidoul]

Yeah, my question should rather have been for the pep discussion, as it is not pip specific, and I agree pip may not need to care about that aspect in a first iteration.

[sbidoul]

pip resolve -r requirements.txt mousebender packaging -o something.pylock.toml

About this example. What would it look like if the application for which we want to lock the dependencies has a pyproject.toml instead of a requirements.txt.in.

We may need a specific mechanism for that use case, as pip resolve . -o something.pylock.toml would need to know that . is special to exclude it from the lock file.

[pradyunsg]

Yeah, my question should rather have been for the pep discussion, as it is not pip specific, and I agree pip may not need to care about that aspect in a first iteration.

Ah! The sub-text from my response answers your PEP-related question too though -- The UX on the installer's end would be the same for well-formed lockfiles, regardless of whether the lockfile is a cross platform or not. It will "just" discard more things that don't match the environment (markers or tags).

What would it look like if the application for which we want to lock the dependencies has a pyproject.toml instead of a requirements.txt.in.

I hadn't thought of this!

We may need a specific mechanism for that use case

I agree, although...

pip resolve . -o something.pylock.toml would need to know that . is special to exclude it from the lock file.

I think local directories should be rejected when given as a top-level input to resolve.

Instead, we could have something like --only-deps [local-path-or-pep508-requirement] in that command. That's more explicit and more clearly describes the behaviour the user would get.

[brettcannon]

First, thanks for the vote of support on the PEP (for those that aren't co-authors 😉).

Second, for pip resolve, an open issue on the PEP is whether the input for the lock file should be standardized (or at least have at least one way to have a default) and I'm happy to discuss on discuss.python.org.

[brettcannon]

One thing to consider with a pip sync command is people who want to include things that are not in the lock file. For instance, a user in the PEP 665 discussions wants to include a bunch of editable installs. I don't expect support for that landing in the PEP (although I could be wrong), and so they would want some side mechanism to install their other packages that they don't have available as wheels (same goes for people wanting to use sdists and source trees for those one-off requirements).

As such, you may want to provide a requirements file escape hatch as extra stuff to pull in that won't fit in a lock file.

[pfmoore]

The PEP positions the lock file as very much an explicit specification of "this is what you install to create the environment for this app". Reproducibility is a key goal. Allowing "extra stuff" to be installed risks compromising that goal, which is one of the key reasons I think we should have a separate command for building an environment from a lockfile.

Without some sort of support in the PEP for the use case of installing "other stuff" alongside the packages listed in a lockfile, and some explanation of the intended/expected semantics, I'd be -1 on supporting that in pip. Also, if we define behaviour and the standard is later extended in a way incompatible with our approach, we end up with a backward compatibility issue, so that's another reason for not immediately going beyond what the standard specifies.

It's easier to add functionality than remove it, so let's start without this. If the PEP authors think the use case is worth supporting, they can do so in the PEP. Otherwise, we should follow their lead.

[pradyunsg]

That aside, it's not like requirements.txt is going away anytime soon -- so there's no reason for the lockfile to accomodate for all the potential ways that pip allows you to do things. Starting with a subset and expanding is fine IMO.

[uranusjr]

My idea to "support" the use case is something like

pip resolve -r requirements.in -o foo.pylock.toml
# If pip finds anything not lock-able, it emits foo-additional-requirements.txt listing them.

pip sync foo.pylock.toml
pip install -r foo-additional-requirements.txt --no-deps

(Persuambly --no-deps is not needed since all the transitive dependencies are already installed by sync, but that makes the command quicker.)

[sbidoul]

First, thanks for the vote of support on the PEP (for those that aren't co-authors wink).

To clarify my position (and sorry to bring a somewhat dissonant voice here), while I support pip implementing PEP 665 in principle, I'm stil assuming that the question of supporting sdists and VCS URLs can be resolved in PEP 665 before we implement this in pip.

Indeed pip provides first class support for both and I think it would be surprising for pip users to offer a replacement to requirements.txt that does not support those.

Also, the question of pinning build dependencies (of dependencies) regularly comes up and I think it's a great opportunity to provide a solution for that.

[hmc-cs-mdrissi]

I'd be very happy with the solution of pip resolve emitting all not lockable (guessing defined as -e/source) requirements to a separate file. That would be a perfect fit for my problem.

I'd be happy too if no additional file is produced as long as the lock produced only included lockable requirements. As long as pip resolve produces fully locked whiles while allowing requirements.in to include things that can't be locked it will fit well for my situation. Manually managing additional-requirements file is pretty doable for me, so I think foo-additional-requirements.txt is a bonus

[sbidoul]

Thinking about pip sync I came across two other potential concerns.

The first is when the local project is installed in editable mode and the lock file has its dependencies (a common use case I assume). In such case, pip sync should not uninstall the local project.

Another, maybe less common, case is to have a lock file for each set of extras (e.g. a lock file for the runtime requirements, a lock file for test requirements, or docs requirements). In this case we may want a mechanism to pip sync several compatible lock files at once.

[uranusjr]

I don’t see why the editable project is a concern, to be honest. If we go with the approach that resolve produces and extra requirements file for currently un-lockable packages, the user can install the local editable project separately after sync rather trivially (we could even just do that automatically if we name the extra requirements file well enough to be picked up by sync).

As for doing sync with multiple lock files, I’d argue we should put down a strict one-lock-file-per-environment rule and not support it. If you want an environment with all extras installed toghether, generate a lock file with all those extras instead (pip resolve .[ex1,ex2...]). One thing we could do to improve the process is to also allow using a lock file as a constraints file (which I believe should be straightforward enough), so you could pretty easily aggregate multiple exiting lock files like this

pip resolve .[ex1] -o ex1.pylock.toml
pip resolve .[ex2] -o ex2.pylock.toml
pip resolve .[ex1,ex2] -o aggregated.pylock.toml -c ex1.pylock.toml -c ex2.pylock.toml

[sbidoul]

I guess the point I want to get across with these examples (and when I say I'm not sure pip should venture into env management) is that there are nuances in workflows around a sync command. You known my view that pip should be an enabler for various workflows, not enforce one. So I think we must be careful to design these commands to be versatile enough so that people can easily build their own workflow on top of them.

[pradyunsg]

I'll note that we don't have to get this right in the first attempt. We explicitly have the ability to use an escape hatch of "this is an experimental command" if we find out that something is more complexity than would be worthwhile for us.