It should be easier to be sure to get a venv

[hynek]

Is your feature request related to a problem? Please describe.

When getting started with pdm, I've spent a lot of time trying to prevent pdm creating a __pypackages__ while bootstrapping a project. FTR by bootstrapping I mean a configured pdm project that is freshly checked out, or starting over.

AFAICT, the best way to end up with an in-tree project venv is:

pdm venv create $(command -v python3-intel64)
pdm use -i .venv/bin/python

(more about the -intel64 in a separate issue ;))

It would be really nice if that could be one, idiomatic call.

Describe the solution you'd like

I would like one of these:

• pdm config python.use_pep582 false: analogue to use_venv. I never want to use __pypackages__ and it would be nice to tell pdm that in general. pdm use would then create .venv just like pdm init would.
• pdm use -i --create-venv: currently it would create __pypackages__ so it would be nice to have an argument to get an venv, instead of pdm guessing depending on whether the Python is in an Virtualenv or not.
• pdm install --venv-python=(command -v python3-intel64): I think this would be my favourite solution because it's most idiomatic and explicit.

[frostming]

The problem is, to create venv implicitly or with --create-venv, the user has to specify the location of venv. We can't just overwrite the .venv directory here. So I'd prefer to create venv explicitly via pdm venv create ....

[hynek]

Hmm, I'm not gonna lie, this is very confusing behaviour:

$ ls -d  .venv/. # .venv exists
.venv/
$ pdm venv create -f (command -v python3-intel64)
Virtualenv /Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-3.11 is created successfully
$ pdm venv create -f -n in-project (command -v python3-intel64)
Virtualenv /Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-in-project is created
successfully
$ pdm venv list
Virtualenvs created with this project:

*  in-project: /Users/hynek/Work/customer-api/.venv
-  3.11: /Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-3.11
-  in-project: /Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-in-project

There should be warnings or something?

Something like:

In-project '.venv' exists and no name has been specified; creating '/Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-3.11' instead.

The case of in-project, wouldn't an option be better to avoid the third case above? Something like: pdm venv create --in-project with an optional --force – or am I missing something?

---

That aside, can you give us pdm config python.use_pep582 false that errors out if it would create __pypackages__ then? I'm not exaggerating when I say that I've spent more than an hour figuring out why that keeps popping up, after I told pdm init to use a venv and just tried starting over.

---

Relatedly, I'm a bit confused by the whole pdm venv thing – it seems to me it's entirely disconnected from the rest of pdm? Like it's impossible to say "I want a venv with -G lint and then run a composite command int there"? Am I missing something here?

To be clear: I do like the idea of having one main venv for development (.venv) and a bunch of auxiliary ones (just test dependencies [i.e. without dev tools], just docs, just linters without installing the package), but I fail to grasp how all these things are supposed to interact.

[frostming]

The case of in-project, wouldn't an option be better to avoid the third case above? Something like: pdm venv create --in-project with an optional --force – or am I missing something?

I've created #1666 to address this. Users are able to overwrite an in-project venv with pdm venv create --force (without name argument)

Like it's impossible to say "I want a venv with -G lint and then run a composite command int there"? Am I missing something here?

pdm venv is no more than a command (group) to manage the creation and removal of venvs associated with the current project. You can't do it in a single command, but three commands:

pdm venv create --name lint
pdm use  # and select the venv python
pdm install -Glint

There is room for improving the second step to specify the venv named lint without interaction.

That aside, can you give us pdm config python.use_pep582 false that errors out if it would create __pypackages__ then?

What if both use_pep582 and use_venv are false then? IMO a friendly hint message will suffice in this case, what would you suggest?

[hynek]

Users are able to overwrite an in-project venv with pdm venv create --force (without name argument)

This is not true for me:

$ pdm venv list
Virtualenvs created with this project:


$ pdm venv create
Virtualenv /Users/hynek/Work/customer-api/.venv is created successfully

$ pdm venv list
Virtualenvs created with this project:

-  in-project: /Users/hynek/Work/customer-api/.venv

$ pdm venv create --force
Virtualenv /Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-3.11 is created successfully

$ pdm venv list
Virtualenvs created with this project:

-  in-project: /Users/hynek/Work/customer-api/.venv
-  3.11: /Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-3.11

Oh wait, you mean with #1666? Leaving the above as documentation for #1666 then. :)

---

There is room for improving the second step to specify the venv named lint without interaction.

Given that pdm use doesn't prefer the venvs that are associated with the project (i.e. doesn't group them at the top), this doesn't seem great for me (pdm use --consider-only-project-venvs?). Also I need these things to be automatable and I don't see any upsides in that compared to managing my venvs myself.

So, I'm not sure whether you consider the current state as fully intentional, given that venv support is rather new, so just spitballing for your consideration:

The simplest case it would be nice to pass named venvs to pdm use:

$ pdm use --venv=for-lint

Next logical step:

$ pdm install --venv=for-lint -G lint

That also might create for-lint if it doesn't exist. But it could really be simplified to something like pdm --venv -G lint,foo and pdm uses some well-known naming scheme like groups-lint,foo.

Ideally in the end one wants something akin to what Hatch is doing tho:

[tool.pdm.scripts]
lint = { venv = { no-self = true, no-default = true, groups = ["lint"] }, cmd = "flake8 src" }

And then pdm run lint is equivalent to:

$ pdm venv create -fn script-lint
# this doesn't work – I suspect pdm install only cares about `pdm use`?
$ (eval $(pdm venv activate script-lint); pdm install --no-self --no-default -G lint)
# the following works, but screws up the work environment
$ pdm use -i "$(eval $(pdm venv activate script-lint); command -v python)"
$ pdm install --no-self --no-default -G lint
$ pdm run lint

Does that make any sense?

---

Fun fact aside: I tried to delete /Users/hynek/Library/Application Support/pdm/venvs/customer-api-Brwe_b2r-in-project (from my last comment) by calling pdm venv remove in-project and it deleted .venv.

---

What if both use_pep582 and use_venv are false then? IMO a friendly hint message will suffice in this case, what would you suggest?

Good question, depends a bit on whether / how you'll somehow tackle the venv confusion from above. I guess a plain "Using __pypackages__ because X." would be fine.

---

Another big wish I've noticed while typing the example above: please give me a pdm venv where foo that returns the absolute path to the python interpreter. Basically the equivalent of: (eval $(pdm venv activate for-lint); command -v python)

---

Sorry, I'm on a roll: am I missing something, or is there no way to list the packages that are installed in an venv, without creating it with --with-pip? A pdm venv list-packages would be great.

---

Sorry if my overall comments sound too negative (the plague of FOSS: you only hear about things that don't work) – pdm is to me the first full-service project manager that I could imagine to use for all my apps! It's just the venv corner that feels… off. I understand if that's something you're not interested in polishing, though.

[frostming]

That is a bunch of feature requests:

• Enable pdm use to use a venv python with the exact name
• Add a general option --venv to install into for pdm install and other commands
• Dependency selection parameters for pdm scripts(that looks like replicating tox though)
• Get the interpreter path for a given venv

I am not likely to implement all but some sound good to me.

[hynek]

Thanks for being so open-minded! I'm sure it's not easy for you to deal with venvs since it's my understanding that PDM's original raison d'être was implementing PEP 582.

Dependency selection parameters for pdm scripts(that looks like replicating tox though)

It's 100% fair to say that you don't want to compete with tox / Nox although I would kinda like you to, because in work apps I only use it for this: running commands in isolated venvs. :)

The whole reason why I push here is because the venvs are right there (as in: pdm venv) and it feels like pdm is not reaping what it has sowed. :) But I wouldn't ask you for matrices like Hatch does for instance.

[hynek]

That is a bunch of feature requests:

If I may:

• pdm list --venv=for-lint or some other way to inspect the contents of venvs would be great too. 😇

edit: realized we have pdm list and it makes more sense to make it venv-aware than adding a new command.

[davidism]

I already asked for dev environment management and the same tox point came up there #1606. I was coming from the angle of not wanting to tell contributors a complex series of steps just to set up a dev environment. You're doing a better job of explaining it / brainstorming, I think.

[hynek]

One more thing while we're at brain storming. :)

On my commute home I realized that I actually don't want you to compete with Tox.

What I want is a "project-local pipx". I (and ISTM David) don't want factors and matrices... just ways to run commands from defined environments that can be put in Makefiles / CONTRIBUTION.md.

[ofek]

What I want is a "project-local pipx". I (and ISTM David) don't want factors and matrices... just ways to run commands from defined environments that can be put in Makefiles / CONTRIBUTION.md.

Do you mind opening up an issue for Hatch as well? I don't quite understand the request.

[hynek]

What I want is a "project-local pipx". I (and ISTM David) don't want factors and matrices... just ways to run commands from defined environments that can be put in Makefiles / CONTRIBUTION.md.

Do you mind opening up an issue for Hatch as well? I don't quite understand the request.

Hatch supports this just fine. I basically want hatch run's features sans matrix (I'm not against matrix; I just don't need it in apps which is where I use PDM for its locking capabilities).

[mforbes]

A couple of comments:

With regard to both use_pep582 and use_venv being False: as I mentioned in issue #973, I think that use_venv=True should have the meaning of use_pep582=False, but however it is spelled, I definitely concur with being able to force use_pep582=False somehow.

---

What about simply pdm shell just working as I mentioned in #1763? I.e.:

1. Find a version of python consistent with the project, or use the one in .pdm.toml if it is consistent. (Raise an error if one cannot be found and suggest pdm use.)
2. Creating the venv if use_venv=True, possibly requiring --force to overwrite it, raising an error if one exists but is not using a compatible python.
3. Update the contents of the environment if needed: __pypackages__ if use_venv=False including the dev dependencies and any groups provided with -G.
4. Launch a new shell with that environment active.

If pdm run worked as described, then one might easily be able to add a shell command that runs your favourite shell, activating the venv, etc. This is the approach I currently take with anaconda-project as discussed in #1758.

As I am new to PDM, I am sure there are issues here somewhere, but something like this would be really nice for getting people to work.

---

I have not looked carefully into plugins yet: perhaps something like pdm shell as I describe (which is already provided by the pdm-shell plugin but does not actually activate an environment) could be provided by a plugin? Please let me know if that is how I should be exploring things, or if that might be more work than it is worth right now.

References

• python.use_venv = True does not cause packages to be installed in venv #973
• After venv activate, packages are still installed in the previous venv #1763
• Controlling PDM: Reproducibility vs. Convenience. #1758: My discussion about what I am currently trying to do, esp. with reference to anaconda-project, which I was using before, and how to precisely control the behaviour of PDM so that I could implement something like the pdm shell I describe here safely and reliably in a Makefile or as a plugin.