Make PEP 582 an opt-in on PDM 2.0?

[frostming]

PEP 582 is what drives this project and the reason why it is born. However, it is also the main criticism about PDM -- PEP 582 is still in draft. People don't tend to recommend PDM to avoid the impression that it has become the CPython standard. That is quite unfortunate but it seems hard to push PEP 582 forward since it will definitely break someone's workflow while benefiting many more.

To change it to an opt-in feature, we can simply absorb pdm-venv into the main project, and venv will become the default. Then, PEP 582 can be enabled by pdm config python.use_venv false. Existing projects with whatever layout can work as before. But before that, I would like to hear more voice from the community, and hence opened this vote. Feel free to comment below on why or why not you support it.

/cc @linw1995 @pawamoy @noirbizarre @ElijahSink

[ElijahSink]

Ah, I'm disappointed that pep-582 isn't going to be accepted. I like it much better than all the variations on virtual environments.

Is there a way that existing projects could be grandfathered in without any need for going back and manually opting in to pep-582?

[frostming]

Is there a way that existing projects could be grandfathered in without any need for going back and manually opting in to pep-582?

It shouldn't need manual change, as long as the interpreter has been stored in .pdm.toml.

[ElijahSink]

In that case I don't have a problem with making it opt in. Maybe there can be a prompt in pdm init about it.

[cye18]

It shouldn't need manual change, as long as the interpreter has been stored in .pdm.toml.

.pdm.toml is in .gitignore by design. Do I need to do extra settings if I clone a project without .pdm.toml, if PEP 582 is made opt-in?

[frostming]

To use PEP 582 or not is more of a per-user flavor, so we don't recommend uploading .pdm.toml to VCS.
But if the repo contains an empty __pypackages__, PEP 582 will be used. Otherwise, virtualenv will be created. This is encouraged by the PEP:

Projects that use a source management system can include a pypackages directory (empty or with e.g. a file like .gitignore). After doing a fresh check out the source code, a tool like pip can be used to install the required dependencies directly into this directory.

See this repo for an example. PDM will create that folder with a default .gitignore for you, so that installed packages won't be uploaded.

[pawamoy]

Thank you for asking your community @frostming 🙂

Sure, if you feel it's what's best for the project, go for it. It's true that PEP-582 still being a draft can make PDM's design look premature, so I think this move would improve the seriousness of this project in the eyes of users and companies (native venv support would be a reassuring feature). And maybe help promoting PEP-582 even more? 🤔

I don't mind opting in 😊

[frostming]

Thanks, the main concern against doing this is it will make PDM no different from Poetry, or Pipenv, from the users' eyes. Maybe a faster development pace? I don't know if it is that important to users.

[pawamoy]

You can still primarily advertise PEP-582 support, while emphasizing it's opt-in and venvs are supported.

And yeah, fast pace development is definitely something I love about PDM. But I don't want it to become a guarantee: you deserve to get rest and nobody should ask you to continue working so hard, not even yourself ^^

(To me PDM has been feature-complete for some time now, maybe if we could write some design documentation, more users would be able to grasp the code base and be willing to contribute? Well that's another issue, we should probably discuss it somewhere else.)

[noirbizarre]

I think the PEP-582 support even if opt-in is already an argument.

Given I started using PDM recently after a painful year (maybe 2) looking for a decent solution (Python project management and packaging have been hell recently):

• it just work, no tricks need, no special version (seems logical but just try any other and you will understand)
• scripts support (game changer to me, Poetry only have it through poet and Pipenv ones are limited (no env, no shell chaining, no composition...)
• supports PEP-621: Poetry and Pipenv don't, setuptools has a just recently experimental support
• lock file support + pnpm-like cache
• editables support (only project so far having both PEP-621 and editables support)
• single tool batteries included (setuptools and flit don't cover the full workflow, poetry is going under heavy refactoring leading to 2 non-stable codebases, and Pipenv does not support packaging and extensions are very limited, mostly IDE integration)
• powerful extensions support + dynamic ecosystem: (I can attest it, I contributed a lot lately, everything has been reviewed and accepted, very quickly and the tone was good/kind). I can add transparent and realistic roadmap
• great doc !
• extensive test suite
• have I said it just work 🎉

And I'm sure some points are missing from the list.

To make it clear: PDM is as of today the only Python project management and packaging tool covering the full lifecycle, supporting all recently published PEP on packaging, supporting lock and being able to handle both libs and apps. (And this this why I'm currently in the process of migrating all my projects on PDM)

[berislavlopac]

Another favourite feature of mine: PDM is the only tool I know of that supports multiple build engines. This allows for neat tricks like supporting hatch's dynamic versioning if the build engine is hatchling.

[Hnasar]

I think opt-in is best. I've migrated my workplace to PDM and the biggest challenge in rolling out this change was explaining the difference between venv and PEP-582. Teaching people to do pdm run <script> vs <script> was a challenge, and the fact that mypy doesn't find stubs in __pypackages__ (python/mypy#10633) means that even if developers initially want to try PEP-582 they have to eventually switch to venv anyway.

When I was evaluating the different options, PEP-582 was a one of the least important aspects compared to Poetry and Pyflow.

1. Single lockfile pdm.lock that handles cross-platform compatibility
2. Supports git tag versioning: scm version
3. Active development with responsive maintainer
4. Implements all the latest latest PEPs:
   • PEP 621 and 631 (pyproject.toml metadata format.)
   • PEP 660 (editable installs)
   • Promised to support PEP 665 (standardized lock file)
   • PEP 582 (local package installer and runner, no need for virtualenv)
5. Fast dependency resolver.
6. Supports hook scripts and plugins

So while I personally do like PEP-582 (and hope the issues around it get resolved), marketing PDM as:

A modern Python package manager with PEP 582 support

is selling itself short.

I think PDM is absolutely the best packaging solution right now in terms of featurefulness and standards compliance, so I'd describe it more as:

A fast & modern Python package manager supporting the latest PEP standards, cross-platform reproducible deployments, and lockfile-based dependency conflict resolution.

[noirbizarre]

Feels like Mypy needs a PEP-582 plugin

[pawamoy]

Mypy 0.970 should naturally support PEP582 since directories in sys.path will now be used to find stubs. See the latest comment in the issue in question.

[frostming]

A fast & modern Python package manager supporting the latest PEP standards

Yes it is much better, let's change the description first

[Hnasar]

I was thinking about this a bit more and wondered, even if PEP 582 is not the default, what should be the initial behavior if someone clones a PDM repo, and runs pdm install?

Should PDM create a venv? Install globally? Install in the user directory? Install in __pypackages__ and log a message?

[frostming]

Should PDM create a venv? Install globally? Install in the user directory? Install in __pypackages__ and log a message?

Whatever they prefer in the user config, or create a venv by default.

[cye18]

Thanks for listening to the community! From a user's perspective I believe either way works for me and I might not even notice the difference, since the change won't break my workflow.

On the other hand, I think it would have more impact to you, especially on how you intend to design, evolve and advertise PDM going forward. I believe your decision will be in the best interest of this project.

By the way, any updates on PEP 582? It being a draft is the main reason why we have to have this discussion and make a choice. Hope it will be adopted soon.

[frostming]

I've started the work in #1193