How does PDM differ from say pyenv/miniconda/poetry?

[jimmywan]

Hello, I just stumbled across this project's introductory blog post.

After reading this:

One day a minor release of Python is out and one has to check all those venvs and upgrade them if required before they can safely delete the out-dated Python version.

I can certainly related to the frustrations of having virtualenvs tied to a particular python install, but this is the kind of thing that I've been able to solve via pyenv+miniconda for many years now.

Something like this and you're mostly done:

pyenv shell miniconda3-latest
conda install -n my_virtualenv -y 'python==3.10.1'
pyenv shell --unset
# if you layer poetry on top
poetry install --no-root --remove-untracked

If you want more repos, stick it in a for loop and you're 99.9% of the way there.

I'm not here to try and bash this tool in any way. I'm genuinely curious as to what spurred someone on to spend the time to build yet another python packaging tool. After skimming this particular repo's README, it's just not entirely clear to me why this tool exists and/or what problem the author(s) set out to solve.

Having played with tooling such as pip-tools and pipenv that offer wildly frustrating partially complete solutions, I can certainly understand wanting a better way to do things.

[frostming]

Correct me if I am wrong, pyenv install python in different paths depending on the versions. So a venv created based on a particular python version may become stale if the hosted Python gets deleted from the system. I am not a heavy user of miniconda so I don't know if the same applies to conda. In fact, PDM is on par with Poetry and Pipenv but not miniconda or pyenv. In fact PDM can work with pyenv pretty well.

PDM is just seeking another way that isolated environments should not rely on an interpreter. It is packages that need to be isolated not the interpreter. Another important reason is you actually don't need a reason to build something that will make yourself happy, isn't it? So I can design the interface my own way and implement rather new PEPs about the Python packaging.

[pawamoy]

So to install a new version of Python you go to www.python.org using Safari, download the macOS installer, then run it and click on buttons to install it? That's a way to do it. I prefer to run basher upgrade pyenv/pyenv && pyenv install 3.9.6. Not trying to dismiss your workflow (if it is actually how you do it), just pointing why/how I find pyenv useful (because you seem to really dislike it without being using it 😅). As we all know, everybody has their own preferences. Something you find uselessly complex, I find usefully convenient.

[justasojourner]

(because you seem to really dislike it without being using it 😅).

It's not that I dislike it (think it's bad), it's just that I don't think it is relevant or necessary.

Again the goal for me with my setup is multiple (if needed) installations of the Python interpreter which are kept pristine (no packages installed, and up to date with point maintenance releases) and projects with their related packages kept self-contained and isolated from each other. The fact that with PDM I can achieve the project part without having virtual environments is the icing on the cake.

For example today I needed to make a quick project to do deduping of 126K customer account names - checking for similarity in the text of the name. That meant writing code to check each record against every other record = 14,829,637,729 permutations.

I just make a directory, pdm init, add pandas, and the data science package for doing string similarity matching using PDM, and run the code. Quick, simple, clean, small directory size, no virtual env - and I'm done.

Doing string similarity matching for a total of 14,829,637,729 permutations, this may take a while...

Similarity match processing on account name for 121,777 accounts completed in 13.9953 seconds

[pawamoy]

I do the same, except that my different Python versions are installed through pyenv 🙂
Quick and simple as well, no virtualenvs either!

[apirogov]

Just stumbled onto this thread researching about PDM, and feeling strongly driven to defend pyenv!

At least at my work place and many others, you can't just willy-nilly install system-wide packages. With pyenv (and nvm for JS) I thankfully can manage my toolchains myself. That is actually how our administrator prefers it to be - because he has more work otherwise. I have no admin rights on my work laptop.

[ForsakenRei]

Read a lot about pdm today, and I'm on the same pyenv boat as one of the projects we worked on was actually developed by another team and client didn't want us to move it to Python 3.9 that might have potential risks, so we need develop/test it on RHEL 9 where python 3.9 is the default installed python version. And pyenv saved my day, we don't have sudo on the server and IT is a mess so we just requested pyenv then everyone using the python version they need.

That's to say, ideally everything running on the server should work with the same python version, but the fact is a lot of time you just don't have that kind of control, or you don't want mess around with it. pyenv is great, but it doesn't mean everyone need it, if your work didn't need different python versions, or maybe you have full control of the environment that's great. But if you are out of luck from the previous two situations, pyenv might save you a lot of time.

[pawamoy]

PDM is closest to Poetry, and is based on PEP582 rather than using virtualenvs.
pyenv lets you install and choose Python versions, while miniconda lets you install a Python version directly in a virtualenv.
Both PDM and Poetry can be used with pyenv indirectly: you use pyenv to make your installed Python versions available on the PATH, then PDM and Poetry can find them.
Both PDM and Poetry can be used with miniconda indirectly as well: you prepare an environment with conda (choosing a Python version), then PDM and Poetry can detect that a virtualenv is activated (though using virtualenvs with PDM is kinda defeating its PEP582 support).

[justasojourner]

Both PDM and Poetry can be used with pyenv indirectly:

Sure, but why bother, pyenv is not necessary and it just adds another layer of complexity for no benefit.

you use pyenv to make your installed Python versions available on the PATH, then PDM and Poetry can find them.

Again why? If the (good) goal is to ensure that a Python project uses a specific version of Python, and one can do so without having work-arounds dynamically messing with the OS path then why use them?

Now, I am just getting started with PDM and testing it for a new project, so I don't profess to be an expert in it yet, but I can invoke Python scripts and have them use the correct interpreter and packages both within the IDE and anywhere on the command line without having to pre-run things like pyenv and/or initialise a virtual environment.

Here's the relevant part of my ENV.

PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
PYTHONPATH=/usr/local/bin/venv/lib/python3.9/site-packages/pdm/pep582

It just works - from the command line inside the project folder and anywhere on the hard drive. Interpreter + project packages correctly available (python3 is alias for python3.9, the latest version of Python on my Mac)

drp % python3 check_uk_vat.py

Company:
XXX BUSINESS SYSTEM LTD

Address:
UNIT XXXX
XXXXX POINT
XXXXXX DRIVE
KETTERING
XX16 9XX
GB
============

drp % cd ~/temp
temp % python3 ~/PycharmProjects/drp/check_uk_vat.py

Company:
XXX BUSINESS SYSTEM LTD

Address:
UNIT XXXX
XXXXX POINT
XXXXXX DRIVE
KETTERING
XX16 9XX
GB
============

[pawamoy]

@justasojourner just to make sure: I'm not arguing in favor of pyenv, just explaining how it can be used with PDM/Poetry 😄

So, I guess your questions can be formluated as "are there benefits to using pyenv?". I see a few answers to that.

The main one: you might not be an administrator on your machine, and therefore cannot install Python versions on the system itself. Sure, you can download the sources, compile them, and install the executables in a user folder. Well, that's exactly what pyenv does. Instead of doing all this manually, for each version you need, you can run only one command: pyenv install VERSION, which is quite convenient.

Second, pyenv allows to install multiple patches of a same minor version, which for example is not possible on Windows, and probably messy on Linux (you only get symlinks like python3.8 or python3.9, not python3.8.1 or python3.9.2). Having multiple patch versions of a same minor version might not be that useful in itself, but maybe it is in some cases (running tests on multiple patches?).

Third, the majority of Python tools still use/need virtualenvs. For example, pipx installs a tool into a dedicated virtualenv. If you run a system upgrade and your Python version is upgraded to the next minor version, this tool's virtualenv is now broken, and you have to recreate it. Using pyenv-installed Python versions instead gives you the control on when Python is upgraded, and would prevent such breakages. Though again, if you manage to install multiple Python versions on your system, they likely stay there when the system is upgraded, so pyenv is not always useful here either.

Finally, on some systems there might be no Python packages to install through the package manager? Then pyenv can save a lot of time. Same when you want to run Python code on a not-yet released version of Python (therefore not available through your package manager), like 3.11: pyenv allows you to install it, with necessary compilation tweaks preconfigured (if any). Or when you want to install other Python implementations/solutions: activepython, anaconda, graalpython, ironpython, jython, mambaforge, micropython, miniconda, miniforge, pypy, stackless, pyston... pyenv can manage all those.

As you can see there are multiple cases where pyenv is necessary and/or useful. But if you don't need it, sure, no point in using it 🙂

[justasojourner]

As always there are 'horses for courses', and everyone has personal preferences. Having said that I have a couple of responses:

The main one: you might not be an administrator on your machine, and therefore cannot install Python versions on the system itself.

If one was a developer and could not install Python versions on their development machine then I would consider that strange. Similarly, if a developer was blocked from installing a required version of Python on a production machine, then I would be asking about the system owners.

As regards versions the pyenv website https://github.com/pyenv/pyenv itself says (my emphasis):

Let you change the global Python version on a per-user basis.

and that is instructive because that is what pyenv seems to be doing, messing around with the path, running 'shims' etc. when, again, I would be saying - I want to run a particular project/app (with specific versions of packages) with a particular version of Python. And I want to do it in the simplest, cleanest, way possible. I do not consider virtual environments to be a 'good thing', I consider them to be a nasty hack.

Second, pyenv allows to install multiple patches of a same minor version

I would class that as a 'bad thing' ™. I do not believe that 3.9.2 is a 'version' of Python different to 3.9.6, there is only version 3.9. What I mean is that maintenance, bug fix, updates are just that. They are not versions. Any package maintainer and/or developer coding to specific things in maintenance updates should not be - because the Python developers could be fixing/changing those things at any time.

Third, the majority of Python tools still use/need virtualenvs.

Why? Isn't it actually the case that they are using virtualenvs simply because there was no better solution? Virtual environments are not an end in themselves, but a means to an end. If the problem virtualenvs solve goes away... then there is no more need for virtualenvs.

For example, pipx installs a tool into a dedicated virtualenv. If you run a system upgrade and your Python version is upgraded to the next minor version, this tool's virtualenv is now broken,

I would say the tool was broken before the upgrade if it was locked into a minor (3.9.x) version, which again is not a version - it's a maintenance fix. But, however, if we are saying it is now broken because the virtualenv is now out of date, then aren't we simply proving again that virtualenvs are bad.

Again though, it's horses for courses.

[pawamoy]

I agree with all your points.

Isn't it actually the case that they are using virtualenvs simply because there was no better solution?

Indeed. I warmly recommend PDM to Python developers, and to get rid of virtualenvs, but it'll take a long time before every Python project uses PEP582. Also, PEP582 might not happen for cross-language projects relying on things like Conda, though that's another story.

I would say the tool was broken before the upgrade if it was locked into a minor (3.9.x) version, which again is not a version

I did actually mean 3.9 (the minor version), not 3.9.x (the patch). Happened to me a few times on ArchLinux: Python was upgraded from 3.7 to 3.8, or 3.8 to 3.9. virtualenvs were broken and I had to recreate them. Not pipx's fault here, definitely virtualenvs'.

then aren't we simply proving again that virtualenvs are bad

Yes 😄

[mforbes]

Another difference between Poetry and PDM is that Poetry has chosen not to comply with PEP-621, which gives it additional flexibility. PDM is committed to upholding the standards, and so dropped support for dynamically updating trove categories, for example as Poetry does. Thus, when migrating or starting a new project that you plan on distributing on PyPI, you should include these in your pyproject.toml file:

# pyproject.toml
requires-python = ">=3.9"
license = {text = "BSD-2-Clause"}

# As per PEP 621, these cannot be dynamically updated so you
# must update them if you chance the license and requires-python data.
classifiers = [
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "License :: OSI Approved :: BSD License",
]

See also:

• Incompatibility with PEP 621 #748
• Update python and license classifiers when publishing. #1773