How to use environment markers correctly to install conditional dependencies

[lnielsen]

Problem

You have a module which is only required on Python 2 and not on Python 3. Example functools32 is only needed on Python 2.

Solution

1. Do ensure that you have the latest versions of pip, setuptools and wheel:

$ pip install pip setuptools wheel --upgrade

2. Use environment markers in extras_require in setup.py:

# GOOD
from setuptools import setup

setup(
    install_requires=[
        # ...
    ],
    extras_require={
        ':python_version=="2.7"': [
            'functools32>=3.2.3-2'
        ]
    }
)

# BAD
from setuptools import setup

setup(
    install_requires=[
        'functools32>=3.2.3-2; python_version="2.7"'
    ],
)

Why should I not use the method from the bad example?

The bad example is a valid method for specifying environment marker according to PEP508, so why not use it?

When you build a wheel using python setup.py bdist_wheel then wheel package will simply strip off the environment marker. Then, when you try to install the just built wheel package using pip or setuptools, the environment marker is not available, and pip/setuptools will consequently try to install functools32 on Python 3.

[diegodelemos]

I am having a problem regarding environment markers, invenio-pidstore is installing enum34 for Python versions which are not supposed to have it as a dependency. See invenio-pidstore/setup.py and travis log.

Note that the syntax for the environment marker is correct and travis is running pip install pip setuptools --upgrade (wheel is missing but I am including it when trying locally and it is installing enum34 anyways for Python 3.5 and 3.6). I am missing something?

[lnielsen]

I tried downloading the PIDStore wheel package and look at the package info at it actually looks correct (environment marker is still there):

...
{
        "environment": "python_version<\"3.4\"",
        "requires": ["enum34 (>=1.0.4)"]
    },
...

I think the problem might be related with the all extras that it doesn't take the marker into account?

[lnielsen]

Yep, just verified:

(pidstore3) lnielsen-mbp:Downloads lnielsen$ pip install invenio-pidstore --pre
...
Successfully installed Babel-2.3.4 Flask-0.12 Flask-BabelEx-0.9.3 Jinja2-2.9.5 MarkupSafe-1.0 Werkzeug-0.12 click-6.7 invenio-pidstore-1.0.0b1 itsdangerous-0.24 pytz-2016.10 speaklater-1.3

No enum34, but when you do (pip install invenio-pidstore[all]):

(pidstore3) lnielsen-mbp:Downloads lnielsen$ pip install invenio-pidstore[all] --pre
...
Collecting enum34>=1.0.4; extra == "all" (from invenio-pidstore[all])
  Downloading enum34-1.1.6-py3-none-any.whl
....

So the problem is with these lines in setup.py:

extras_require['all'] = []
for name, reqs in extras_require.items():
    if name in ('mysql', 'postgresql', 'sqlite'):
        continue
    extras_require['all'].extend(reqs)

Not sure what a nice solution to this is. The quick fix for PIDStore is to change it into something like:

extras_require['all'] = []
for name, reqs in extras_require.items():
    if name in ('mysql', 'postgresql', 'sqlite') or name.startswith(':'):
        continue
    extras_require['all'].extend(reqs)

But it won't work generically as you can also have something like this:

extra_require = {
   'test:python_version<"3.4"' : [ 
       # ... 
    ]
}

[Zac-HD]

When you build a wheel using python setup.py bdist_wheel then wheel package will simply strip off the environment marker [in install_requires].

Note that this is true for setuptools < 36.2.1; since then the marker is preserved correctly. However old versions of setuptools only support conditional dependencies in extras_require, so I'd also advise using extras_require instead if you want your package to be installable with old Python environments.

See HypothesisWorks/hypothesis#1091 for discussion.

[wimglenn]

Hmm, I don't get it. What do you mean by this line:

... wheel package will simply strip off the environment marker

Strip it off where & when? I have a library where I use what is described above as the "bad example", it has a dependency on mock conditional on older Python version:

install_requires=["mock ; python_version<'3.3'"]

Here is the relevant line in setup.py.

That generates correct line in pkg info like this, with environment marker intact:

Requires-Dist: mock; python_version < "3.3"

And the wheel on PyPI has the correct metadata, the condition is even listed right there in the HTML on the landing page (ctrl+f for "Requires Distributions").

The above was .dist-info but I checked and python setup.py bdist_wheel also generates correct .egg-info condition, a requires.txt file with environment marker like this:

[:python_version < "3.3"]
mock

So, why exactly is bad example bad? Please help me understand..

[Zac-HD]

"Bad example" is only bad if your packaging toolchain (pip, setuptools and wheel) are out of date.

Your tools are up-to-date, and so it works! Unfortunately, your users might not be so diligent...

[wimglenn]

But users install the .whl, they don't build .whl. And if you want to protect against out of date tools in executing setup.py, you can just check the version of setuptools directly in there, to force anyone packaging to upgrade before building a release.

It sounds like you trying to say there a version of pip/setuptools which will correctly read the conditional deps from an existing wheel if it was written like the "GOOD example" and it will incorrectly install the conditional deps when it was written like the "BAD example", correct me if that's wrong.

It's written "wheel package will simply strip off the environment marker", but I don't see that, is there an old version of wheel where that happened? If so, could it be mentioned in the original post the version specifier where wheel has such a problem?

[lnielsen]

@wimglenn This post stems from tons of failed Travis builds that we traced back to Wheel stripping of the environment marker when written as in the bad example above. At the time of the post (i.e March last year), I can confirm that wheel definitely was stripping off the environment marker when you built the package back. Unfortunately, I can tell you which version of Wheel it was except that it was the most recent version at the time of writing (March 2017).

If Wheel has been fixed in the meantime, then that's just great and we might start using the bad example again - however that just wasn't the case in March 2017, hence this post to share the experience with our fellow Invenio developers.

[srinivas-kandula]

I am having issue while putting conditional dependency for importlib. I have below snippet in my setup.py
install_requires = ['docopt','requests', 'boto', 'importlib; python_version<"3.3"'] .
After building the package i see that conditional import statement got added to the requires.txt file
docopt requests boto importlib; python_version<"3.3"
However when installing using pip3 it complains
Could not find a version that satisfies the requirement importlib (from ) (from versions: ) No matching distribution found for importlib (from )