pip install ignores environment markers?

[mgedmin]

Under some circumstances pip install appears to ignore PEP 426 environment markers. So far I've only been able to reproduce this on Windows.

Here's the failing Jenkins job:

1. It uses tox to build an sdist of zodbbrowser and pip install it into a virtualenv.
2. zodbbrowser depends (i.e. install_requires) on ZODB3
3. ZODB3 depends on ZODB>=4.0.0dev
4. ZODB 4.2.0 has a conditional dependency on zodbpickle>=0.6.0 via extras_require[':python_version != "2.6"']
5. pip installs ZODB 4.2.0 but doesn't try to install zodbpickle

Excerpts from the pip install log:

Collecting ZODB>=4.0.0dev (from ZODB3->zodbbrowser==0.12.1.dev0)
  Using cached https://debesis.gedmin.as/wheels/ZODB-4.2.0-py2-none-any.whl

If you download https://debesis.gedmin.as/wheels/ZODB-4.2.0-py2-none-any.whl and unzip it, inside you'll find a ZODB-4.2.0.dist-info/METADATA with

Provides-Extra: :python_version != "2.6"
Requires-Dist: zodbpickle (>=0.6.0); extra == ':python_version != "2.6"'

and a metadata.json with

    "extras": [
        ":python_version != \"2.6\"",
        "test"
    ],
...
    "run_requires": [
        {
            "extra": ":python_version != \"2.6\"",
            "requires": [
                "zodbpickle (>=0.6.0)"
            ]
        },
...

so the requirement is not lost through the torturous sdist -> bdist_wininst -> wheel convert process.

Why is it being ignored?

(As I said, I cannot reproduce this on Linux, where zodbbrowser's tox succeeds.)

I've got pip version 7.0.3 and virtualenv version 13.0.3.

[rbtcollins]

    ':python_version != "2.6"': 'zodbpickle >= 0.6.0', 

thats invalid AIUI.

':python_version != "2.6"': ['zodbpickle >= 0.6.0'],

should be better. May be unrelated, but I've never seen extras that were string literals.

[mgedmin]

Fixed in zopefoundation/ZODB@aa94658, in case that matters.

I might've been too hasty with that commit: the setuptools documentation says

extras_require
A dictionary mapping names of “extras” (optional features of your project) to strings or lists of strings specifying what other distributions must be installed to support those features.

(emphasis mine). Also, the metadata.json in the wheel file had it as a list of strings.

[mgedmin]

I can reproduce this on Linux if I set PIP_FIND_LINKS=https://debesis.gedmin.as/wheels/.

Simpler steps to reproduce:

1. virtualenv /tmp/testenv
2. PIP_FIND_LINKS=https://debesis.gedmin.as/wheels /tmp/testenv/bin/pip install ZODB
3. /tmp/testenv/bin/python -c 'import ZODB'

This fails with ImportError: No module named zodbpickle.pickle.

[mgedmin]

I've unzipped the bad ZODB wheel from https://debesis.gedmin.as/wheels and compared it with the wheel I get if I let pip build it locally. There are some interesting differences:

--- bad/ZODB-4.2.0.dist-info/METADATA   2015-06-02 14:48:14.000000000 +0300
+++ good/ZODB-4.2.0.dist-info/METADATA  2015-06-04 09:27:36.000000000 +0300
@@ -34,11 +34,10 @@
 Requires-Dist: zc.lockfile
 Requires-Dist: zdaemon (>=4.0.0a1)
 Requires-Dist: zope.interface
+Requires-Dist: zodbpickle (>=0.6.0); python_version != "2.6"
 Provides-Extra: test
 Requires-Dist: zope.testing; extra == 'test'
 Requires-Dist: manuel; extra == 'test'
-Provides-Extra: :python_version != "2.6"
-Requires-Dist: zodbpickle (>=0.6.0); extra == ':python_version != "2.6"'

 The Zope Object Database provides an object-oriented database for
 Python that provides a high-degree of transparency. Applications can


--- bad/ZODB-4.2.0.dist-info/WHEEL      2015-06-02 14:48:14.000000000 +0300
+++ good/ZODB-4.2.0.dist-info/WHEEL     2015-06-04 09:27:36.000000000 +0300
@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: bdist_wheel (0.23.0)
+Generator: bdist_wheel (0.24.0)
 Root-Is-Purelib: true
 Tag: py2-none-any

Minified JSON is, of course, undiffable, but if I pipe it through json.tool, I get

--- bad/ZODB-4.2.0.dist-info/metadata.json      2015-06-02 14:48:14.000000000 +0300
+++ good/ZODB-4.2.0.dist-info/metadata.json     2015-06-04 09:27:36.000000000 +0300
@@ -19,62 +19,57 @@
         "Operating System :: Unix",
         "Framework :: ZODB"
     ],
-    "commands": {
-        "wrap_console": {
-            "fsdump": "ZODB.FileStorage.fsdump:main",
-            "fsoids": "ZODB.scripts.fsoids:main",
-            "fsrefs": "ZODB.scripts.fsrefs:main",
-            "fstail": "ZODB.scripts.fstail:Main",
-            "repozo": "ZODB.scripts.repozo:main"
-        }
-    },
-    "contacts": [
-        {
-            "email": "zodb-dev@zope.org",
-            "name": "Zope Foundation and Contributors",
-            "role": "author"
-        }
-    ],
-    "document_names": {
-        "description": "DESCRIPTION.rst"
-    },
-    "exports": {
-        "console_scripts": {
-            "fsdump": "ZODB.FileStorage.fsdump:main",
-            "fsoids": "ZODB.scripts.fsoids:main",
-            "fsrefs": "ZODB.scripts.fsrefs:main",
-            "fstail": "ZODB.scripts.fstail:Main",
-            "repozo": "ZODB.scripts.repozo:main"
+    "extensions": {
+        "python.commands": {
+            "wrap_console": {
+                "fsdump": "ZODB.FileStorage.fsdump:main",
+                "fsoids": "ZODB.scripts.fsoids:main",
+                "fsrefs": "ZODB.scripts.fsrefs:main",
+                "fstail": "ZODB.scripts.fstail:Main",
+                "repozo": "ZODB.scripts.repozo:main"
+            }
+        },
+        "python.details": {
+            "contacts": [
+                {
+                    "email": "zodb-dev@zope.org",
+                    "name": "Zope Foundation and Contributors",
+                    "role": "author"
+                }
+            ],
+            "document_names": {
+                "description": "DESCRIPTION.rst"
+            },
+            "project_urls": {
+                "Home": "http://www.zodb.org/"
+            }
+        },
+        "python.exports": {
+            "console_scripts": {
+                "fsdump": "ZODB.FileStorage.fsdump:main",
+                "fsoids": "ZODB.scripts.fsoids:main",
+                "fsrefs": "ZODB.scripts.fsrefs:main",
+                "fstail": "ZODB.scripts.fstail:Main",
+                "repozo": "ZODB.scripts.repozo:main"
+            }
         }
     },
     "extras": [
-        ":python_version != \"2.6\"",
         "test"
     ],
-    "generator": "bdist_wheel (0.23.0)",
-    "keywords": "database nosql python zope",
+    "generator": "bdist_wheel (0.24.0)",
+    "keywords": [
+        "database",
+        "nosql",
+        "python",
+        "zope"
+    ],
     "license": "ZPL 2.1",
     "metadata_version": "2.0",
     "name": "ZODB",
     "platform": "any",
-    "project_urls": {
-        "Home": "http://www.zodb.org/"
-    },
     "run_requires": [
         {
-            "extra": ":python_version != \"2.6\"",
-            "requires": [
-                "zodbpickle (>=0.6.0)"
-            ]
-        },
-        {
-            "extra": "test",
-            "requires": [
-                "zope.testing",
-                "manuel"
-            ]
-        },
-        {
             "requires": [
                 "persistent (>=4.1.0)",
                 "BTrees (>=4.1.3)",
@@ -85,6 +80,19 @@
                 "zdaemon (>=4.0.0a1)",
                 "zope.interface"
             ]
+        },
+        {
+            "environment": "python_version != \"2.6\"",
+            "requires": [
+                "zodbpickle (>=0.6.0)"
+            ]
+        },
+        {
+            "extra": "test",
+            "requires": [
+                "zope.testing",
+                "manuel"
+            ]
         }
     ],
     "summary": "Zope Object Database: object database and persistence",

Conclusion: bdist_wheel 0.23.0 did not support environment markers and I need to upgrade.

[dholth]

Lamentably older versions of bdist_wheel supported environment markers specified in a bdist_wheel-specific setup.cfg section but not those from setup.py. That is because bdist_wheel predates the environment-markers-in-setup.py feature.

[mgedmin]

When was the environment-markers-in-setup.py feature introduced? The only environment-marker-related thing in setuptools changelog is for 0.7, which happened ages ago.