Dependencies specified in extra_requires don't work from 1.5.0 onwards

[mridsole]

Example:

setup.py:

from setuptools import setup, find_packages

setup(
    name = 'bad-pex-example',
    version = '0.1',
    python_requires = '>=3.5',
    install_requires = [],
    extras_require = { 'master': [ 'schema>=0.6' ] }
)

bad/__init__.py:

import schema

def entry():
    print('it works')

In version 1.4.9, pex -o bad.pex -e 'bad:entry' '.[master]' produces a working executable. In all later versions (up to current latest 1.6.7) the resulting executable fails to import the dependency:

Traceback (most recent call last):
  File ".bootstrap/pex/pex.py", line 352, in execute
  File ".bootstrap/pex/pex.py", line 284, in _wrap_coverage
  File ".bootstrap/pex/pex.py", line 315, in _wrap_profiling
  File ".bootstrap/pex/pex.py", line 397, in _execute
  File ".bootstrap/pex/pex.py", line 495, in execute_entry
  File ".bootstrap/pex/pex.py", line 509, in execute_pkg_resources
  File ".bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2384, in resolve
  File "/home/david/work/bad-pex-example/bad/__init__.py", line 1, in <module>
    import schema
ModuleNotFoundError: No module named 'schema'

pex does still get the dependency and does something with it:

❯ pex -o bad.pex -e 'bad:entry' '.[master]' -vvvv
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8be4908>       
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8be4240>
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8b982b0>                                                                                         
pex: crawling link i=0 link=Link('file:///home/david/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.org/simple/schema/') follow_links=False
pex: Building pex :: Resolving distributions :: Fetching file:///home/david/.pex/build/schema-0.7.0-py2.py3-none-any.whl :: Fetching file:///home/david/.pex/build/schema-0.7.0-py2.ppex: Building pex :: Resolving distributions :: Translating /tmp/tmpa2mbu1dq/schema-0.7.0-py2.py3-none-any.whl into distribution                                                     pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f83f8374208>                                    
pex: crawling link i=0 link=Link('https://pypi.org/simple/contextlib2/') follow_links=False
pex: Building pex :: Resolving distributions :: Fetching file:///home/david/.pex/build/contextlib2-0.5.5-py2.py3-none-any.whl :: Fetching file:///home/david/.pex/build/contextlib2-0pex: Building pex :: Resolving distributions :: Translating /tmp/tmpf5_mo387/contextlib2-0.5.5-py2.py3-none-any.whl into distribution                                                  bad-pex-example==0.1 -> bad-pex-example 0.1
  schema>=0.6; extra == "master" -> schema 0.7.0
  contextlib2==0.5.5 -> contextlib2 0.5.5
pex: Building pex: 470.8ms                                                                                                           
pex:   Resolving interpreters: 0.0ms
pex:   Installing /tmp/tmp1zk6gp55: 209.2ms
pex:     Isolating pex in Chroot(/tmp/tmp6s9psqsi {fs:}): 15.1ms
pex:   Resolving distributions: 260.6ms
pex:     Fetching file:///tmp/tmp1zk6gp55/bad-pex-example-0.1.tar.gz: 0.5ms
pex:       Fetching file:///tmp/tmp1zk6gp55/bad-pex-example-0.1.tar.gz: 0.1ms
pex:     Translating /tmp/tmppx_oldwr/bad-pex-example-0.1.tar.gz into distribution: 169.7ms
pex:       Packaging bad-pex-example: 168.0ms
pex:         Installing /tmp/tmpuosc6ijy: 166.6ms
pex:     Fetching file:///home/david/.pex/build/schema-0.7.0-py2.py3-none-any.whl: 0.4ms
pex:       Fetching file:///home/david/.pex/build/schema-0.7.0-py2.py3-none-any.whl: 0.2ms
pex:     Translating /tmp/tmpa2mbu1dq/schema-0.7.0-py2.py3-none-any.whl into distribution: 0.8ms
pex:     Fetching file:///home/david/.pex/build/contextlib2-0.5.5-py2.py3-none-any.whl: 1.2ms
pex:       Fetching file:///home/david/.pex/build/contextlib2-0.5.5-py2.py3-none-any.whl: 0.4ms
pex:     Translating /tmp/tmpf5_mo387/contextlib2-0.5.5-py2.py3-none-any.whl into distribution: 1.9ms
pex: Vendoring pkg_resources from VendorSpec(key='setuptools', version='40.6.2') @ /home/david/.local/lib/python3.6/site-packages/pex/vendor/_vendored/setuptools
Saving PEX file to bad.pex

and schema still does end up in ~/.pex/install after running the executable:

❯ ls -lah ~/.pex/install
total 20K
drwxrwxr-x 5 david david 4.0K Jun 17 11:42 .
drwxrwxr-x 3 david david 4.0K Jun 17 11:42 ..
drwxrwxr-x 4 david david 4.0K Jun 17 11:42 bad_pex_example-0.1-py3-none-any.whl.09110f078728861fa50039d90343b57aebf726ca
drwxrwxr-x 4 david david 4.0K Jun 17 11:42 contextlib2-0.5.5-py2.py3-none-any.whl.edad6efe1aebf29bea2d8a9e8dab93fbbc2555bc
drwxrwxr-x 4 david david 4.0K Jun 17 11:42 schema-0.7.0-py2.py3-none-any.whl.5b7491a4df1576f5f2912f4158d096cd06d90199

Many thanks!

[jsirois]

I repro after:

$ diff -u /tmp/setup.py setup.py 
--- /tmp/setup.py	2019-06-22 15:38:38.599110687 -0600
+++ setup.py	2019-06-22 15:08:11.585058403 -0600
@@ -4,6 +4,7 @@
     name = 'bad-pex-example',
     version = '0.1',
     python_requires = '>=3.5',
+    packages = ['bad'],
     install_requires = [],
     extras_require = { 'master': [ 'schema>=0.6' ] }
 )

The bug was introduced in #582 here:
https://github.com/pantsbuild/pex/blob/fb35498778bf722a33133ed351cea5b47a523c39/pex/resolver.py#L327

The requirement string returned by a distribution for itself will never contain extras that were asked for when installing it. We have that information though in the resolvable the distribution was created from, so this is a one line fix with more lines to add a test.