Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pex 1.0.0 fails to resolve deps that appear in multiple resolve rounds when using the cache #120

Closed
jsirois opened this issue Jun 10, 2015 · 3 comments
Closed

pex 1.0.0 fails to resolve deps that appear in multiple resolve rounds when using the cache #120

jsirois opened this issue Jun 10, 2015 · 3 comments
Assignees
@jsirois
Labels
bug

Comments

@jsirois
Copy link
Member

jsirois commented Jun 10, 2015

I discovered this trying to upgrade pants from the 0.8 pex series to 1.0.0.
A minimal repro using twitter.common.collections which has a single dependency on twitter.common.lang follows:

The failing case with 1st resolve round including [twitter.common.collections, twitter.common.lang] and the second round including [twitter.common.lang]:

jsirois@gill ~ $ pex --version
pex 1.0.0
jsirois@gill ~ $ rm -rf ~/.pex/
jsirois@gill ~ $ pex -vvv twitter.common.collections twitter.common.lang -o example.pex
/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
pex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befapex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befapex: Validated wheel-0.24.0.tar.gz (md5=3b0d66f0d127ea8befaa5d11453107fd)                                                                                                                                                                                                                                                                                                                                                         
pex: Validated wheel-0.24.0.tar.gz (md5=3b0d66f0d127ea8befaa5d11453107fd)
pex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8bepex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8bepex: Building pex :: Resolving distributions                                                                                                                                                                                                                                                  /home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
pex: Validated twitter.common.collections-0.3.3.tar.gz (md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f)                                                                                                             
pex: Validated twitter.common.collections-0.3.3.tar.gz (md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f)
pex: Validated twitter.common.lang-0.3.3.tar.gz (md5=d4f9ad4fc0f1974570d575f25713ebe6)                                                                                                                              
pex: Validated twitter.common.lang-0.3.3.tar.gz (md5=d4f9ad4fc0f1974570d575f25713ebe6)
Could not satisfy all requirements for twitter.common.lang: /tmp/tmpq6cetK/twitter.common.lang-0.3.3.tar.gz into distribution :: Packaging twitter.common.lang :: Installing /tmp/tmpbxXozQ           
    twitter.common.lang, twitter.common.lang==0.3.3(from: twitter.common.collections)
^jsirois@gill ~ $

As opposed to 2 round that do not have overlapping deps, ie round 1 [twitter.common.collections], round 2 [twitter.common.lang]:

jsirois@gill ~ $ rm -rf ~/.pex/
jsirois@gill ~ $ pex -vvv twitter.common.collections -o example.pex
/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
pex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befapex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befapex: Validated wheel-0.24.0.tar.gz (md5=3b0d66f0d127ea8befaa5d11453107fd)                                                                                                                                                                                                                                                                                                                                                         
pex: Validated wheel-0.24.0.tar.gz (md5=3b0d66f0d127ea8befaa5d11453107fd)
pex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8bepex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8bepex: Building pex :: Resolving distributions                                                                                                                                                                                                                                                  /home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
pex: Validated twitter.common.collections-0.3.3.tar.gz (md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f)                                                                                                             
pex: Validated twitter.common.collections-0.3.3.tar.gz (md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f)
pex: Validated twitter.common.lang-0.3.3.tar.gz (md5=d4f9ad4fc0f1974570d575f25713ebe6)                                                                                                                              
pex: Validated twitter.common.lang-0.3.3.tar.gz (md5=d4f9ad4fc0f1974570d575f25713ebe6)
  twitter.common.lang 0.3.3ing distributions :: Translating /tmp/tmpKIVBtn/twitter.common.lang-0.3.3.tar.gz into distribution :: Packaging twitter.common.lang :: Installing /tmp/tmpnIAgGX           
  twitter.common.collections 0.3.3
pex: Building pex: 1294.3ms                                                                                                                                                                
pex:   Resolving interpreter: 653.6ms
pex:     Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python: 653.6ms
pex:       Interpreter cache resolving wheel<0.25.0,>=0.24.0: 403.1ms
pex:         Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befaa5d11453107fd'): 270.4ms
pex:           Fetching https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befaa5d11453107fd: 269.0ms
pex:         Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befaa5d11453107fd'): 132.5ms
pex:           Installing /tmp/tmpiaylJN: 124.5ms
pex:   Resolving distributions: 636.5ms
pex:     Fetching https://pypi.python.org/packages/source/t/twitter.common.collections/twitter.common.collections-0.3.3.tar.gz#md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f: 64.1ms
pex:     Fetching file:///home/jsirois/.pex/build/twitter.common.collections-0.3.3.tar.gz: 0.3ms
pex:       Fetching file:///home/jsirois/.pex/build/twitter.common.collections-0.3.3.tar.gz: 0.1ms
pex:     Translating /tmp/tmpEs5eu8/twitter.common.collections-0.3.3.tar.gz into distribution: 114.4ms
pex:       Packaging twitter.common.collections: 111.2ms
pex:         Installing /tmp/tmp3PLIYG: 109.2ms
pex:     Fetching https://pypi.python.org/packages/source/t/twitter.common.lang/twitter.common.lang-0.3.3.tar.gz#md5=d4f9ad4fc0f1974570d575f25713ebe6: 55.5ms
pex:     Fetching file:///home/jsirois/.pex/build/twitter.common.lang-0.3.3.tar.gz: 0.6ms
pex:       Fetching file:///home/jsirois/.pex/build/twitter.common.lang-0.3.3.tar.gz: 0.2ms
pex:     Translating /tmp/tmpKIVBtn/twitter.common.lang-0.3.3.tar.gz into distribution: 109.1ms
pex:       Packaging twitter.common.lang: 104.9ms
pex:         Installing /tmp/tmpnIAgGX: 104.0ms
Saving PEX file to example.pex

Note though that the multi-round with repeated dep case does work with the cache turned off:

jsirois@gill ~ $ rm -rf ~/.pex/
jsirois@gill ~ $ pex -vvv --disable-cache twitter.common.collections twitter.common.lang -o example.pex
/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
pex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befapex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befapex: Validated wheel-0.24.0.tar.gz (md5=3b0d66f0d127ea8befaa5d11453107fd)                                                                                                                                                                                                                                                                                                                                                         
pex: Validated wheel-0.24.0.tar.gz (md5=3b0d66f0d127ea8befaa5d11453107fd)
pex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8bepex: Building pex :: Resolving interpreter :: Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python :: Interpreter cache resolving wheel<0.25.0,>=0.24.0 :: Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8bepex: Building pex :: Resolving distributions                                                                                                                                                                                                                                                  /home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
pex: Building pex :: Resolving distributions :: Fetching https://pypi.python.org/packages/source/t/twitter.common.collections/twitter.common.collections-0.3.3.tar.gz#md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f :: Fetching https://pypi.python.org/packages/source/t/twitter.common.collections/twpex: Validated twitter.common.collections-0.3.3.tar.gz (md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f)                                                                                                                                                                                                                                                                           
pex: Validated twitter.common.collections-0.3.3.tar.gz (md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f)
pex: Building pex :: Resolving distributions :: Fetching https://pypi.python.org/packages/source/t/twitter.common.lang/twitter.common.lang-0.3.3.tar.gz#md5=d4f9ad4fc0f1974570d575f25713ebe6 :: Fetching https://pypi.python.org/packages/source/t/twitter.common.lang/twitter.common.lang-0.3pex: Validated twitter.common.lang-0.3.3.tar.gz (md5=d4f9ad4fc0f1974570d575f25713ebe6)                                                                                                                                                                                                                                                      
pex: Validated twitter.common.lang-0.3.3.tar.gz (md5=d4f9ad4fc0f1974570d575f25713ebe6)
  twitter.common.lang 0.3.3ing distributions :: Translating /tmp/tmpQ1eZhv/twitter.common.lang-0.3.3.tar.gz into distribution :: Packaging twitter.common.lang :: Installing /tmp/tmpg5vN8z
  twitter.common.collections 0.3.3
pex: Building pex: 1349.9ms                                                                                                                                                                
pex:   Resolving interpreter: 712.8ms
pex:     Setting up interpreter /home/jsirois/.pyenv/versions/pex/bin/python: 712.8ms
pex:       Interpreter cache resolving wheel<0.25.0,>=0.24.0: 403.7ms
pex:         Fetching SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befaa5d11453107fd'): 270.9ms
pex:           Fetching https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befaa5d11453107fd: 269.6ms
pex:         Installing SourcePackage(u'https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz#md5=3b0d66f0d127ea8befaa5d11453107fd'): 132.5ms
pex:           Installing /tmp/tmpBB31J4: 124.4ms
pex:   Resolving distributions: 632.6ms
pex:     Fetching https://pypi.python.org/packages/source/t/twitter.common.collections/twitter.common.collections-0.3.3.tar.gz#md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f: 62.5ms
pex:       Fetching https://pypi.python.org/packages/source/t/twitter.common.collections/twitter.common.collections-0.3.3.tar.gz#md5=902ebdb7e3daa5d6a5c33f5cc6ea5f8f: 62.1ms
pex:     Translating /tmp/tmpDgFCAa/twitter.common.collections-0.3.3.tar.gz into distribution: 108.6ms
pex:       Packaging twitter.common.collections: 104.9ms
pex:         Installing /tmp/tmpACpPAa: 103.9ms
pex:     Fetching https://pypi.python.org/packages/source/t/twitter.common.lang/twitter.common.lang-0.3.3.tar.gz#md5=d4f9ad4fc0f1974570d575f25713ebe6: 52.3ms
pex:       Fetching https://pypi.python.org/packages/source/t/twitter.common.lang/twitter.common.lang-0.3.3.tar.gz#md5=d4f9ad4fc0f1974570d575f25713ebe6: 52.0ms
pex:     Translating /tmp/tmpQ1eZhv/twitter.common.lang-0.3.3.tar.gz into distribution: 104.6ms
pex:       Packaging twitter.common.lang: 101.8ms
pex:         Installing /tmp/tmpg5vN8z: 100.9ms
Saving PEX file to example.pex
@jsirois jsirois added the bug label Jun 10, 2015
@jsirois jsirois self-assigned this Jun 10, 2015
@jsirois
Copy link
Member Author

jsirois commented Jun 10, 2015

I have a fix for this that I'm polishing for review as part of work on pantsbuild/pants#1440

@jsirois jsirois mentioned this issue Jun 10, 2015
Closed
@jsirois
Copy link
Member Author

jsirois commented Jun 10, 2015

This is addressed by: https://rbcommons.com/s/twitter/r/2341/

@jsirois jsirois mentioned this issue Jun 11, 2015
Closed
@jsirois
Copy link
Member Author

jsirois commented Jul 3, 2015

Fix submitted @ b1ea8a7

@jsirois jsirois closed this as completed Jul 3, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsirois jsirois

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jsirois