Error during installing vendored packages

[EmPiFree]

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?
cf version 6.47.1+0921a21b7.2019-10-18

What version of the buildpack you are using?
https://github.com/cloudfoundry/python-buildpack
v1.7.1

If you were attempting to accomplish a task, what was it you were attempting to do?
Installling vendored dependencies

What did you expect to happen?
Successfully install all vendored dependencies

What was the actual behavior?
Erorr during installing vendored dependencies:

-----> Python Buildpack version 1.7.1
   -----> Supplying Python
   -----> Installing python 3.8.0
          Copy [/tmp/cache/final/dependencies/cabbf104215db0e7ded4b2d65b3de8abfed4bb146a4cd61eacda7dbcc561925a/python-3.8.0-linux-x64-cflinuxfs3-6af4e503.tgz]
   -----> Installing pip-pop 0.1.3
          Copy [/tmp/cache/final/dependencies/3b86fb2cd9c13268f8858fd0ccb2f9eaf11295a51804b7075fe38d7e2a471191/pip-pop-0.1.4-0a3b0f1b.tar.gz]
          Generating 'requirements.txt' from Pipfile.lock
   -----> Running Pip Install
          Using the pip --no-build-isolation flag since it is available
          Looking in links: file:///tmp/app/vendor
          Collecting attrs==19.3.0 (from -r /tmp/app/requirements.txt (line 2))
              Preparing wheel metadata: started
              Preparing wheel metadata: finished with status 'error'
              ERROR: Command errored out with exit status 1:
               command: /tmp/contents687858878/deps/0/bin/python /tmp/contents687858878/deps/0/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_h2bnk_n
                   cwd: /tmp/pip-install-d90u0avf/attrs
              Complete output (14 lines):
              running dist_info
              creating /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info
              writing /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/PKG-INFO
              writing dependency_links to /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/dependency_links.txt
              writing requirements to /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/requires.txt
              writing top-level names to /tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/top_level.txt
              writing manifest file '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/SOURCES.txt'
              reading manifest file '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/SOURCES.txt'
              reading manifest template 'MANIFEST.in'
              no previously-included directories found matching 'docs/_build'
              warning: no previously-included files found matching 'changelog.d/*.rst'
              writing manifest file '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.egg-info/SOURCES.txt'
              creating '/tmp/pip-install-d90u0avf/attrs/pip-wheel-metadata/attrs.dist-info'
              error: invalid command 'bdist_wheel'
              ----------------------------------------
          ERROR: Command errored out with exit status 1: /tmp/contents687858878/deps/0/bin/python /tmp/contents687858878/deps/0/python/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_h2bnk_n Check the logs for full command output.
          Running pip install without indexes failed. Not all dependencies were vendored. Trying again with indexes.
Collecting attrs==19.3.0 (from -r /tmp/app/requirements.txt (line 9))
            Downloading https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl

So the build first tries to install from file:///tmp/app/vendor after one dependency fails, it reads the requirements.txt and download the package, which then works correctly...

I think it's an error according the pip version:

WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
          You should consider upgrading via the 'pip install --upgrade pip' command.

On my local machine with pip version 19.3.1 it works, when I run pip download -r requirements.txt --no-binary=:all: -d vendor. However it failed when I downgrade to the same version as in the buildpack (19.2.3). Even after I updated wheel to 0.33.6 and setuptools to 41.6.0 (was 41.2.0 before).

So the error could be solved if we try to upgrade pip to the newest version. However I don't know if the process with pip download -r requirements.txt --no-binary=:all: -d vendor is the same as the process the python buildpack does, when installing from vendor directory

Please confirm where necessary:

• I have included a log output
• My log includes an error message
• I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/169730915

The labels on this github issue will be updated when the story is started.

[ryanmoran]

Hi @EmPiFree. We've spent a bit of time today reproducing this and we think there are a couple of things that might be going wrong.

We are able to reproduce this issue if we include a requirements.txt file that includes wheel and then also use the --no-binary=:all: option when running pip install to vendor our dependencies. We believe that the issue is that wheel itself provides a command to setup_tools called bdist_wheel that is not included in the "binary-less" vendored copy of wheel. This means that when pip goes to install attrs in your application, it finds no bdist_wheel binary on the system and fails. The subsequent installation process falls back to ignoring the vendor directory and re-downloading all of your dependencies which ultimately works, but is not ideal.

We don't think you should be including wheel in your requirements.txt, but we also aren't sure what you have in your application source code. Could you please provide us with a Pipfile, Pipfile.lock, and requirements.txt for your app so that we can do some more specific reproductions?

Cheers!

[ForestEckhardt]

We're gonna close this as inactive. Please feel free to re-open if you have further issues.

[Bijaya196]

requests 2.21.0 has requirement idna<2.9,>=2.5, but you'll have idna 2.10 which is incompatible.
django-image-cropping 1.1.0 has requirement django-appconf==1.0.1, but you'll have django-appconf 1.0.4 which is incompatible.
Installing collected packages: appdirs, asn1crypto, soupsieve, beautifulsoup4, certifi, pycparser, cffi, chardet, urllib3, idna, requests, uritemplate, itypes, MarkupSafe, Jinja2, coreschema, coreapi, six, cryptography, defusedxml, distlib, dj-database-url, pytz, Django, django-admin-rangefilter, django-admin-sortable2, django-appconf, django-autocomplete-light, django-autoslug, django-ipware, django-axes, django-cors-headers, django-dynamic-raw-id, elasticsearch, python-dateutil, elasticsearch-dsl, django-elasticsearch-dsl, django-nine, djangorestframework, django-elasticsearch-dsl-drf, typing, django-extensions, django-filter, URLObject, django-spurl, django-folders, django-grappelli, django-hotspots, django-image-cropping, django-mptt, django-multiupload, django-parler, django-parler-rest, django-tinymce, python-magic, django-upload-validator, django-waffle, waitress, WebOb, WebTest, django-webtest, lxml, docx-mailmerge, olefile, Pillow, easy-thumbnails, future, eight, et-xmlfile, text-unidecode, Faker, factory-boy, feincms-cleanse, filelock, maxminddb, geoip2, gunicorn, hsaudiotag3k, webencodings, setuptools, html5lib, http-ece, httplib2, typing-extensions, zipp, importlib-metadata, importlib-resources, jdcal, oauthlib, requests-oauthlib, lti, mimesis, openpyxl, polib, psycopg2, py-vapid, PyJWT, pyOpenSSL, PyPDF2, pywebpush, radar, raven, reportlab, rules, signxml, virtualenv, whitenoise, xhtml2pdf, xmltodict, pyenchant, more-itertools, atomicwrites, pluggy, attrs, py, pytest, pytest-django, inflection, pytest-factoryboy, pytest-mock, pyparsing, packaging, termcolor, pytest-sugar, freezegun
The script chardetect is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for itypes: started
Running setup.py install for itypes: finished with status 'done'
Running setup.py install for coreschema: finished with status 'done'
The script django-admin is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for django-admin-sortable2: started
Running setup.py install for django-admin-sortable2: finished with status 'done'
Running setup.py install for django-autocomplete-light: started
Running setup.py install for django-autocomplete-light: finished with status 'done'
Running setup.py install for django-ipware: started
Running setup.py install for django-ipware: finished with status 'done'
Running setup.py install for URLObject: started
Running setup.py install for URLObject: finished with status 'done'
Running setup.py develop for django-folders
Running setup.py install for django-grappelli: started
Running setup.py install for django-grappelli: finished with status 'done'
Running setup.py develop for django-hotspots
Running setup.py install for django-image-cropping: started
Running setup.py install for django-image-cropping: finished with status 'done'
Running setup.py install for django-multiupload: started
Running setup.py install for django-multiupload: finished with status 'done'
Running setup.py install for django-upload-validator: started
Running setup.py install for django-upload-validator: finished with status 'done'
The script waitress-serve is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for olefile: started
Running setup.py install for olefile: finished with status 'done'
Running setup.py install for easy-thumbnails: started
Running setup.py install for easy-thumbnails: finished with status 'done'
Running setup.py install for future: started
Running setup.py install for future: finished with status 'done'
Running setup.py install for et-xmlfile: started
Running setup.py install for et-xmlfile: finished with status 'done'
The script faker is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for feincms-cleanse: started
Running setup.py install for feincms-cleanse: finished with status 'done'
Running setup.py install for maxminddb: started
Running setup.py install for maxminddb: finished with status 'done'
The scripts gunicorn and gunicorn_paster are installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for hsaudiotag3k: started
Running setup.py install for hsaudiotag3k: finished with status 'done'
Running setup.py install for http-ece: started
Running setup.py install for http-ece: finished with status 'done'
Running setup.py install for mimesis: started
Running setup.py install for mimesis: finished with status 'done'
Running setup.py install for openpyxl: started
Running setup.py install for openpyxl: finished with status 'done'
Running setup.py install for py-vapid: started
The script pyjwt is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for PyPDF2: started
Running setup.py install for PyPDF2: finished with status 'done'
Running setup.py install for pywebpush: started
Running setup.py install for pywebpush: finished with status 'done'
Running setup.py install for radar: started
Running setup.py install for radar: finished with status 'done'
The script raven is installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for rules: started
Running setup.py install for rules: finished with status 'done'
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for xhtml2pdf: started
Running setup.py install for xhtml2pdf: finished with status 'done'
The scripts py.test and pytest are installed in '/tmp/contents617317607/deps/0/python/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for pytest-factoryboy: started
Running setup.py install for pytest-factoryboy: finished with status 'done'
Running setup.py install for termcolor: started
Running setup.py install for termcolor: finished with status 'done'
Successfully installed Django-1.11.29 Faker-5.0.2 Jinja2-2.11.2 MarkupSafe-1.1.1 Pillow-4.3.0 PyJWT-1.7.1 PyPDF2-1.26.0 URLObject-2.4.3 WebOb-1.8.6 WebTest-2.0.35 appdirs-1.4.4 asn1crypto-1.4.0 atomicwrites-1.4.0 attrs-20.3.0 beautifulsoup4-4.9.3 certifi-2017.11.5 cffi-1.14.4 chardet-3.0.4 coreapi-2.3.3 coreschema-0.0.4 cryptography-2.7 defusedxml-0.6.0 distlib-0.3.1 dj-database-url-0.5.0 django-admin-rangefilter-0.3.6 django-admin-sortable2-0.6.21 django-appconf-1.0.4 django-autocomplete-light-3.2.10 django-autoslug-1.9.8 django-axes-4.5.4 django-cors-headers-2.1.0 django-dynamic-raw-id-2.4 django-elasticsearch-dsl-0.4.5 django-elasticsearch-dsl-drf-0.7.2 django-extensions-1.9.9 django-filter-1.1.0 django-folders django-grappelli-2.10.4 django-hotspots django-image-cropping-1.1.0 django-ipware-2.1.0 django-mptt-0.8.7 django-multiupload-0.5.2 django-nine-0.1.13 django-parler-1.9.2 django-parler-rest-2.0 django-spurl-0.6.7 django-tinymce-2.7.1 django-upload-validator-1.0.2 django-waffle-0.14.0 django-webtest-1.9.7 djangorestframework-3.7.7 docx-mailmerge-0.4.0 easy-thumbnails-2.7.1 eight-0.4.2 elasticsearch-5.5.3 elasticsearch-dsl-5.4.0 et-xmlfile-1.0.1 factory-boy-3.2.0 feincms-cleanse-8 filelock-3.0.12 freezegun-0.3.11 future-0.16.0 geoip2-2.9.0 gunicorn-19.9.0 hsaudiotag3k-1.1.3.post1 html5lib-1.0b10 http-ece-1.1.0 httplib2-0.18.1 idna-2.10 importlib-metadata-3.3.0 importlib-resources-3.3.1 inflection-0.5.1 itypes-1.1.0 jdcal-1.4.1 lti-0.9.5 lxml-4.1.1 maxminddb-2.0.3 mimesis-0.0.10 more-itertools-8.6.0 oauthlib-3.1.0 olefile-0.46 openpyxl-2.5.14 packaging-20.8 pluggy-0.13.1 polib-1.0.8 psycopg2-2.7.3.2 py-1.10.0 py-vapid-1.7.0 pyOpenSSL-17.5.0 pycparser-2.20 pyenchant-3.1.1 pyparsing-2.4.7 pytest-4.0.0 pytest-django-3.4.4 pytest-factoryboy-2.0.2 pytest-mock-1.10.0 pytest-sugar-0.9.2 python-dateutil-2.6.1 python-magic-0.4.20 pytz-2017.3 pywebpush-1.10.0 radar-0.3 raven-6.6.0 reportlab-3.5.57 requests-2.21.0 requests-oauthlib-1.3.0 rules-1.3 setuptools-52.0.0 signxml-2.5.2 six-1.11.0 soupsieve-2.1 termcolor-1.1.0 text-unidecode-1.3 typing-3.6.6 typing-extensions-3.7.4.3 uritemplate-3.0.1 urllib3-1.21.1 virtualenv-20.2.2 waitress-1.4.4 webencodings-0.5.1 whitenoise-5.1.0 xhtml2pdf-0.2b1 xmltodict-0.11.0 zipp-3.4.0
You are using pip version 18.1, however version 21.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Running python /tmp/app/manage.py collectstatic --noinput --traceback
Error staging application global-trainingByBijay: timed out after 15 minute(s)
FAILED

[Bijaya196]

could you please any one help me on this

[ryanmoran]

There's not a lot in these logs to indicate what might be going on here. I can see that the build timed out after 15 minutes, which is rather surprising for a Python app. Can you provide the entirety of the log output so that we might be able to have a better idea of what's happening? It would also help if you could include an example that reproduces this error.