New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2018.11.14 regression: `pipenv lock -r > ...` creates corrupted requirements file #3239

Closed
Zebradil opened this Issue Nov 16, 2018 · 9 comments

Comments

Projects
None yet
4 participants
@Zebradil

Zebradil commented Nov 16, 2018

Issue description

I'm trying to re-build an image from the following Dockerfile:

FROM python:3.6-alpine
WORKDIR /opt/app
RUN pip install pipenv
COPY Pipfile .
RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt

and getting error on the last step.

Expected result

pip install packages from the requirements.txt successfully.

Actual result

Sending build context to Docker daemon  8.192kB
Step 1/5 : FROM python:3.6-alpine
 ---> 1d981af1e3b4
Step 2/5 : WORKDIR /opt/app
 ---> Using cache
 ---> 1f6e844884d3
Step 3/5 : RUN pip install pipenv
 ---> Using cache
 ---> fe80c45921ee
Step 4/5 : COPY Pipfile .
 ---> 1410d77cb800
Step 5/5 : RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt
 ---> Running in 837048256df0
Creating a virtualenv for this project…
Pipfile: /opt/app/Pipfile
Using /usr/local/bin/python (3.6.7) to create virtualenv…
Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/app-ueEJiAOq/bin/python
Installing setuptools, pip, wheel...
done.
Virtualenv location: /root/.local/share/virtualenvs/app-ueEJiAOq
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (8407de)!
Invalid requirement: '⠋ Running..'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 93, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/constructors.py", line 253, in install_req_from_line
    req = Requirement(req)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8], e.msg
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'⠋ Runnin'": Expected W:(abcd...)

The command '/bin/sh -c pipenv lock -r > requirements.txt; pip install -U -r requirements.txt' returned a non-zero code: 1

Steps to replicate

  1. Create Dockerfile with the contents from the above
  2. Create Pipfile with the following content:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pylint = "*"

[dev-packages]

[requires]
python_version = "3.6"
  1. Run docker build -t pipenv-test .

docker run python:3.6-alpine ash -c 'pip install pipenv; pipenv --support' produces:

$ pipenv --support

Pipenv version: '2018.11.14'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

  • 3.6.7: /usr/local/bin/python
  • 3.6.7: /usr/local/bin/python3.6m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.9.93-boot2docker',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Thu Jul 19 18:29:50 UTC 2018',
 'python_full_version': '3.6.7',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

  • HOSTNAME
  • PYTHON_PIP_VERSION
  • SHLVL
  • HOME
  • GPG_KEY
  • PATH
  • LANG
  • PYTHON_VERSION
  • PWD
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_SHIMS_BASE_MODULE
  • PIP_PYTHON_PATH
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  • LANG: C.UTF-8
  • PWD: /

@duplicate-issues

This comment has been minimized.

duplicate-issues bot commented Nov 16, 2018

Hey @Zebradil,

We did a quick check and this issue looks very darn similar to

This could be a coincidence, but if any of these issues solves your problem then I did a good job 😄

If not, the maintainers will get to this issue shortly.

Cheers,
Your Friendly Neighborhood ProBot

@Zebradil

This comment has been minimized.

Zebradil commented Nov 16, 2018

Tried the same with the previous version (2018.10.13) — works as expected.
I also checked the content of requirements.txt:

✔ Complete .
✔ Success! .
-i https://pypi.org/simple
astroid==2.0.4
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
pylint==2.1.1
six==1.11.0
typed-ast==1.1.0 ; python_version < '3.7' and implementation_name == 'cpython'
wrapt==1.10.11

Look at first two lines. However, I'm not sure why in the error message I see '⠋ Runnin'.

bincyber added a commit to bincyber/pkictl that referenced this issue Nov 16, 2018

Updated schemas with new metadata section
- added a new metadata section to more closely emulate Kubernetes manifests
- renamed kv_backend to kv_engine
- enabled static analysis and security scans in CI
- bumped the version to 0.2.0
- pinned pipenv in Dockerfile to avoid regression in new release, see: pypa/pipenv#3239
@karlsvec

This comment has been minimized.

karlsvec commented Nov 16, 2018

When you run pipenv lock -r, and no virtualenv exists yet, Pipenv will create one. On 2018.11.14, it looks like the animated spinner goes to stdout (instead of stderr with the rest of Pipenv's verbose/informative output). The spinner output ends up in your requirements.txt, which is why you see Invalid requirement: '⠋ Running..'.

Some context: I use the following pattern for bundling Python dependencies as part of an archive builder script for AWS Lambda:

pip3 install -r <(pipenv lock -r) --target .

As a workaround, I was able to restore the old behavior with PIPENV_NOSPIN=1 pipenv --bare lock -r 2>/dev/null in place of pipenv lock -r.

@techalchemy

This comment has been minimized.

Member

techalchemy commented Nov 17, 2018

hrm, this is a good observation. We need to fix this

@techalchemy

This comment has been minimized.

Member

techalchemy commented Nov 17, 2018

(I believe only the frozen final frame of the spinner goes to stdout)

@techalchemy

This comment has been minimized.

Member

techalchemy commented Nov 18, 2018

I have been attempting to track this down but am actually struggling to find the frame which gets written to stdout

@techalchemy

This comment has been minimized.

Member

techalchemy commented Nov 18, 2018

Got it! sarugaku/vistir#41

Successfully installed certifi-2018.10.15 pipenv-2018.11.15.dev0 virtualenv-16.1.0 virtualenv-clone-0.4.0
Removing intermediate container 032e11add394
 ---> c8b1be7e75a5
Step 6/6 : RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt
 ---> Running in 42668600ea5d
Creating a virtualenv for this project…
Pipfile: /opt/app/Pipfile
Running...
Using /usr/local/bin/python (3.6.7) to create virtualenv…
Running... virtual environment...
⠴ Creating virtual environment...Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/app-ueEJiAOq/bin/python
Installing setuptools, pip, wheel...
done.

Removing intermediate container 42668600ea5d
 ---> 312bc682aa72
Successfully built 312bc682aa72
Successfully tagged pipenv-test:latest

techalchemy added a commit that referenced this issue Nov 18, 2018

Redirect all spinner output to stderr
- Fixes #3239

Signed-off-by: Dan Ryan <dan@danryan.co>
@Locustv2

This comment has been minimized.

Locustv2 commented Nov 23, 2018

@Zebradil - i am having the same issue. Can you share what you did?
I usually get some other errors with pipenv and when i downgrade the pip version using pipenv run pip install pip==18.0 then it works. But i haven't tested for this issue yet.

@Zebradil

This comment has been minimized.

Zebradil commented Nov 23, 2018

As a workaround you can just downgrade pipenv version:

pip install pipenv==2018.10.13

To check all available versions use

pip install pipenv==
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment