2018.11.14 regression: `pipenv lock -r > ...` creates corrupted requirements file #3239

Zebradil · 2018-11-16T12:37:26Z

Issue description

I'm trying to re-build an image from the following Dockerfile:

FROM python:3.6-alpine
WORKDIR /opt/app
RUN pip install pipenv
COPY Pipfile .
RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt

and getting error on the last step.

Expected result

pip install packages from the requirements.txt successfully.

Actual result

Sending build context to Docker daemon  8.192kB
Step 1/5 : FROM python:3.6-alpine
 ---> 1d981af1e3b4
Step 2/5 : WORKDIR /opt/app
 ---> Using cache
 ---> 1f6e844884d3
Step 3/5 : RUN pip install pipenv
 ---> Using cache
 ---> fe80c45921ee
Step 4/5 : COPY Pipfile .
 ---> 1410d77cb800
Step 5/5 : RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt
 ---> Running in 837048256df0
Creating a virtualenv for this project…
Pipfile: /opt/app/Pipfile
Using /usr/local/bin/python (3.6.7) to create virtualenv…
Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/app-ueEJiAOq/bin/python
Installing setuptools, pip, wheel...
done.
Virtualenv location: /root/.local/share/virtualenvs/app-ueEJiAOq
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (8407de)!
Invalid requirement: '⠋ Running..'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 93, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/constructors.py", line 253, in install_req_from_line
    req = Requirement(req)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8], e.msg
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'⠋ Runnin'": Expected W:(abcd...)

The command '/bin/sh -c pipenv lock -r > requirements.txt; pip install -U -r requirements.txt' returned a non-zero code: 1

Steps to replicate

Create Dockerfile with the contents from the above
Create Pipfile with the following content:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pylint = "*"

[dev-packages]

[requires]
python_version = "3.6"

Run docker build -t pipenv-test .

docker run python:3.6-alpine ash -c 'pip install pipenv; pipenv --support' produces:

$ pipenv --support

Pipenv version: '2018.11.14'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

3.6.7: /usr/local/bin/python
3.6.7: /usr/local/bin/python3.6m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.9.93-boot2docker',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Thu Jul 19 18:29:50 UTC 2018',
 'python_full_version': '3.6.7',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

HOSTNAME
PYTHON_PIP_VERSION
SHLVL
HOME
GPG_KEY
PATH
LANG
PYTHON_VERSION
PWD
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG: C.UTF-8
PWD: /

The text was updated successfully, but these errors were encountered:

duplicate-issues · 2018-11-16T12:37:43Z

Hey @Zebradil,

We did a quick check and this issue looks very darn similar to

#3238 - 2018.11.14 regression: support for editables within an src dir broken
#3237 - Fix race condition when installing 2+ editable non-VCS pkgs at once
#3236 - 2018.11.14 regression: Installing Multiple local editable packages leaves some unusable

This could be a coincidence, but if any of these issues solves your problem then I did a good job 😄

If not, the maintainers will get to this issue shortly.

Cheers,
Your Friendly Neighborhood ProBot

Zebradil · 2018-11-16T12:44:12Z

Tried the same with the previous version (2018.10.13) — works as expected.
I also checked the content of requirements.txt:

✔ Complete .
✔ Success! .
-i https://pypi.org/simple
astroid==2.0.4
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
pylint==2.1.1
six==1.11.0
typed-ast==1.1.0 ; python_version < '3.7' and implementation_name == 'cpython'
wrapt==1.10.11

Look at first two lines. However, I'm not sure why in the error message I see '⠋ Runnin'.

- added a new metadata section to more closely emulate Kubernetes manifests - renamed kv_backend to kv_engine - enabled static analysis and security scans in CI - bumped the version to 0.2.0 - pinned pipenv in Dockerfile to avoid regression in new release, see: pypa/pipenv#3239

karlsvec · 2018-11-16T22:15:13Z

When you run pipenv lock -r, and no virtualenv exists yet, Pipenv will create one. On 2018.11.14, it looks like the animated spinner goes to stdout (instead of stderr with the rest of Pipenv's verbose/informative output). The spinner output ends up in your requirements.txt, which is why you see Invalid requirement: '⠋ Running..'.

Some context: I use the following pattern for bundling Python dependencies as part of an archive builder script for AWS Lambda:

pip3 install -r <(pipenv lock -r) --target .

As a workaround, I was able to restore the old behavior with PIPENV_NOSPIN=1 pipenv --bare lock -r 2>/dev/null in place of pipenv lock -r.

techalchemy · 2018-11-17T02:43:22Z

hrm, this is a good observation. We need to fix this

techalchemy · 2018-11-17T02:43:52Z

(I believe only the frozen final frame of the spinner goes to stdout)

techalchemy · 2018-11-18T22:05:00Z

I have been attempting to track this down but am actually struggling to find the frame which gets written to stdout

techalchemy · 2018-11-18T22:48:09Z

Got it! sarugaku/vistir#41

Successfully installed certifi-2018.10.15 pipenv-2018.11.15.dev0 virtualenv-16.1.0 virtualenv-clone-0.4.0
Removing intermediate container 032e11add394
 ---> c8b1be7e75a5
Step 6/6 : RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt
 ---> Running in 42668600ea5d
Creating a virtualenv for this project…
Pipfile: /opt/app/Pipfile
Running...
Using /usr/local/bin/python (3.6.7) to create virtualenv…
Running... virtual environment...
⠴ Creating virtual environment...Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/app-ueEJiAOq/bin/python
Installing setuptools, pip, wheel...
done.

Removing intermediate container 42668600ea5d
 ---> 312bc682aa72
Successfully built 312bc682aa72
Successfully tagged pipenv-test:latest

- Fixes #3239 Signed-off-by: Dan Ryan <dan@danryan.co>

Locustv2 · 2018-11-23T12:11:58Z

@Zebradil - i am having the same issue. Can you share what you did?
I usually get some other errors with pipenv and when i downgrade the pip version using pipenv run pip install pip==18.0 then it works. But i haven't tested for this issue yet.

Zebradil · 2018-11-23T15:48:30Z

As a workaround you can just downgrade pipenv version:

pip install pipenv==2018.10.13

To check all available versions use

pip install pipenv==

techalchemy mentioned this issue Nov 17, 2018

Provide option to redirect frozen spinner frame to stderr instead of stdout sarugaku/vistir#40

Closed

techalchemy added a commit that referenced this issue Nov 18, 2018

Redirect all spinner output to stderr

589b1a4

- Fixes #3239 Signed-off-by: Dan Ryan <dan@danryan.co>

techalchemy mentioned this issue Nov 18, 2018

Redirect all spinner output to stderr #3251

Merged

techalchemy closed this as completed in #3251 Nov 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2018.11.14 regression: `pipenv lock -r > ...` creates corrupted requirements file #3239

2018.11.14 regression: `pipenv lock -r > ...` creates corrupted requirements file #3239

Zebradil commented Nov 16, 2018

duplicate-issues bot commented Nov 16, 2018

Zebradil commented Nov 16, 2018

karlsvec commented Nov 16, 2018 •

edited

Loading

techalchemy commented Nov 17, 2018

techalchemy commented Nov 17, 2018

techalchemy commented Nov 18, 2018

techalchemy commented Nov 18, 2018 •

edited

Loading

Locustv2 commented Nov 23, 2018

Zebradil commented Nov 23, 2018

2018.11.14 regression: pipenv lock -r > ... creates corrupted requirements file #3239

2018.11.14 regression: pipenv lock -r > ... creates corrupted requirements file #3239

Comments

Zebradil commented Nov 16, 2018

Issue description

Expected result

Actual result

Steps to replicate

duplicate-issues bot commented Nov 16, 2018

Zebradil commented Nov 16, 2018

karlsvec commented Nov 16, 2018 • edited Loading

techalchemy commented Nov 17, 2018

techalchemy commented Nov 17, 2018

techalchemy commented Nov 18, 2018

techalchemy commented Nov 18, 2018 • edited Loading

Locustv2 commented Nov 23, 2018

Zebradil commented Nov 23, 2018

2018.11.14 regression: `pipenv lock -r > ...` creates corrupted requirements file #3239

2018.11.14 regression: `pipenv lock -r > ...` creates corrupted requirements file #3239

karlsvec commented Nov 16, 2018 •

edited

Loading

techalchemy commented Nov 18, 2018 •

edited

Loading