Can't lock package with `git` and `extras` #2504

JacobHayes · 2018-07-03T18:33:17Z

Issue description

A package spec with git and extras fails to lock. This happens regardless of the editable flag.

Expected result

The package should be installed with all extras.

Actual result

[I] ~/test $ pipenv --version
pipenv, version 2018.7.1
[I] ~/test $ ll
total 8
-rw-r--r--  1 jacobhayes    96B Jul  3 13:12 Pipfile
[I] ~/test $ cat Pipfile
[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}
[I] ~/test $ pipenv --rm ; pipenv install --verbose
Removing virtualenv (/Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt)...
Creating a virtualenv for this project...
Pipfile: /Users/jacobhayes/test/Pipfile
Using /usr/local/opt/python/bin/python3.6 (3.6.5) to create virtualenv...
⠋Already using interpreter /usr/local/opt/python/bin/python3.6
Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python3.6
Also creating executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python
Installing setuptools, pip, wheel...done.
Setting project for test-qFl43Lkt to /Users/jacobhayes/test

Virtualenv location: /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1943, in do_install
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1312, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1140, in do_lock
    project.write_lockfile(lockfile)
  File "/usr/local/lib/python3.6/site-packages/pipenv/project.py", line 648, in write_lockfile
    content, indent=4, separators=(u',', u': '), sort_keys=True,
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ArrayElement' is not JSON serializable

Steps to replicate

[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}

pipenv install

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

2.7: /usr/local/bin/python2.7
2.7: /usr/local/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/local/bin/python3.6m
3.6: /usr/local/bin/python3.6
2.7.15: /usr/local/bin/python
2.7.10: /usr/bin/python
2.7.15: /usr/local/bin/python2
3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT '
                     '2018; root:xnu-4570.61.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

Apple_PubSub_Socket_Render
COLORFGBG
COLORTERM
COPYFILE_DISABLE
EDITOR
GOPATH
HOME
ITERM_PROFILE
ITERM_SESSION_ID
LANG
LOGNAME
PAGER
PATH
PWD
SECURITYSESSIONID
SHELL
SHLVL
SSH_AUTH_SOCK
TERM
TERM_PROGRAM
TERM_PROGRAM_VERSION
TERM_SESSION_ID
TMPDIR
TMUX
TMUX_PANE
TMUX_PLUGIN_MANAGER_PATH
USER
XPC_FLAGS
XPC_SERVICE_NAME
__CF_USER_TEXT_ENCODING
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /Users/jacobhayes/bin:/Users/jacobhayes/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/jacobhayes/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
SHELL: /usr/local/bin/fish
EDITOR: vim
LANG: en_US.UTF-8
PWD: /Users/jacobhayes/test

Contents of Pipfile ('/Users/jacobhayes/test/Pipfile'):

[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}

The text was updated successfully, but these errors were encountered:

JacobHayes · 2018-07-03T19:25:22Z

This works with v2018.05.18, but fails in v2018.6.25 and v2018.7.1

techalchemy · 2018-07-03T21:41:26Z

Interesting! I can definitely replicate this.

theunraveler · 2018-07-04T05:03:47Z

git bisect tells me that 18530df is the culprit, if that helps.

bryanforbes · 2018-07-05T21:07:49Z

I did some digging today and an easy fix for this is to make the following change in Project.write_lockfile():

    def write_lockfile(self, content):
        """Write out the lockfile.
        """
        newlines = self._lockfile_newlines

        def encode_objects(obj):
            if isinstance(obj, ArrayElement): # prettytoml.elements.array.ArrayElement
                return obj.primitive_value
            return obj

        s = simplejson.dumps(   # Send Unicode in to guarentee Unicode out.
            content, indent=4, separators=(u',', u': '), sort_keys=True,
            default=encode_objects
        )
        with atomic_open_for_write(self.lockfile_location, newline=newlines) as f:
            f.write(s)
            if not s.endswith(u'\n'):
                f.write(u'\n')  # Write newline at end of document. GH #319.

techalchemy · 2018-07-05T22:16:15Z

thanks for digging into this, I know very little about the toml parser -- do you have a sense of what's going on here?

bryanforbes · 2018-07-05T22:28:31Z

When the lockfile is read from the file, it just copies the structure over from the parsed TOML. I'm not sure why, but the TOML structure returns a specialized ArrayElement instead of a Python list (which is why simplejson.dumps() throws an error). Rather than iterate through the TOML coming out of the parser, it looked easier to have simplejson.dumps() convert the ArrayElements to lists.

notEvil · 2018-07-10T13:52:14Z

Development since v2018.5.18 completely killed our deployment. This bug and #2317 are easily worked around (patch and external variable substitution) ~~but #2552 is probably not~~. It would be great to have pipenv tested so that these issues don't find their way to the open :)
Otherwise great work, its much appreciated

techalchemy · 2018-07-10T13:56:55Z

This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1. I took the weekend off of working on this project, but we’d definitely merge a failing test / fix if we haven’t already (not totally caught up here)

theunraveler · 2018-07-10T13:58:24Z

This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1.

This is definitely happening on 3.6.* for me.

JacobHayes · 2018-07-10T18:46:32Z

@bryanforbes would you be willing to submit a PR with your changes or mind if I do?

bryanforbes · 2018-07-10T22:26:55Z

@JacobHayes you're more than welcome to do the PR. I'm swamped this week.

techalchemy added the Type: Bug 🐛 This issue is a bug. label Jul 3, 2018

sigmavirus24 mentioned this issue Jul 6, 2018

Update pipenv version heroku/heroku-buildpack-python#727

Closed

JacobHayes mentioned this issue Jul 10, 2018

Can't bootstrap development environment #2557

Closed

JacobHayes added a commit to JacobHayes/pipenv that referenced this issue Jul 11, 2018

[pypa#2504] Serialize prettytoml ArrayElements as lists

2981e2c

JacobHayes mentioned this issue Jul 11, 2018

Serialize prettytoml ArrayElements as lists #2561

Merged

JacobHayes added a commit to JacobHayes/pipenv that referenced this issue Jul 11, 2018

[pypa#2504] Add test for VCS dep with extras

ad5d928

JacobHayes added a commit to JacobHayes/pipenv that referenced this issue Jul 11, 2018

[pypa#2504] Add test for VCS dep with extras

d1c5ec6

JacobHayes added a commit to JacobHayes/pipenv that referenced this issue Jul 11, 2018

[pypa#2504] Serialize prettytoml ArrayElements as lists

241bfa9

JacobHayes added a commit to JacobHayes/pipenv that referenced this issue Jul 11, 2018

[pypa#2504] Add test for VCS dep with extras

f0a1413

JacobHayes added a commit to JacobHayes/pipenv that referenced this issue Jul 11, 2018

[pypa#2504] Serialize prettytoml ArrayElements as lists

6e05a3e

JacobHayes added a commit to JacobHayes/pipenv that referenced this issue Jul 11, 2018

[pypa#2504] Replace security extra with socks in test

6de915a

uranusjr closed this as completed in #2561 Jul 13, 2018

GonzaloUlla mentioned this issue Aug 24, 2018

Using Swagger for API docs Jumpi96/helpo#125

Merged

hroncok mentioned this issue Oct 12, 2018

Pin pip and pipenv versions for now pyvec/naucse.python.cz#469

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't lock package with `git` and `extras` #2504

Can't lock package with `git` and `extras` #2504

JacobHayes commented Jul 3, 2018

JacobHayes commented Jul 3, 2018

techalchemy commented Jul 3, 2018

theunraveler commented Jul 4, 2018

bryanforbes commented Jul 5, 2018 •

edited

Loading

techalchemy commented Jul 5, 2018

bryanforbes commented Jul 5, 2018

notEvil commented Jul 10, 2018 •

edited

Loading

techalchemy commented Jul 10, 2018

theunraveler commented Jul 10, 2018

JacobHayes commented Jul 10, 2018

bryanforbes commented Jul 10, 2018

Can't lock package with git and extras #2504

Can't lock package with git and extras #2504

Comments

JacobHayes commented Jul 3, 2018

Issue description

Expected result

Actual result

Steps to replicate

JacobHayes commented Jul 3, 2018

techalchemy commented Jul 3, 2018

theunraveler commented Jul 4, 2018

bryanforbes commented Jul 5, 2018 • edited Loading

techalchemy commented Jul 5, 2018

bryanforbes commented Jul 5, 2018

notEvil commented Jul 10, 2018 • edited Loading

techalchemy commented Jul 10, 2018

theunraveler commented Jul 10, 2018

JacobHayes commented Jul 10, 2018

bryanforbes commented Jul 10, 2018

Can't lock package with `git` and `extras` #2504

Can't lock package with `git` and `extras` #2504

bryanforbes commented Jul 5, 2018 •

edited

Loading

notEvil commented Jul 10, 2018 •

edited

Loading