Using VCS dependencies throws KeyError when writing Pipfile.lock

[al-the-x]

When converting an existing project from requirements.txt to use pipenv, I performed a pipenv install, which created the following Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
boto = "==2.36.0"
boto3 = "==1.4.5"
braintree = "==3.37.2"
cachetools = "==1.1.6"
celery = "==4.2.1"
certifi = "*"
cffi = "==1.5.2"
cloudinary = "==1.5.0"
coverage = "==4.0.3"
customerio = "==0.1.10"
dj-static = "==0.0.6"
django-ace = "==1.0.2"
django-admin-sortable = "==2.0.20"
django-admin-tools = "==0.8.1"
django-admin-view-permission = "==1.8"
django-allauth = "==0.32.0"
django-annoying = "==0.8.7"
django-cors-headers = "==2.0.2"
django-dbsettings = {ref = "2eb24b58b50c3d47c185f09c17b99c36a4ec2d76",git = "git://github.com/Anton-Shutik/django-dbsettings.git",editable = true}
django-debug-toolbar = "==1.6"
django-filter = "==1.0.2"
django-mailgun = "==0.8.0"
django-nose = "==1.4.4"
django-oauth-toolkit = "==0.10.0"
django-pylibmc = "==0.6.1"
django-reversion = "==1.10.1"
django-seo-js = {ref = "9a0db3856c2647fe6a09dd0cf4103f9f8b1740f5",git = "git://github.com/groveco/django-seo-js.git",editable = true}
django-storages = "==1.6.5"
djangorestframework = "==3.8.2"
djangorestframework-csv = "==1.4.1"
djangorestframework-jsonapi = "==2.6.0"
django-watson = "==1.3.1"
drf-extensions = "==0.3.1"
freezegun = "==0.3.6"
elasticsearch = "<7.0.0,>=6.0.0"
elasticsearch-dsl = "==6.1.0"
gunicorn = "==19.4.5"
hasoffers = "==0.2.1"
jsmin = "==2.2.0"
kombu = "==4.2.0"
librato-metrics = "==0.8.6"
lxml = "==3.5.0"
mailchimp = "==2.0.9"
mock = "==1.3.0"
psycopg2 = "==2.7.3.2"
pylibmc = "==1.5.1"
pytz = "==2018.5"
raven = "==5.32.0"
redis = "==2.10.5"
requests-aws4auth = "==0.9"
requests = "==2.9.1"
selenium = "==3.3.1"
sauceclient = "==0.2.1"
slacker = "==0.9.9"
sqlparse = "==0.2.2"
stripe = "==1.56.0"
tinys3 = "==0.1.11"
user-agents = "==1.1.0"
wsgiref = "==0.1.2"
zenpy = "==2.0.7"
django-sprinklers = {editable = true,git = "https://github.com/chrisclark/django-sprinklers",ref = "e53053f53e0c5a3ff45a3c6c6b8c70af374854b1"}
django-segments = {ref = "00c9591ab72b099b567f855ca0c3b86367164910",git = "https://github.com/groveco/django-segments",editable = true}
sailthru-client = "==2.3.1"
nose-exclude = "==0.5.0"
ddtrace = "==0.12.0"
zeep = "==2.4.0"
flake8 = "==3.5.0"
urllib3 = "==1.22"
Avalara = "==18.5.2"
Django = "==1.11.6"
Beaker = "==1.9.0"
factory_boy = "==2.8.1"
django_celery_beat = "==1.3.0"
Pillow = "==3.3.1"
rest_condition = "==1.0.3"
XlsxWriter = "==0.8.5"
nose_ittr = "==0.0.4"
Explorer = {ref = "7373383598ce9bba2c6e43e3d32342fef11035ba",git = "https://github.com/groveco/django-sql-explorer",editable = true}

[requires]
python_version = "2.7"

Then pipenv attempts to generate the Pipenv.lock file and generates the following error:

Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==2018.11.26', 'console_scripts', 'pipenv')()
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/cli/command.py", line 332, in lock
    pypi_mirror=state.pypi_mirror,
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/core.py", line 1068, in do_lock
    lockfile=lockfile
  File "/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 682, in venv_resolve_deps
    lockfile[lockfile_section][k].update(v)
KeyError: 'idna'

I have run pipenv lock --clear multiple times and have gotten different KeyError failures for chardet as well, but mostly for idna. The stack trace is the same, regardless of the key that fails the lookup. When I used pipenv install --skip-lock, all the dependencies installed except for one that needed native headers from libmemcached. The error message that pip reported from that failure was very different and Google-able, so I was able to resolve and get it installed. But I still have no Pipfile.lock and pipenv lock --clear any number of times just generates the same error as above. Help?

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.11.26/libexec/bin/python3.7'

Python installations found:

• 3.7.1: /usr/local/bin/python3
• 3.7.1: /usr/local/bin/python3.7m
• 2.7.15: /Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov/bin/python
• 2.7.15: /usr/local/bin/python
• 2.7.15: /usr/local/bin/pythonw
• 2.7.10: /usr/bin/python
• 2.7.10: /usr/bin/pythonw
• 2.7.10: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.1',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '18.2.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST '
                     '2018; root:xnu-4903.231.4~2/RELEASE_X86_64',
 'python_full_version': '3.7.1',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

• PATH
• GREP_COLOR
• LESS_TERMCAP_mb
• TERM_PROGRAM
• LESS_TERMCAP_md
• NVM_CD_FLAGS
• LESS_TERMCAP_me
• PIP_PYTHON_PATH
• SHELL
• TERM
• TMPDIR
• Apple_PubSub_Socket_Render
• TERM_PROGRAM_VERSION
• DIRENV_DIR
• TERM_SESSION_ID
• LESS_TERMCAP_ue
• NVM_DIR
• USER
• LS_COLORS
• GREP_COLORS
• RPROMPT
• SSH_AUTH_SOCK
• __CF_USER_TEXT_ENCODING
• VIRTUAL_ENV
• VIRTUAL_ENV_DISABLE_PROMPT
• PAGER
• WORKON_HOME
• TMUX
• DIRENV_WATCHES
• LESS_TERMCAP_us
• VIRTUALENVWRAPPER_PYTHON
• PIPENV_ACTIVE
• _git_log_oneline_format
• _
• PWD
• EDITOR
• LANG
• TMUX_PANE
• XPC_FLAGS
• PYTHONDONTWRITEBYTECODE
• XPC_SERVICE_NAME
• HOME
• SHLVL
• LESS
• LOGNAME
• PIP_DISABLE_PIP_VERSION_CHECK
• VISUAL
• LESS_TERMCAP_so
• TMUX_PLUGIN_MANAGER_PATH
• NVM_BIN
• BROWSER
• _git_log_medium_format
• DIRENV_DIFF
• SECURITYSESSIONID
• LESS_TERMCAP_se
• PIP_SHIMS_BASE_MODULE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: /usr/local/Cellar/pipenv/2018.11.26/libexec/tools:/usr/local/opt/nvm/versions/node/v10.4.1/bin:/Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov/bin:/Users/al_the_x/bin:/usr/local/Cellar/pipenv/2018.11.26/libexec/tools:/Users/al_the_x/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin
• SHELL: /usr/local/bin/zsh
• EDITOR: /usr/local/bin/nvim
• LANG: en_US.UTF-8
• PWD: /Users/al_the_x/Projects/grove.co
• VIRTUAL_ENV: /Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov

---

Contents of Pipfile ('/Users/al_the_x/Projects/grove.co/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
boto = "==2.36.0"
boto3 = "==1.4.5"
braintree = "==3.37.2"
cachetools = "==1.1.6"
celery = "==4.2.1"
certifi = "*"
cffi = "==1.5.2"
cloudinary = "==1.5.0"
coverage = "==4.0.3"
customerio = "==0.1.10"
dj-static = "==0.0.6"
django-ace = "==1.0.2"
django-admin-sortable = "==2.0.20"
django-admin-tools = "==0.8.1"
django-admin-view-permission = "==1.8"
django-allauth = "==0.32.0"
django-annoying = "==0.8.7"
django-cors-headers = "==2.0.2"
django-dbsettings = {ref = "2eb24b58b50c3d47c185f09c17b99c36a4ec2d76",git = "git://github.com/Anton-Shutik/django-dbsettings.git",editable = true}
django-debug-toolbar = "==1.6"
django-filter = "==1.0.2"
django-mailgun = "==0.8.0"
django-nose = "==1.4.4"
django-oauth-toolkit = "==0.10.0"
django-pylibmc = "==0.6.1"
django-reversion = "==1.10.1"
django-seo-js = {ref = "9a0db3856c2647fe6a09dd0cf4103f9f8b1740f5",git = "git://github.com/groveco/django-seo-js.git",editable = true}
django-storages = "==1.6.5"
djangorestframework = "==3.8.2"
djangorestframework-csv = "==1.4.1"
djangorestframework-jsonapi = "==2.6.0"
django-watson = "==1.3.1"
drf-extensions = "==0.3.1"
freezegun = "==0.3.6"
elasticsearch = "<7.0.0,>=6.0.0"
elasticsearch-dsl = "==6.1.0"
gunicorn = "==19.4.5"
hasoffers = "==0.2.1"
jsmin = "==2.2.0"
kombu = "==4.2.0"
librato-metrics = "==0.8.6"
lxml = "==3.5.0"
mailchimp = "==2.0.9"
mock = "==1.3.0"
psycopg2 = "==2.7.3.2"
pylibmc = "==1.5.1"
pytz = "==2018.5"
raven = "==5.32.0"
redis = "==2.10.5"
requests-aws4auth = "==0.9"
requests = "==2.9.1"
selenium = "==3.3.1"
sauceclient = "==0.2.1"
slacker = "==0.9.9"
sqlparse = "==0.2.2"
stripe = "==1.56.0"
tinys3 = "==0.1.11"
user-agents = "==1.1.0"
wsgiref = "==0.1.2"
zenpy = "==2.0.7"
django-sprinklers = {editable = true,git = "https://github.com/chrisclark/django-sprinklers",ref = "e53053f53e0c5a3ff45a3c6c6b8c70af374854b1"}
django-segments = {ref = "00c9591ab72b099b567f855ca0c3b86367164910",git = "https://github.com/groveco/django-segments",editable = true}
sailthru-client = "==2.3.1"
nose-exclude = "==0.5.0"
ddtrace = "==0.12.0"
zeep = "==2.4.0"
flake8 = "==3.5.0"
urllib3 = "==1.22"
Avalara = "==18.5.2"
Django = "==1.11.6"
Beaker = "==1.9.0"
factory_boy = "==2.8.1"
django_celery_beat = "==1.3.0"
Pillow = "==3.3.1"
rest_condition = "==1.0.3"
XlsxWriter = "==0.8.5"
nose_ittr = "==0.0.4"
Explorer = {ref = "7373383598ce9bba2c6e43e3d32342fef11035ba",git = "https://github.com/groveco/django-sql-explorer",editable = true}

[requires]
python_version = "2.7"

[teeberg]

I'm also seeing this issue and trying to get more info to help with debugging here. The KeyError happens on line 693 here in on the current master branch:

pipenv/pipenv/utils.py

Lines 692 to 693 in 3b3ebb4

	if k in getattr(project, vcs_section, {}) or k not in lockfile[lockfile_section]:
	lockfile[lockfile_section][k].update(v)

The condition before even reads ... or k not in lockfile[lockfile_section], i.e. it seems that it's somewhat expected that k, which is the name of a dependency here, might not be in lockfile[lockfile_section] yet, but then the next line tries to read it anyway? Even though we just checked that it's not in there? 🤔

If I'm interpreting this code correctly, vcs_lockfile contains the dependencies of the latest versions of my vcs dependencies, so it seems to be stumbling over one of my git dependencies having added a dependency in a later version than I have installed.

[al-the-x]

I think you're right @teeberg. Here's the commit where that was intro'd:

17a9c3f#diff-4b9d1c4c6c693a89e0cd2e6d9ca27631R678

It seems like that predicate is wrong. I edited my locally installed pipenv/utils.py to invert the predicate, and pipenv install and pipenv lock --clear both produced a consumable Pipfile.lock with the VCS deps properly checked out as editable packages as confirmed by pipenv graph and pip freeze within the pipenv shell. I'll annotate the commit message and link it to this issue.

[al-the-x]

Per the commit comment linked above:

-         if k in getattr(project, vcs_section, {}) or k not in lockfile[lockfile_section]:
+         if k in getattr(project, vcs_section, {}) and k in lockfile[lockfile_section]:

Does resolve the problem with pipenv lock for me. I'll look around for a corresponding test...

[teeberg]

That fixed it for me too! :)

[al-the-x]

In the case of your change, the second predicate after the or would never be tested, I think. The vcs_lockfile is generated from project[vcs_section], if I'm reading that correctly, so each k in vcs_lockfile.items() must exist in project[vcs_section]... Maybe the update line isn't even necessary? Or maybe it should be a conditional update if the key exists and an assignment if it does not? Like:

    for k, v in vcs_lockfile.items():
        if k in getattr(project, vcs_section, {}):
            if k in lockfile[lockfile_section] lockfile[lockfile_section][k].update(v)
            else lockfile[lockfile_section][k] = v

Someone might be able to code-golf or Pythonify that for me. It would also be helpful to get @techalchemy's take on the above...

[al-the-x]

I wrote @techalchemy via the email address that he lists on GitHub. Hopefully he can weigh in.

[al-the-x]

Thanks for checking in on my line note @techalchemy. Happy to submit a PR for those changes, but they seem pretty small if your prefer to tackle them.

[techalchemy]

Most likely I’ll catch them in my current net of changes. I have two rounds of fixes to get working before I start crossing out the small wins and open PRs. I took a long break so my dream of handling it in a weekend wasn’t to be

[al-the-x]

No worries. Best of luck.