Incorrect index in lockfile

[ryanhiebert]

Issue description

When using a custom index for a package that is also on PyPI, but with different versions, it uses the custom index for the version (which is the higher version, so this is correct), but the lock file tags it as coming from the PyPI index. The version that is locked does not exist on the pypi index, and the PyPI versions do not exist on the custom index.

Expected result

I expected it to get the latest version, and annotate the lockfile with the correct index that the version was retrieved from.

Actual result

Locking works cleanly, but produced the odd output in the lockfile.

Steps to replicate

Use this Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "aspiredu"
verify_ssl = true
url = "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/"

[dev-packages]

[packages]
django-enumfields = "*"

[requires]
python_version = "3.7"

to lock and get this Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "8e1f91149b67ec93c8a959eb66c89860a7dcfba35b07c058890607a0bc16eb7c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "aspiredu",
                "url": "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django-enumfields": {
            "hashes": [
                "sha256:b32faacf10ac20201dc2910a5f440d8132e7cae80e689e14a97bd1394e3a8dd4"
            ],
            "index": "pypi",
            "version": "==0.10.1+aspiredu1"
        }
    },
    "develop": {}
}

---

$ pipenv --support

Pipenv version: '2018.11.14'

Pipenv location: '/home/ryan/.local/lib/python3.6/site-packages/pipenv'

Python location: '/home/ryan/.pyenv/versions/3.6.6/bin/python3.6'

Python installations found:

• 3.7.1: /home/ryan/.pyenv/versions/3.7.1/bin/python3
• 3.7.1: /home/ryan/.pyenv/versions/3.7.1/bin/python3.7m
• 3.7.0: /home/ryan/.pyenv/versions/3.7.0/bin/python3
• 3.7.0: /home/ryan/.pyenv/versions/3.7.0/bin/python3.7m
• 3.6.7: /home/ryan/.pyenv/versions/3.6.7/bin/python3
• 3.6.7: /home/ryan/.pyenv/versions/3.6.7/bin/python3.6m
• 3.6.6: /home/ryan/.pyenv/versions/3.6.6/bin/python3
• 3.6.6: /home/ryan/.pyenv/versions/3.6.6/bin/python3.6m
• 3.6.6: /usr/bin/python3
• 3.6.6: /usr/bin/python3.6m
• 2.7.15rc1: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-38-generic',
 'platform_system': 'Linux',
 'platform_version': '#41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• CLUTTER_IM_MODULE
• LS_COLORS
• LESSCLOSE
• XDG_MENU_PREFIX
• LANG
• GDM_LANG
• DISPLAY
• QT_STYLE_OVERRIDE
• COLORTERM
• PYENV_VIRTUALENV_INIT
• XDG_VTNR
• SSH_AUTH_SOCK
• MANDATORY_PATH
• XDG_SESSION_ID
• XDG_GREETER_DATA_DIR
• USER
• DESKTOP_SESSION
• QT4_IM_MODULE
• TEXTDOMAINDIR
• GNOME_TERMINAL_SCREEN
• DEFAULTS_PATH
• QT_QPA_PLATFORMTHEME
• PWD
• HOME
• TEXTDOMAIN
• SSH_AGENT_PID
• QT_ACCESSIBILITY
• XDG_SESSION_TYPE
• XDG_DATA_DIRS
• XDG_SESSION_DESKTOP
• GTK_MODULES
• TERM
• SHELL
• VTE_VERSION
• XDG_SEAT_PATH
• QT_IM_MODULE
• XMODIFIERS
• IM_CONFIG_PHASE
• XDG_CURRENT_DESKTOP
• GPG_AGENT_INFO
• GNOME_TERMINAL_SERVICE
• XDG_SEAT
• SHLVL
• PYENV_SHELL
• LANGUAGE
• GDMSESSION
• GNOME_DESKTOP_SESSION_ID
• LOGNAME
• DBUS_SESSION_BUS_ADDRESS
• XDG_RUNTIME_DIR
• XAUTHORITY
• XDG_SESSION_PATH
• XDG_CONFIG_DIRS
• PATH
• SESSION_MANAGER
• LESSOPEN
• GTK_IM_MODULE
• OLDPWD
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/ryan/bin:/home/ryan/.local/bin:/home/ryan/.pyenv/plugins/pyenv-virtualenv/shims:/home/ryan/.pyenv/shims:/home/ryan/.pyenv/bin:/home/ryan/.local/bin:/home/ryan/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/ryan/Code/aspiredu/pipenv-indices

---

Contents of Pipfile ('/home/ryan/Code/aspiredu/pipenv-indices/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "aspiredu"
verify_ssl = true
url = "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/"

[dev-packages]

[packages]
django-enumfields = "*"

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/home/ryan/Code/aspiredu/pipenv-indices/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "8e1f91149b67ec93c8a959eb66c89860a7dcfba35b07c058890607a0bc16eb7c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "aspiredu",
                "url": "https://pypi.fury.io/OHXIw-9Umgf3jlbpG5ULSp97w456Q/aspiredu/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django-enumfields": {
            "hashes": [
                "sha256:b32faacf10ac20201dc2910a5f440d8132e7cae80e689e14a97bd1394e3a8dd4"
            ],
            "index": "pypi",
            "version": "==0.10.1+aspiredu1"
        }
    },
    "develop": {}
}

[techalchemy]

It just lists pypi as the index specifically because it locked it from the alternate index despite the fact that pypi is listed first (i.e. is the primary index), so it knows it should check there first

I am curious to know if this causes issues though, I'm not sure we thought this through much

[ryanhiebert]

My thought process is that if it's in the lockfile, it's because it's part of what needs to be considered as locked. If we don't actually care which index it comes from, should it even be in the lockfile?

[ryanhiebert]

I do sometimes have issues with multiple indexes where the index chosen to install from first doesn't have the package, so it does seem to me that the index being specified is important to get correct.

[nickwilliams-eventbrite]

I'm also seeing this problem. Here's a bit more info:

Our project Pipfiles have two sources at the top:

[[source]]
url = "https://our.jenkins.server/job/wheelhouse-index/Indexes/16.04/simple/"
verify_ssl = true
name = "our_wheelhouse"

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

Our private wheelhouse holds closed-source projects that we don't release publicly. When we run pipenv lock on a project, every dependency in the Pipfile.lock ends up having "index": "our_wheelhouse" in it, including all the open source projects that actually come from PyPi. This is despite the fact that we don't mirror anything in our private wheelhouse: It contains only our closed-source projects.

The order of the [[source]]s does not matter. If we put PyPi first, our private wheelhouse is still listed as the index for every dependency, even those on PyPi.

Interestingly, it doesn't seem to affect anything. Future installs using the lock file still work fine. There are no errors. But this is clearly not accurate data in the lock file, and I'm concerned that, in future pipenv versions, it could break something.

[ryanhiebert]

That's pretty much the way it's happening for me as well, except that, for reasons I don't understand, it doesn't lock everything to the internal index, some things get set to PyPI, and some to the internal index. I don't have a clue how it chooses, and it's very commonly incorrect.

I also don't see it affecting the installs for the most part, except in the case I describe in #3250, which is related to the extent that it is an example of the index specified in the lockfile being ignored during the install.

[erdnaxeli]

I have the exact same behavior as @nickwilliams-eventbrite . Is this a problem? is this normal? What does this index entry actually means (subdependencies do not have it)?

[timsavage]

Just come across this problem.

In my case, it appears to be a bit random. Some packages from pypi are being attributed to my private (mainly containing prebuilt wheels for our environment) repository but not all.

The main side effect is the install stage takes longer as packages will initially fail to install, but are then subsequently installed when they are retried (presumably using the other repository listed).

While the outcome is all expected packages are installed, it takes longer than required and causes a few double takes when reading through the lock file seeing packages that I know are not in the private repository listed. Our junior developer was confused by the issue when they first encountered it.

[mlgarchery]

Same issue here, a package coming from a private package index is labeled with index="pypi". This is confusing because looking at either the Pipfile or Pipfile.lock, there is no way to know from where index the package comes from.
Before looking at the Pipfile.lock, I was expecting the index to be indicated in the Pipfile (maybe for non-PyPi package only), once a pipenv install occurs, so that it looks like in the doc :

[packages]
requests = {version="*", index="home"}
maya = {version="*", index="pypi"}
records = "*"

[fridex]

Observed this a few times as well. This behavior breaks integration with other tools in our cases (e.g. micropipenv).

[matteius]

These issues I believe have since been resolved.